Elevator Pitch

We failed, now what. Let’s talked about in front of bunch of strangers. Let’s dive into all the miss steps my team and I took trying to bring DevOps to a large mature enterprise. With hindsight as my guide, I want to bring to light some of decisions that turned out be trouble and why they were.

Description

Hindsight is 20/20: How I failed at DevOps; lessons learned

Two years ago I started at a large billion dollar corporation as the Sr. Lead Engineer of DevOps, brought in to help usher in DevOps at the enterprise level. I was the leader of new team and my team of three was ready to take on the status quo and red tape at the enterprise level. I saw us as the team that was going to help lift the development teams, help fight their battles at the enterprise level. Something, I thought was needed at the large corporate level. Yes, I understand DevOps is best when started at the grass roots level but low level success needs enterprise level push to bring it to the next level. My team was going to be the catalyst for the teams, coaching them and helping them understand what the next steps were. I was highly optimistic and hopefully.

Fast forward two years; DevOps is failing and its time to figure out why. Was it the effort? Was it the direction? Was it the tools? What happen? Why were we still spinning our wheels and not onboarding teams quickly. Why wes on-boarded teams still not moving quicker?And how in heck is quality on a downward trend?

Please do not get me wrong, we have had our faire share of successes, development teams have gained a lot. One issue is we just have not been able to quantify or show the gains make to date. Metrics and data has been an issue. Also, we had lost our way a bit got narrowly focused on pipeline creation and automating all the best practices and processes. Not all best practices are best for every origination.

We have a ton of validated learnings for all our ups and downs. I think it is time to learn more about it by speaking about it. I am going to attempt to speak/touch on the miss steps my team and I took over the past 24 months, what worked and what didn’t. The difference between DevOps automation and DevOps culture. Key critical hypothesis’ we took for truths when they were not. How I even came to the realization that DevOps was failing.

P.S. Hopefully, I can touch on how we righted the ship, how we got back on track but that story is still being written.

Notes

I am the Sr. Lead Engineer of DevOps for Lincoln Financial, a large mature origination that is performing very well but wants to position its IT in a way that will support the next 100 years of growth. Growth we think requires a DevOps based IT model that allows for quick flow of changes to the consumer with ample feedback and improvement. We used all the buzz phrases “push left”, “peer view”, “git”, “pipelines”, “quality gates” and so forth. We asked the right questions and we assumed we knew what we were doing. We brought in industry standard tools, best of breed type platforms to build our pipelines on. We knew best practices and how to implement and enforce them. We were ready to bring LFG IT into the 21st century.

In the past two years, I have read and re-read many books on DevOps, “The Phoenix Project”, “DevOps Handbook”, “Lean Startup”, “Measure what matters”, “DevOps for the modern Enterprise”, “The Goal” and more. Yet, I still found it hard to change culture and found my team mainly working on automating processes and integrating tools AKA working on pipelines. Moreover, I got my associate level AWS certificate and I have very knowledgeable people helping me. Yet, we somehow lost our way.

We had lost the ‘why” we needed/wanted DevOps, we knew how and what to do to automated and changes processes but we lost the reason why we needed it. And to paraphrase “Start With Why”, no matter how good your “what” and “how” are for making these changes, it will not be a success without a compelling “why” we are doing/need this.

With that said, I want to reflect on my miss steps, enlighten people to the tremendous amount of validated learnings we made.

I have been in the DevOps world for many years before DevOps was coined when I was just an developer who automated builds, releases, deployments, etc.

I’ve worked with many best of breed tools and frameworks AWS, docker, git, gitlab, github, tfs, clearcase, jenkins, sonarqube, .net, powershell, java, etc.

tags

failure, improvement, culture, vulnerability, toolstack

bio

My name is Brian Hopenwasser, most call me Brian Hop. I am a fun loving, hard working DevOps engineer and coach. I’ve worked primarily with very large enterprises IT shops. This has given me a different view point of how DevOps lives and grows.

Truths I believe in are:

  • Version control, it is the truth of all things;
  • It’s not how fast you develop but how well you develop fast;
  • Failing fast is great, but learning fast from failure is best
  • I know nothing without my team

I eat and drink too much but you only live once.

I am a proud philly sports fan; I cheer, I boo, I love my philly teams.

I once traveled the world for a year, and loved traveling, then I had kids. Now I work on my dad-bod.

additional info

Employer?

Lincoln Financial Group

Do you live in Philadelphia?

Philly area, Skippack PA

Would this be your first time speaking at a conference?

Yes

Why is this talk valuable to the community? Give some detail about your plans for the format of this talk. Tell us what makes it particularly valuable to the Philadelphia DevOps community.

When I go to tech conferences, social Media, etc. I only seem to hear how well teams and companies are doing DevOps. Yet when I talk to attendees I hear a different story. I hear a story of hardship and trouble and I want to embrace the fact DevOps is hard. To Automate and Integration is not truly complex and not true DevOps, the culture of DevOps is very hard transition make. I think showing a bit of vulnerability in my talk should stir up very good honest conversation.

My plan is to lay it all out there warts and all. I am unsure my Employer will support me but I think I can get them if I put in the right frame.

Some talking points:

  • Jenkins vs Gitlab CI
    • I believe the new age of CICD tool sets like Gitlab CI, CircleCI , TravisCI, etc. are the next wave of pipeline platform. Yet, Jenkins is used by most large enterprises b/c is dumbs it down for you and create many block box type process with it plugin model.
    • We/I choose to move forward with Gitlab CI, first Gitlab CI is great, yet it is much more complex than Jenkins
      • Trouble is a large number of dev teams I worked with are not ready to take this type of complexity on, most release engineer are not ready either, finding good contractors that know how integrate beyond a Jenkins plugin is difficult.
  • Coupling a migration from SVN to Git (gitlab) with pipelines is not a good idea. It only slows down the migration off the old SVN.
    • we did not do this purposely it just happened because of messaging
  • Having a clear “why” we need a want DevOps is a key, need way to keep focus

Do you require any specials meals, assistance, etc?

None

Are you a member of an under represented community?

Nope