Executive Exchange: Why every business and executive needs DevOps now: My 15-year journey studying high-performance IT organizations

header_text_exec_1170In a game of charades, if you were asked to describe IT innovation with two syllables, you’d be right if you guessed either DevOps or Gene Kim. Kim is an author and speaker, and he describes his work bringing developers and IT operations together as a “moral crusade.” He is singularly focused on preventing the signs of the downward spiral of IT: breaking promises, and lengthening deployment times.

And that starts with unifying developers and IT operations team. And that’s not easy.

Kim describes the differences between developers and operations like a true Trekkie:

Developers are Spock: A little bit weird, sits closer to the boss, thinks too hard
Operations are Scotty: Pulls levers and turns knobs, easily excited, yells a lot in emergencies.

The opportunity cost of wasted IT efforts is, Kim says, approximately $2.5 trillion/year. That’s why he co-wrote the book The Phoenix Project: A novel about IT, DevOps, and helping your business win. Kim believes the next surge of productivity will come from applying manufacturing’s lean principles to the IT value stream.

The characteristics of the “downward spiral” Kim described might sound familiar. The first is the backlog of promises make when, say, product managers take what one Red Hat Summit Executive Exchange attendee called the “PowerPoint over the wall” approach, and request new projects. Our fragile infrastructure breaks, causing an outage. (Kim’s joke: “Show me a developer who’s not causing an outage, and I’ll show you one who’s on vacation.”) When we break promises, we tend to then over-promise, and end up with what Kim calls “technical debt.” This is an accumulation of “crap” in a datacenter that accrues—and “we’ll fix it when we have time,” Kim said.

Kim’s second downward spiral characteristic: deployments take longer. When operations and developers are at war, we can’t be agile and nimble. There are too many steps to completion. And operations can get mired in unplanned work with whittling away at all that technical debt its accrued.

Everyone loses. And IT becomes order takers.

There is a better way. Companies like Google, Amazon, Netflix, Spottily, Spottily, Twitter, and Facebook have all succeeded at DevOps–and created competitive advantage. They’re prolific. And they’ve found a way to escape what slower IT departments risk: irrelevance.

In 2009, 10 deploys a day was fast. Today, Amazon deploys once every 11.6 seconds. “Winning in the marketplace involves out experimenting the competition,” Kim said.

Kim quoted Intuit founder Scott Cook. “By installing a rampant innovation culture, they now do 165 experiments in the three months of tax season,” Cook said. “Business result? Conversion rate of the website is up 50%. Employee result? The folks just love it, because now their ideas can make it to market.”

He also shared some stats on why companies from many different verticals are adopting a DevOps workflow:

  • They’re more agile
  • 30x more frequent deployments
  • 8,000x faster lead time than peers
  • They’re more reliable
  • 2x the change success rate
  • 12x faster MTTR (mean time to recover)

As a student of efficiency and better workflow management, Kim has identified three characteristics of DevOps:

Flow – This measures deployments per day versus lead time. Rudolph finds that long lead time leads to catastrophic deployment mistakes. That’s why he suggests operations provide environments on demand so anyone who wants one can get one—which is why PaaS [Platform-as-a-Service] is so important. Developing code in production-like environment lessens the risk of deployment errors and business risks. The highest performing IT departments have two characteristics: 89% are using both infrastructure version control and automated code deployments. Flow bottlenecks include environment creation, code deployment, test setup and run, overly tight architecture, development, product management.

Feedback – On the assembly line at Toyota, there’s an “andon” cord anyone can pull to stop production to solve a problem. It’s pulled 3,500 times a day. It’s the only way they can build 2,000 vehicles a day. So many stops and starts might seem disruptive, but it prevents the technical debt from building up. If it goes downstream, we can’t overcome it, Rudolph said. And he suggests that operations pull that cord as well. “To have shared goals, you have to have shared pain,” Rudolph said.

Culture of experimentation and learning – Break things early and often.

If your team exhibits these behaviors, they aren’t high performers:

  • Info is hidden
  • Messengers are shot
  • Responsibilities are shirked
  • Failed is covered up
  • New ideas are crushed
  • Bridging is discouraged

Kim says the downward spiral happens everywhere. But if you employ DevOps and let people control their own outcomes, you can avoid it.



Event: Red Hat Executive Exchange
Date: Tue, April 15, 2014