Lessons learned on the DevOps front: A Red Hat tale

I love a good story. I love it even more when the story is true. Today’s session with two fellow Red Hatters, Katrinka McCallum, VP of product and technology operations, and Jay Ferrandini, senior director of worldwide DevOps, gave me both. And it makes me even more excited that their session was about Red Hat eating its own dogfood or, if you prefer, drinking its own champagne.

“We might not be delivering what our customers want…”

Red Hat engineering had a problem when it came to dealing with Red Hat operations. This was referred to as the “banana and pickle problem.” Engineering/QE would come to operations asking for something–a solution that they desperately needed. Let’s call the requested solution the banana. They came to the team and asked for a banana. Ops went away into a black box development cycle and delivered…a pickle. Not exactly the same thing. Similar in some ways, but not what was requested.

This is an issue that many teams face. They’re segmented in such a way that there’s no collaboration or communication across the teams, at least not in a meaningful way.

Maybe DevOps can save the day?

Enter DevOps and Red Hat technologies

Jay runs the PnT (products and technologies) DevOps team or, more appropriately, the DevOps enablement team. This team is very aware that many DevOps initiatives fail when culture isn’t addressed from the beginning. So, a key piece in this new approach was placing stakeholders within the ops team and having them work beside and along with the team as the solution is being built. On top of that, this team used Red Hat technologies to make all of this happen, such as Ansible by Red Hat, Red Hat Storage, Red Hat CloudForms, Red Hat JBoss Middleware, OpenShift by Red Hat, Red Hat Enterprise Virtualization, OpenStack, Red Hat Enterprise Linux, and Red Hat Satellite.


Photo by Robin Zebrowski

The result of using all of this has been incredibly useful for those products in return. Many of the Ansible playbooks made available have come from this experiment and 1100+ defects were filed against Red Hat products and resolved–many times before they made it out to general availability.


DevOps at work in Red Hat

Katrinka and Jay went on to tell of 4 instances where this new team was able to make big changes inside Red Hat. The first example focused on “Ansibilizing” 400 hosts this past year. Through these efforts, they were able to recoup 10 weeks of infrastructure capacity and take the time to patch thousands of lab systems from 16-20 days down to 10 minutes.

“Automation doesn’t take the people out of the process, it brings them together”

It turns out that by doing this, worlds did not collide. Friendships were born.

(And everyone is much less cranky.)

The second example was around containers and keeping up with demand. The manual approach wasn’t working and there was no way that people could take care of this. So, what do you do when you’re faced with this problem? Make one of the fastest container build systems in the world using OpenShift Dedicated, of course. This is a true hybrid model and can scale out to the cloud when needed. The results:

Then: 116 image rebuilds, 3,120 minutes, 12 people

Now: 116 image rebuilds, 70 minutes, 0 people


Photo by Frank Hebbert

Next up, Katrinka and Jay talked about scaling at Red Hat for efficiency. They wanted to find 15 minutes a day for all Red Hat developers. To make this a reality, they tackled the disparate message bus systems at play in Red Hat–4 separate ones, to be exact. They were able to combine these into a single Red Hat Fuse instance. It wasn’t easy, but they were eventually able to get the 4 groups to “let go and share the road.” This solution ended up saving more than 2000 person days per year. I’d call that success.


Finally: saving money. Skillset wasn’t the area of concern here. Rather, the team looked to workflows and ticketing queues for inspiration. This solution involved making the “DevOps Savings and Loan,” a way of checking out and returning servers for maximum utilization. Guess what…they did it and saw a 42% reduction in deployment times and more than $800,000 in projected savings per year. Through this approach and program, the PnT DevOps team can also now guarantee 100% utilization of hardware.

“You may be the smartest person in the room, but you’re not smarter than the whole room.”


Photo by Cristian Carrara

As it turns out, collaboration and great tools can lead to world peace. And with the right infrastructure, you can do anything you want–gain time, deliver what the people want, be more efficient, and save money.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s