No More Big Bang Releases

That was awesome all! 🥳 Thanks for all the extra hours, the extra push to make this release happen!

Let’s never do that again.

I have career-long struggles with many things, and deadlines and big bang releases definitely make an appearance in my top 100.

This struggle started probably around 12 years ago at Cloud9 IDE (well before its AWS acquisition). It was my first time working on “serious,” non-hobby, non-academic software. And it was the first job where I had somehow navigated myself into a management role. We had a lot of VC money in the bank; the inventor of JavaScript on our advisory board; and the ambition to change the (software development) world. What could possibly go wrong?

We were ramping towards a huge release. Cloud9 IDE 2.0, which would revolutionize software development forever. We were building a browser-based Integrated Development Environment (IDE) in the cloud, y’all! An amazing code editor implemented in JavaScript. Real-time collaborative code editing. Isolated cloud-hosted runtime environments (this was way before mainstream container technology).

The Future™: we are bringing it to you (too) early, courtesy of Cloud9 IDE.

You are welcome, world.

A launch date was set. The tech press had received private demos of what we were going to ship; they had articles ready to publish.

And obviously, the team crunched like crazy. We worked day and night as the launch date neared.

And there it was, the big moment.

We pushed the button and flipped over our IPs to the freshly deployed system.

So this is why they call these Big Bang releases.

It was a complete disaster.

Servers died due to traffic. Race conditions happened that we never saw before. Our (node.js v0.6) processes would randomly crash. After weeks of struggle, we finally got back into a somewhat stable state. Probably, because a lot of customers had given up trying. You kind of expect your IDE to just, you know, load.

It was bad.

Never again. I told myself.

No more big bang releases.

This is one of those traumatic experiences that at some level you would never wish onto anybody, while on the other hand has a level of impact on you that is hard to replicate. Hearing stories from people (like me) still talking about such events 12 years later is one thing, actually going through it yourself is quite another.

So I suggest that on the side, you spin up a quick YOLO start-up (I recommend in the area of AI), let it explode over the weekend with a big bang release and then come back and join us here on Monday.

I’ll wait.

Ever since, my credo has been: “iterate all the things.”

What can we do to cut this into smaller slices? Ok, how can we make those slices even smaller? Even smallerer? Let’s ship individual key strokes continuously.

What can we do to roll this out to a small group first? Yeah, let’s start with just me. I have the magic power to radically speed up Murphy’s Law, by having anything that could go wrong immediately blow up in my face. Stand aside. It’s a talent, what can I say.

Luckily, technology and industry practices have improved a lot this my experience 12 yeas ago. Feature flagging, gradual roll-outs, continuous delivery, trunk-based development have become more of a standard practice. You don’t just flip over your DNS. You ship to 5%, 20% first to see how badly it explodes. Even if you still do “big bang,” you can now control its bigness.

As a fun aside, this practice has become so prevalent that companies can seemingly do big announcements about exciting new products that “will roll out over the next few days” (I’m looking at you OpenAI) and then months later say “yes yes, still rolling out, we should get to you later this fall. We just had my grandmother in the roll-out target group for the first few months. She’s a big Scarlett Johansson’s fan.”

I will not claim I have managed to avoid big bang releases since that event, 12 years ago. I have participated in a few. I know sometimes they are inevitable to some degree. Still, I would gauge the success rate is a bit of a coin flip.

Some of them just work and are successful, and all is great.

Others go wrong. And confusingly, not always in the SpaceX “whoopies, I blew up my rocket” kind of way. They fail in more subtle ways. You spend months developing a feature. Then you launch, it sees very low adoption. Why? As it turns out, nobody cares. Who knew? Could we have known earlier? Yes, by shipping something that was moving in that direction earlier. A smaller iteration. A smaller slice.

Smaller is always better^*†‡