On Thursday, November 18, 2021, Dropbox did not go down. It so | CatOps

On Thursday, November 18, 2021, Dropbox did not go down. It sounds like a beginning of some modernist novel. Yet, this is rather a post-mortem on an incident that never happened.

This is a great story of leadership and dedication, which lasted for a few years. Results? Dropbox were able to literally pull the network cord from their data center and “do not go down on 18 of November, 2021”. And not just a data center, but their main one.

I really enjoyed this story because it proves once again a few basic things that we prefer not to think about:
- Any project take time and effort. Big projects take a lot of time and effort. If you’re looking into rebuilding your system from scratch, that won’t take a week or two.
- Big projects require dedication. You cannot just add them as a side hassle for your existing team and expect them to deliver everything with the highest quality.
- Iterative improvements. Apollo 11 was the first mission to reach the Moon not because 11 is a pretty number.
- Test and exercise. It’s not enough just to “implement the best practices”, you have to validate if those are actually working as expected. And if there’s a process involved, you have to repeat it frequently enough to not to get rusty.

#infrastructure #culture #postmortem