The First & Longest Post-Mortem We’ve Ever Written

I’m writing this on my iPad on a table at GitHub Universe 2019 (in San Francisco). This time last year, Nate and I decided to scrap the Elixir codebase and re-write it in Ruby. This has proven to be the right decision. We are now serving upwards of 1.2 million ads per day on top of this Rails monolith we’ve built.

Since then, we’ve expanded our team. Joseph handles the money. Justin is helping the company grow on the publisher’s side. Andrew and Curtis are insanely talented developers that now help out Nate. Solid team. Best I’ve ever worked with.

The goals we came up with for Q4 are all centered around scalability and simplicity. We broke down those goals into milestones and created assignments to make them happen.

A couple of these goals were assigned to me: new website launch and app UI/UX redesign for simplicity and support of the new features coming.

I knew this would take some time, but I had thought it was something I could handle. I updated my calendly availability to only schedule calls on Tuesdays and Thursdays, leaving the remaining days for me to get into the code and hit some home runs.

Then things started becoming complicated. Some call it “spinning too many plates” or “wearing multiple hats,” the bottom line is, I can’t do multiple jobs anymore. If my primary role is to ensure that CodeFund continues to exist and become self-sustaining, then I need to focus on growing the business, fend off competitors, and all of the other responsibilities that come with being the leader. Things were falling through the cracks, and as much as I want to keep the open source software developer plate spinning, I know that I will continue this unsustainable cycle.

Post-mortem

On November 12th, 2019, a few of our publishers emailed and tweeted about some of their ads not displaying as expected.

   

Here is the timeline of what happened:

EventTime
I (Eric) committed some code with changes that were supposed to only be for the `bottom-bar` ad template Nov 7 at 9:49 AM PST
My PR was merged and Deployed Nov 12 at 7:09 AM PST
Publishers start reporting issues ~8 AM PST

We rolled back the previous deploy~10:08 AM PST
Issues were still being reported
Andrew then put in a hot-fix~11:30 AM PST
Issues were still being reported
Nate then reverted my PR10:10 PM PST

This whole situation led me to have some hard conversations with myself. Although my passion has always been in the code, I accept the responsibility to my peers to focus on ways that they can continue doing what they are passionate about.

So starting today, I am committing to just focus on those responsibilities that will help the community. That includes staying out of the code.

So there you have it. Our first and longest post-mortem we’ve ever written. If you have any questions or concerns or you just want to say hi, feel free to email [email protected]

Thanks for your continued support.

Your friend,

Eric

Image by Gleb Tagirov (CC BY-NC-ND 4.0)

Previous

Next

Share This