Heroku Is Not Dead — Things

When I read the blog , I had also thought Heroku was done. Then I talked to my friends who still work there, and I don’t think Heroku is dead.

I don’t want to drill on about myself, but it warrants being transparent so you see where my perspectives come from.

I was the tech lead of the production engineering operations experiences team. My team owned, operated, and developed over thirty internal services between 2021 and 2024, covering AWS accounts management, AWS resource introspection, CloudTrail log ingestion pipeline, vulnerability report system, internal service catalog, centralized secrets management, AWS ODCR management, AWS credentials provisioning, certificate management, DNS management, Infrastructure-as-code,OSQuery fleet, etc.

I directly served all of the Heroku engineering teams. I ran 9-11 projects in parallel at any given time. I reported to a manager in Argentina, and mentored three Argentinian engineers on my team. I published 6 technical design docs in a year. I worked with all of the architects. We interfaced with dozens of Salesforce teams from security, compliance, and reliability orgs₂. I wrote a platform and named it uwu, I publicly criticized Heroku for its naming traditions₃. uwu was part of a larger project that was meant to address the pains that I am about to talk about. I made our CEO at the time say uwu in chat during the presentation of this platform, that was my career shitposting peak, I think. I had a Minecraft video in that presentation, man I had so much fun.

In the subsequent times, many of my former colleagues will likely share their insights on the fall of Heroku. Each of us will have a unique view, generated from our own experiences. I know a few folks who would have very thoughtful things to say about things like product engineering, visions, marketing, directions, perhaps even management and leadership. I am going to offer an adjacent side of things, with a focus on organization dynamics in a start-up post acquisition, and how complex system dynamics and poorly-mitigated entropy can get the best of us.

The pains I’ve witnessed and lessons I’m about to share are not unique to Heroku. They are a part of larger patterns I’ve observed in tech across many roles and shared stories. This read will be valuable to people who work at, operate, or care about a tech company making software for other tech companies, with software products that are well-rounded enough to transition from mid-market to enterprise either now or some time in the future. Readers will walk away with some pains and lessons learned about growth, or un-growth, pains.

Here are the questions every tech person writing about a former employer must ask oneself:

• am I sharing things in an ethical and legal way

• does this benefit or hurt my former employer, and the people still working there, whom I care about

These are the questions heavy on my mind as I write this piece: what is ok to share with the public and what is not? I am being very cautious, I want to share lessons learned, and retro in a blameless way.

A Little Preamble

Sun in Vancouver is rare for February, when there is sun we taken advantage of it. I was in the middle of Chunjie preparations when several friends sent https://www.heroku.com/blog/an-update-on-heroku/ (along with various other social media discussions) to me. So there I was, in my pastel-toned jumper, a matcha latte in my hand and attempting to juggle questions from junk removers, cleaners, and my personal assistant — I’d made a commitment to focus on my life technical debts before I delve back into tech full force. My mind was the furthest thing from the role that I almost killed myself over. The rare sunshine was warm on my lilac periwinkle hair (I’d developed a love for purple tones since Heroku), but my fingers felt very cold, and my heart dropping like a rock when I read the announcement.

It’s been a while since I left Heroku after a medical leave caused by my Staff-Engineering-Way-Too-Hard. Separation from things we care about always takes time, I suppose I went through all the stages of grief after I left the projects that meant the world to me as an engineer and tech lead (e.g. I never got to finish building the control plane that was going to manage the thousands of AWS accounts, and that work helped us find quite a good chunk in cost-to-serve savings!) — I’d have found myself in acceptance. Those that have grieved would know: grief is non-linear, and I found myself in grief once again over Heroku, the best and the worst role I’d ever held. I responded to a few messages and read everything in the group chats quietly, with very little to say. Sometimes my feelings get so big I become non verbal, thank you autism.

I have not published any writing for a while ₁ , haven’t really had the urge to talk, really. Today I do, I’d like to talk about what really happened from the perspective of a high-performing IC/tech lead who crashed and burned at a peak, right as Heroku descended further into a less-than-glorious path of a befuddled confusion (perhaps to no individual’s fault, because the system does what the system does).

The Pain of Acquisition: The Push and Pull Dynamic

The Cool Tech Companies always get bought, post-acquisition things are always rough. Leo Tolstoy stated — “All happy families are alike; each unhappy family is unhappy in its own way.” Replace families with any complex system entity, and you have yourself a set of patterns. That is the Anna Karenina Principle:

“All well-adapted systems are alike, all non-adapted systems experience maladaptation in their own way,… But in the chaos of maladaptation, there is an order. It seems, paradoxically, that as systems become more different they actually become more correlated within limits.(source). “

Up until the early 2020s, a lot of people were surprised to hear Heroku was acquired by Salesforce in 2010. This became less and less of a shock as Heroku worked to make the fact more obvious in marketing posts and product decisions. Here are two lines from my personal work journal, the year I joined Heroku in 2021:

• “Heroku is Salesforce’s least favourite step child”

• “Our team is Heroku’s most hated team, everyone hates us lol”

I was able to change the second thing, and my transformative projects aiming to change the first thing did not work.

When a subsystem is to be adapted into a main system, friction occurs. Existing engineering practices and and processes between the enterprise and the acquired company creates internal exhaustion. Suddenly things like identity management, security, compliance, infrastructure resources data must be shared with Another Org Which Does Not Do Things Our Way. I observed that Salesforce was not as efficient as post acquisition integration as it is now back in 2010 — everyone learns from experience. The centralization efforts of Heroku happened much later than it should have, it looked like. I walked right into the middle of it, ten years after acquisition.

I will quote a snippet from my personal work journal in 2023:

“The push and pull dynamic for centralization and decentralization of critical functions is observed across many enterprise spaces: “Should each team handle their own infrastructure provisioning/security/dependency management/incident response and oncall processes, or should all of these be centralized?”

Localizing control means higher expertise and better solution fit at the cost of lack of external introspection, siloed knowledge, and mismatched standards, whereas centralizing control means slowed movement in changes, ineffective collaboration patterns, and solutions that are misfits for subsystems.

Heroku’s historical slowness to accept centralization led to increased pressure to centralize, and the outcomes of eventual mandated centralization have led to deep system ramifications.”

The Pain of Scale: Bigger Things Move Slower

What I saw was inefficiencies produced by good people trying to do good work from multiple sides. With an organization this large (at that time Salesforce had around 80,000 employees), coordination of movements is behemothian. While everybody chuckled at the name, Salesforce’s V2MOM framework is ideally effective for getting people to focus on things together. It is a great framework with a lot of structure, that structure can be a double edged sword. Planning season always sucks, planning season sucks extra hard when interfacing is involved. Planning season sucks extra extra hard for work around compliance, security, and infrastructure stuff when you have around hundreds and thousands of₄ ephemeral instances up and down everyday. Once the planning is done there is the execution, never mind the interruptions, because entropy is a real deal, and Heroku had been under resourced for a while.

So everything moved slowly, even though people worked very hard. Things also happened very quickly, because there were many very smart and hard-working people, and they all wanted to Heroku to do well. Things happened both slow and fast, it really depended on what interruptions meant and what the stakes were. Frankly, there were simply not enough hands.

The Pain of Scale, Again: Many Things No Can Count

One of the biggest issues with Heroku engineering was under-resourcing, actually. I have seen over-resourced organizations fail, that’s a different story. Under-resourced organizations fail in a destitute and honourable way, it’s almost poetic. I’d like to talk more about it, but I’m afraid of over sharing. Everyone did their best, and I believe everyone is still doing their best.

Budgets are negotiated with values offered, and it had appeared to me, that Heroku was not surfacing its full value to its parent org. Heroku was the least favourite step child, because Heroku did a lot of work silently. A lot of Salesforce internal services ran on Heroku, which increased cost-to-serve. So Heroku was spending all this money on behalf of other internal orgs, but there was no way to surface that, unless there were more systemic inventorying. This was tricky to do due to the dimensions and cardinality of resources. Essentially, we were wrangling with infinity. It was a big data problem, and the data would have informed decisions. This is an IC’s take — data-driven decisions could have influenced resourcing needs, one could hope.

In my most feverish burnout IC dreams, I’d imagined myself with the decision-making power to focus on improving internal operations and reducing cost-to-serve. I’ve heard many smart takes on Heroku’s product directions(the shoulda woulda coulda’s), I have nothing new to offer. I will talk about what I know, and what I know is: Heroku could have seemed less like a cost centre, and that was a technical problem to solve with event-driven systems for resource accounting and insights, platformized ready-to-serve API designs with self-serve limited privilege on/off boarding, minimal-friction data egress.

Heroku suffered from other problems too, but this problem was the one we as a team set out to solve. Heroku was one of AWS’s first customers, with the size of its fleet it had many valuable lessons to offer. Modernizing the AWS accounts and resources management for Heroku became my dragon to chase, and I burned myself out doing it. I had proposed to productize the internal platform learnings too — what if Heroku published a Twelve-Factor AWS? We had all these unique learnings, nobody in this industry had seen what we’d seen. I was in talks with folks from AWS about speaking at AWS Re:Invent about the work we were doing. What if we’d have done that? Infrastructure provisioning, maintenance, costs, security, compliance things, all of this stuff being done at scale is a disaster , nobody knows how to tackle it and it seemed like we were figuring it out with data-driven designs and security first APIs₅. I don’t know. I may just be another engineer yelling into the void.

The Pain of Growth: Stuck in Teenage Years

There comes a time when every scrappy start-up must enter the stage of transformation: time to put on grown-up pants, get all systems in order. Here are all the things I’ve seen start-ups run into a wall on, because when people found a thing, no one thinks about what happens in 3, 5, or 10 years.

• identity and access management

• compliance

• security

• infrastructure provisioning and cost-to-serve

• enterprise-level team/org collaboration agreements

• consistent culture (because once it’s set, change is glacial)

The transition of start-ups from small-medium businesses to enterprise-contract ready often lives or dies during this phase. These problems apply to bootstrapped startups, VC-funded ones, and acquired ones alike. Heroku also suffered from some of these at one point or another, and interestingly enough, due to the acquisition time period, Heroku rested at this phase for longer than I’ve seen any other organization did. Perhaps that is why it produced such unique outputs, and perhaps that is why we are witnessing what appears to be a descend into a downward spiral.

When I quit Heroku, I was more of an IC than a leader, and I talked more to IC friends than leader friends. 1.5 years later I find myself retrospecting and mulling on issues with the combined perspectives of an IC and a leader. Both of these roles in me hold great compassion and empathy for everyone in Salesforce and Heroku, and the Heroku community. Heroku was the coolest, it had set precedent on many things DevOps. It was the foundation for many PaaS products we see today, and I feel honoured to have been a part of it. The problems I witnessed at Salesforce are not unique to one enterprise, but are rather problems all organizations face in and outside of tech. Change is hard, change-making when everything happens so fast and everything is so big and so many is extra hard. Things don’t happen the way we want even though we all wanted the same things, but the system simply wasn’t set up for us to want the same things in the same way, or even interpret those things as the same things. Alas, we could keep trying though.

I Don’t Think This is The End For Heroku

A lot of Heroku folks I know are avoiding HackerNews and other sites, for mental health purposes. With an organization this big they cannot speak without redirecting questions to corporate PR and legal departments. As a former employee I have a little more liberty, but I also must observe legalities. All I can say is: it sounds to me like there is hope, as a lot of these pains are being addressed actively.

Other tech friends send their condolences to me, but while I grieve i still have hope. Salesforce is a large organization with many brilliant minds, who knows what’s going to happen next. In the spirit of open source on knowledge, I share these lessons learned. I know personally of many friends in various other enterprises and small-medium sized companies experiencing similar pains, and I sincerely hope there is a way to mitigate some of these for us as an industry. It does seem like we cycle through the same problems every 12 years, and these days I find myself explaining things to mentees like it’s the 2010s again. I was talking about this with my spouse, who is also a Very Burned Out Tech Person. They said:

“It’s more like a tide than a cycle, each wave comes in and makes a little difference.”

I find solace in the fact that system entropy is eternal and unavoidable, but our active actions to address it, though Sisyphean, are meaningful and can bring joy. Change-making is hard, but without change-making no changes are made. Heroku is not dead, it’s changing.

Footnotes

₁ My life and career plans were interrupted this summer due to stalker activities and incidents. Police has been contacted, security systems are now in place, and I’ve spoken to lawyers, as well as private investigators. I finally feel comfortable talking a little more now. I had been in a “share as little as possible” state as being stalked kinda sucks lol

₂ I am not going to use accurate names for orgs and products and things in this article because there is no need for such details for security and legal reasons ya?

₃ Heroku had a tradition of naming things with Japanese words. I had actually intended to name the service Weeb, but got some constructive input around the offensive nature of the word. After speaking to several colleagues, I went with uwu for pragmatism, joy , and constructive social criticism. Technology does not exist in a vacuum, because people do not exist in a vacuum. In our autonomy to make and inspire changes, I retained “uwu” as a gentle mockery and criticism of Heroku’s culturally appropriative naming practices. A few months later Kubernetes released its uwu version, and caused some shitposting on the uwuification of tech, etc etc. But alas, the nuance in the initial uwu naming is in the foot notes.

₄ I was the guy that did the rough math for how many ephemeral instances Heroku spun up and down during a 24-hour period. This ball park is ok to share, but perhaps further details should remain internal.

₅ I even forced myself to write front-end code, and the portal for data egress had a cyberpunk mode where you can toggle between light and dark mode! Colleagues gave accessibility suggestions too and that was nice. The idea was data egress could be done by humans or machines, and it should be completely self-serve and frictionless. Under the hood Big Data Processing Stuffs went on etc etc. Man that was so much fun.

₆ A part of me remains hopeful, a part of me is destitute and grieving. I know a lot of Heroku folks both former and current feel similarly. Hope is a thing we have to cling onto. I have more thoughts here, it’s a basic auth protected page. Folks who know me can reach out for the password.