I don't think you understand the fanout/fanin problems their product has.
Twitter's raison d'être is allowing folks with millions of follows to see their posts in near real time, then monetize it with ads. That means analytics, billing, and more. From pretty basic requirements you can create a huge amount of work, especially when you're managing your own infrastructure.
I'm not saying that it's easy, just that most work could be done by much fewer people. I'm not talking out of my ass, I have been working on scalable systems for more than a decade.
I'm reminded of when, out of sheer curiosity, I once skimmed the proof for Fermat's last theorem by Andrew Willes. It was a monstrous work of which I could understand very little, but that clearly couldn't have been made by a team of a hundred mathematicians working together.
It's not that it isn't possible for a single person to do a monstrous amount of work, but that it takes an organization to do a breadth of things and maintain it over time.
I also don't disagree that some problems are self-inflicted. Large organizations breed large organization problems. But large organizations usually grow from seemingly simple requirements and compliance efforts.
My team certainly got a lot more done before we required tons of process, safety checks, data storage requirements, etc., where we now spend a non-trivial amount of HC as overhead and avoid building other things entirely.
> I'm not talking out of my ass, I have been working on scalable systems for more than a decade.
Ditto, friend. I currently work in an organization that builds a billion user product with far fewer than Twitter does. However, we also depend on a battalion of internal services, and other shared resources (HR, recruiting, PR, finance, ...). Twitter maintains their own datacenters, are among the few users of Aurora and Mesos at this point, and purpose-builds a lot of their infra for requirements we probably don't have a great grasp on.
Twitter, like many products, is like an iceberg: the majority of the complexity is underneath.
Well, I think we're pretty much on the same page regarding where the complexity comes from.
For Twitter though I'm thinking though that they should have put a lot more effort in simplifying the systems/processes/data they store. This might even mean pushing to remove some business requirements that a PM might have introduced just for a promotion to reduce software complexity.
If you don't reward people for refactoring and removing complexity, I don't think you can just blame it all on organic growth.
Twitter's raison d'être is allowing folks with millions of follows to see their posts in near real time, then monetize it with ads. That means analytics, billing, and more. From pretty basic requirements you can create a huge amount of work, especially when you're managing your own infrastructure.