Hi, one of the Airbnb engineers involved in this op here. Yeah... that does sound a lot like 3 years ago. The situation has gotten a lot better, especially with 5.6 and PIOPS. These days, things work pretty smoothly (even as the volume of traffic and data has scaled massively).

Ah. That is great to hear! It was a special kind of hell having to deal with it with such regularity, at any and all hours (4am?! of course!). ;_;

From my understanding, Aurora is a modified MySQL with a few components substantially rewritten and a new, log-structured storage engine added that is well integrated with Amazon's storage infrastructure (volumes auto-grow, quorum is handled at the storage layer, replicas are just caching views on top of shared, replicated storage). MySQL has support for multiple storage engines over a relatively simple API, while Postgres is tightly integrated with its storage engine.

In addition, MySQL just has a lot fewer features and points of extensibility, so it would likely be easier to support.

We used sparkey for hammerspace, which is very similar to cdb. cdb was actually the first thing we evaluated, and it validated parts of our approach.

This is what we did, except leaning on file system cache instead of explicit mmapping.

Reading code is the best way to get better at programming. It's good to start with really smart, really good code, but you should also read bad and mediocre code (and more importantly, be able to figure out from reading why it is bad or whether it only seems bad at first glance.)

Reading code starts out being really hard, but with practice it can become almost as easy as reading prose. It's a winning proposition on all levels as you can pick up new techniques, understand how things are done (rather than just what they do), and see the trade offs and decisions that are actually made. Practicing code reading also improves debugging skills as you have to use the same skills to understand the what and the why of perfectly working but unfamiliar code.

Whenever anyone asks me what they should do to improve their programming, I usually suggest reading more source code. Another benefit is that code can't lie as easily or as convincingly as the documentation can.

> but you should also read bad and mediocre code (and more importantly, be able to figure out from reading why it is bad or whether it only seems bad at first glance.)

Are there books or websites that have snippets of bad code for readers to test their skills with? I know I've seen something like that for C (maybe as part of a larger book) but what about all the other languages?

One of the neat tricks with using the localhost HAProxy is that developer mode can just be a single Vagrant VM with all services configured to run on the port that they would have in Synapse. With HAProxy all the production servers think they are talking to localhost, and in development they actually are. The Vagrant VM can then be configured/reconfigured using Chef and the production cookbooks (with some overrides in the roles or environments). It should also be possible (though not entirely trivial) to run some of the services in the cloud for development, allowing developers to switch in and out the components that they need to actively develop.

South Africa was hard, but the hole for Lesotho was the big tip off for me.

The #10 state is Virginia, which is 20% African American (compared to California's 6.2%). I'd say that poverty is the more probable root cause.

Sadly, I think this is an increasingly recent phenomenon (get off my lawn). It's kind of funny given the style and length of HN favorites like pg and Steve Yegge.

I think in this case it is more accurate to label it as a parody. It sets up an exaggerated scenario that is clearly ridiculous, but it is only far enough from the current tulip mania to strip away some of the artifice. It's more akin to the I.T. Crowd (but with more verisimilitude), in which case we are exactly the right audience for it.