In this case the 50 worker processes started by apache were just not enough to handle all requests. Each page hit creates about 30 requests for html,js,css,png and gif files, so each hit keeps quite a few workers busy. ;)
Well ... actually we can. The web server configuration has been changed to handle this amount of requests and it is working pretty good right now. :) We did not really anticipate that much traffic, as it is far more than what we usually see. So the web server was not configured for such a situation and ran out of workers to handle the requests.
Creating a static site at first was just a much faster and safer solution, especially as the original site did not contain anything apart from the two lines of text.
The link is working again now, but has been replaced with a low bandwidth version. (The original version just contained additional images.) The amount of request generated by this site was unfortunately a bit too much for the web server.
(I'm one of the server admins ... and have been in fear of a real DDOS attack during the last couple of minutes. ;) )
The link got about 13000 hits so far, thereof about 5000 hits with a
referrer value containing "ycombinator". But as only 3481 hits occured
during the 3 days before it was posted on HN, I guess that actually
about 9000 hits are due to HN.
I think the hardware platform is not really interesting, as the DDOS was
only caused by the worker pool of Apache2 being too small. Each hit of
the site creates about 30 additional request for css,js,png and gif
files. So 9000 hits in 3h mean something like 270.000 requests in 3h.
Apparently this was too much for 50 workers.
That looks as if the majority of the hits went directly over to the google cache links mentioned here (otherwise most likely this would be > 50k hits at least).
When the link was hitting the HN front page your site was already blocked - when the first person mentioned that the site was not reachable this HN post had about 60 points - anyway guess its the req/sec that killed you (and no cache - guess you don't have that much static content).
Have you seen many facebook referrers from people who repost HN front page stories to Facebook?
From the 23000 hits so far only about 300 came from faceboook ... not really that much.
I'm pretty sure that the amount of req/sec caused the problems. After a configuration change between 150 and 200 apache2 workers were constantly occupied and neither network ,cpu or ram were even closely used to their capacity. (Before the change we had a hard limit of 50 workers.)
