It's more concentrated than you give it credit: the wealthy overwhelmingly hold their money in business capital (read: stocks) not housing. Housing is predominantly owned by the middle class. Having stocks go up and down is what mostly changes the ultra-rich wealth, and in this case they're going down.