I’ve been here for four or five years, and I only recognize two or three people by their username. I don’t pay attention to the people - only the comments.
I've been here for just under three years, and I'm in the same boat, except I don't really even have but one username that I'm specifically familiar with due to HN (and that'd be dang). I've recognized familiar usernames from other places (such as when byuu occasionally shows up to talk shop about emulation), but that's a bit different.
I honestly think that's one of HN's strengths. Too often, in general discourse, who is talking becomes as important, if not more important, than what is being said. I feel that the lack of prominence of a poster/commenter's name is a plus - it invites the reader to focus on the message being written, rather than the source.
I pay attention to comments first, too, but I can't help but remember the handles - especially if I had a back-and-forth with someone over the course of a day, and had to verify that I'm still replying to the same commenter. Like, I recognize your handle, I think we've been replying to each other on some thread some weeks ago.
(Then again, maybe I'm projecting, assuming more community in this place than there really is. I'm a HN addict after all.)