I have a somewhat radical, sadly not novel suggestion: build what you're looking for.
I now have a semi-private "HN Reader" which has completely broken my HN habit while still feeding me stuff I might be interested in. Because I built it, I can make it do anything I want. I've started to turn it into a slightly broader search engine so that I can find the things that I'm looking for; I got sick of seeing search engines brag about the hundreds-of-thousands or millions of search results they were returning when I was trying to find something (really, what's the point of that?), so I'm building my own. I got sick of feeding some psychological trigger in my brain that made me nervously check the HN front page numerous times throughout the day, and I'd find myself clicking on items that had lots of comments and activity even if the subject was something I wasn't interested in. I guess I was thinking, "wow, lots of people over there, I should go check that out."
What did it for me was a bit of foggy nostalgia one day. I was thinking about "the good ol' days", how I -- we, all of us if we were lucky enough to be born at the right time in the right environment -- used to modify the crap of out of programs, change their interface, tweak their colors, cheat at games even when we were the only ones playing. We used to take things we didn't like and turn them into things we did like.
But nobody, or very few people, do that for the web, even though there are piles and piles of tools that make it easy and doable.
So I did it.
And it is glorious.
It's some of the most fun I've had at programming in years. Now when I'm feeling like a wet cat, I'll just go tweak my little reader-search-engine-toy, and then I feel better. Now I never feel like I'm missing out on something on HN, because my little toy is keeping an eye on it for me and saving the stuff I might care about it.
And if you're looking for a new community ... well, build that too! It's clear from numerous threads on HN and other places that people are ready for something new. Make what you want, share it if you feel like, if enough other people like it maybe they'll join in and you'll have your community.
Thanks. I got this far after posting my comment. Figured out I needed NodePackageManager. I hate when instructions say npm and dont say what npm means. Now I am getting another error when I run mrt inside the telescope folder. I am working on it.. I love this. I spent weeks trying to get newscloud going, but never did...
Bayesian filtering seems to work great on HN headlines. I trained mine with about two years of data scrapped from http://www.daemonology.net/hn-daily/ (apologies to whoever runs that), then just basically wrapped that with some code that grabs hacker news's main page and displays the filtered headlines to me. It nails politics and startup crap with stunning accuracy.
The only problem is that now I find myself using both that system and the website itself.
HNSearch is great and I use it for my Wayback Letter project, but it hasn't been around that long so I couldn't use it before. When I started Hacker Newsletter about 2.5 years ago there was nothing, so scraping was the only solution. One of these days I will convert the application I built to build out each issue, but there also is a risk of HNSearch going away just like the last search engine did.
I'm curious why you disagree, aside from the quota thing? I have written many API consumers over the years and there is basically no difference between parsing HTML or any other format; XML especially, for obvious reasons.
HTML APIs to come with the added benefit of being less prone to change, which I realize goes against conventional wisdom, but seems to hold in practice.
I agree that parsing can be easily done, although I don't think it is necessarily equal... but I guess that would depend on a case by case basis since a lot of API's are terrible. What makes it not an API IMHO is that you can't consume it when needed, but rather you have to consume it all the time. I guess you will call that a streaming API though. :)
HTML APIs to come with the added benefit of being less prone to change
Really? I don't think that does hold in a lot of cases. Using HN as an example, it broke a year or so ago when PG changed how the job postings were listed. Again, the quality of an API can vary, but at least you would know what changed in that case.
Also, make sure that every time someone performs an action on the site, you generate a random number. Add that random number to the number of seconds since their last performed action. If that number exceeds a threshold (that is assigned randomly and valid for a randomly assigned amount of time) delete a random part of the session.
Worthwhile communities need to be maintained by barriers. The kind of barrier that I personally enjoy most is, "Crappy enough interface that only people who really like the content will be motivated to show up."
I think it just correlates with the presence if images. Also StackExchanges have excellent UIs, APIs and still very good content. StackOverflow itself could also be worse.
That is a good comparison. StackOverflow has a good API, but maintains itself with carefully thought through and zealously enforced moderation rules.
I grant the existence of useful content there and a good API. However after one too many brushes with how differently moderators think about stuff than the people who I liked talking to did, I have demonstrated the truth of my comment by choosing to leave.
StackOverflow works for a lot of people. But for me, personally, a stripped down interface and less moderation is a better experience.
I guess something like the possibility to have a mobile application (either web based or native app) which would be usable (the tiny up- and down-vote arrows of HN are not easy to use using a finger on a touchscreen) and which would actually work by not relying on html-parsing or search kludge.
Sounds interesting, can you give us more details on how it works? I had a quick look at your homepage, but couldn't see any links for it, or a github profile.
I've thought about designing a community with script hacking in mind (a simple initial layout with an api) and then letting users be able to script it themselves through their accounts. If it could even be done safely and sanely, everyone could have the features they wanted by writing their own.
In lieu of that of course there's always greasemonkey.
I now have a semi-private "HN Reader" which has completely broken my HN habit while still feeding me stuff I might be interested in. Because I built it, I can make it do anything I want. I've started to turn it into a slightly broader search engine so that I can find the things that I'm looking for; I got sick of seeing search engines brag about the hundreds-of-thousands or millions of search results they were returning when I was trying to find something (really, what's the point of that?), so I'm building my own. I got sick of feeding some psychological trigger in my brain that made me nervously check the HN front page numerous times throughout the day, and I'd find myself clicking on items that had lots of comments and activity even if the subject was something I wasn't interested in. I guess I was thinking, "wow, lots of people over there, I should go check that out."
What did it for me was a bit of foggy nostalgia one day. I was thinking about "the good ol' days", how I -- we, all of us if we were lucky enough to be born at the right time in the right environment -- used to modify the crap of out of programs, change their interface, tweak their colors, cheat at games even when we were the only ones playing. We used to take things we didn't like and turn them into things we did like.
But nobody, or very few people, do that for the web, even though there are piles and piles of tools that make it easy and doable.
So I did it.
And it is glorious.
It's some of the most fun I've had at programming in years. Now when I'm feeling like a wet cat, I'll just go tweak my little reader-search-engine-toy, and then I feel better. Now I never feel like I'm missing out on something on HN, because my little toy is keeping an eye on it for me and saving the stuff I might care about it.
And if you're looking for a new community ... well, build that too! It's clear from numerous threads on HN and other places that people are ready for something new. Make what you want, share it if you feel like, if enough other people like it maybe they'll join in and you'll have your community.