The hype people are excited because they're guessing where it's going.
This is notable because it's a milestone that was not previously possible: a driver that works, from someone who spent ~zero effort learning the hardware or driver programming themselves.
It's not production ready, but neither is the first working version of anything. Do you see any reason that progress will stop abruptly here?
Not Windows: Operating systems. We did get more capable operating systems. The point of the quote is "this is the worst the SOTA will ever be".
If Windows XP were fully supported today I still wouldn't use it, personally, despite having respect for it in its era. The core technology of how, eg OS sandboxing, security, memory, driver etc stacks are implemented have vastly improved in newer OSes.
Of course not. But I believe your Windows example was implying fundamental tech got worse.
The original "worst" quote is implying SOTA either stays the same (we keep using the same model) or gets better.
People have been predicting that progress will halt for many years now, just like the many years of Moore's law. By all indications AI labs are not running short of ideas yet (even judging purely by externally-visible papers being published and model releases this week).
We're not even throwing all of what is possible on current hardware technology at the issue (see the recent demonstration chips fabbed specifically for LLMs, rather than general purpose, doing 14k tokens/s). It's true that we may hit a fundamental limit with current architectures, but there's no indication that current architectures are at a limit yet.
>> Do you see any reason that progress will stop abruptly here?
I do. When someone thinks they are building next generation super software for 20$ a month using AI, they conveniently forget someone else is paying the remaining 19,980$ for them for compute power and electricity.
People abstract upon new leaps in invention way too early though. Believing these leaps are becoming the standard. Look at cars, airplanes, phones, etc.
After we landed on the moon people were hyped for casual space living within 50 years.
The reality is it often takes much much longer as invention isn't isolated to itself. It requires integration into the real world and all the complexities it meets.
Even moreso, we may have ai models that can do anything perfectly but it will require so much compute that only the richest of the rich are able to use it and it effectively won't exist for most people.
> Do you see any reason progress will stop abruptly here?
Yeah, money and energy. And fundamental limitations of LLM's. I mean, I'm obviously guessing as well because I'm not an expert, but it's a view shared by some of the biggest experts in the field ¯\_(ツ)_/¯
I just don't really buy the idea that we're going to have near-infinite linear or exponential progress until we reach AGI. Reality rarely works like that.
So far the people who bet against scaling laws have all lost money. That does not mean that their luck won’t change, but we should at least admit the winning streak.
No I don't mean that. I mean the LLM parameter scaling laws. More importantly, it doesn't matter if I mean that or Moore's law or anything else, because I'm not making a forward looking prediction.
Read what I wrote.
I'm saying is if you bet AGAINST [LLM] scaling laws--meaning you bet that the output would peter out naturally somehow--you've lost 100% so far.
You gonna go read up on some 0% success rate strategies on the way?
What I’m saying is that we act as though claims about these scaling laws have never been tested. People feel free to just assert that any minute now the train will stop. They have been saying that since the Stochastic parrots.
It has not come true yet.
Tomorrow could be it. Maybe the day after. But it would then be the first victory.
At the very least, computers are still getting faster. Models will get faster and cheaper to run over time, allowing them more time to "think", and we know that helps. Might be slow progress, but it seems inevitable.
I do agree that exponential progress to AGI is speculation.
I know some proponents have AGI as their target, but to me it seems to be unrelated to the steadily increasing effectiveness of using LLMs to write computer code.
I think of it as just another leap in human-computer interface for programming, and a welcome one at that.
If you imagine it just keeps improving, the end point would be some sort of AGI though. Logically, once you have something better at making software than humans, you can ask it to make a better AI than we were able to make.
The other possibility is, as you say, progress slows down before its better than humans. But then how is it replacing them? How does a worse horse replace horses?
I said I don’t think it follows, and you certainly gave no support for the idea that it must follow. Logically speaking, it’s possible for improvements to continue indefinitely in specific domains, and never come close to AGI.
Progress in LLMs will not slow down before they are better at programming than humans. Not “better than humans.” Better at programming. Just like computers are better than humans at a whole bunch of other things.
Computers have gotten steadily better at adding and multiplying and yet there is no AGI or expectation thereof as a result.
Either the AI can do better than humans at programming, or it can't. If I ask it to make an improved AI, or better tools for making an improved AI, and it can't do it, then at best it's matching human output.
All the current AI success is due to computers getting better at adding and multiplying. That's genuinely the core of how they work. The people who believe AGI is imminent believe the opposite of that last claim.
No one is talking about AGI in this thread except you, though. The post said nothing about it. It's an absolute non sequitur that you brought up yourself.
I've thought for a while now that we'll end up moving to stricter languages that have safer concurrency, etc, partly for this reason. The most prominent resistance against such languages was the learning curve, but humans like OP aren't looking at the code now.
This is standard practice. They need to use current lossless formats to display examples to people who don't have the format yet. They are still showing accurate examples of compression artifacts. I'm not sure what else you'd expect them to do.
Strange, as Cloudinary's test had the opposite conclusion -- jpegxl was significantly faster to decode than avif. Did the decoders change rapidly in a year, or was it a switch to new ones (the rust reimplementation)?
If decode speed is an issue, it's notable that avif varied a lot depending on encode settings in their test:
> Interestingly, the decode speed of AVIF depends on how the image was encoded: it is faster when using the faster-but-slightly-worse multi-tile encoding, slower when using the default single-tile encoding.
They saw a huge uptick in users during the COVID pandemic. As the corona virus is a protein shell, and their software folds protein molecules, they were able to apply it to look for targets for other molecules to attach to the virus where it would normally latch onto a cell, this could then lead to treatments.
They'd found some promising results, and were working with a pharmaceutical company to manufacture the first compounds that could then be tested. Unfortunately that company's facility was located in eastern Ukraine. =(
Folding@home got a boost recently from Pewdipie deploying his 12-stack 4090 build against it and then getting a bunch of his fanbase to also participate in his folding@home squad.
World Community Grid at https://www.worldcommunitygrid.org/ is also running, though it has had struggles since moving datacenters, and it seems their external stats are still out of commission.
I've recently decided to end my own participation, mainly because I've run three systems into the ground, and we're now in the "save what you can" era. There's one motherboard I want to get refurbished, since it became unstable when idle but loved 24x7 crunching. It would make a great NAS if I could find some DDR4 at a price I could stomach, or I could lay it in as a spare if the new motherboard goes south in the future.
How many papers have been published as a result of this, and more pertinently, how many "real" things are now being made or used based on that? I'm hoping it's not all just perpetual "regrowing teeth" territory where nothing ever comes from it.
This is extremely far from any of my expertises, but I'll offer an answer while no one else did (please correct me!). Basically, all medicine (i.e. drugs) we have are proteins or certain compounds that fit within some of our cell's (or viruses) molecules and does funny stuff to them, like disabling certain parts, acting as a signal to regulate behavior, and so on. Doing funny stuff is basically about fitting into another molecule. So research about how proteins (most molecules (after water) in our body, I guess) interact is incredibly important in basically all medicine, specially in the discovery of medicine (like suggesting compounds (drug) that could fit in certain receptors or perform certain function), and understanding disease/pathologies (which give ideas on how to prevent and treat them).
If folding@home helps to understand and model this behavior of molecules (which I guess tends to be difficult and unreliable to do without the aid of computers), it is extremely helpful. Now I don't know other details like, perhaps molecular biology is the bottleneck and there is scant available molecules to analyze (reducing its impact/marginal sensitivity), or perhaps compute really is a bottleneck in this particular problem. But nonetheless it seems like a great project for which contributions do make a difference.
(Note: although, that said, if you were expecting something like 'compute->miracle drug comes out', I believe that's not quite how it works; research in general rarely works that way, I think because the constraint space and problem space that would require this approach is too large and complicated; and in fact I believe many if not most significant discoveries have resulted from playing around and investigating random molecules, often from (nonhuman) animals, plants and bacteria[1]; although molecular sciences (molecular biology) seem to enable a slightly more methodological approach)
Do you think they used an AI or something? Seems to be answering a question I didn't even ask. The strange performative replies I've had to my question makes me more suspicious about folding@home.
Honestly, I doubt it was an LLM because an LLM would have stuck closer to answering the question (avoiding non-sequiturs is the only thing they do, after all) .
I'm not quite sure what the point of the response was.
I wasn't aware asking a question was FUD. That's also a list of achievements with no links without any information regarding how much if any volunteer contributed computing has contributed to them.
> That's also a list of achievements with no links without any information regarding how much if any volunteer contributed computing has contributed to them.
That's papers that are citing them. The reason for no links is explained on the page,
> The distribution rules for published papers vary by the publication in which the paper appears. Due to these rules, a public web-source of each paper may not be immediately available. If full version is not linked below or available elsewhere on the Internet (Google Scholar can be helpful for this), most, if not all of these publications are freely available at a local municipal or collegial library. These articles are written for scientists, so the contents are fairly technical.
I think you’re missing the main limiting resource: money.
Some of these projects could occupy entire regions of cloud compute in some cases for awhile, some even more depending on the problem. But running that for even a short time or decades needed would cost more money than anyone has to do.
Academic HPCs existed long before cloud compute options and for certain problem spaces could also be used even in non-distributed memory cases to handle this stuff. But you still needed allocation time and sometimes even funding to use them, competing against other cases like drug design, cancer research, nuclear testing… whatever. So searching for ET could be crowdsourced and the cost distributed which is something that made it alluring and tractable.
I used to run a small academic cluster that was underutilized but essentially fully paid for. I’d often put some of these projects running as background throttled processes outside scheduler space so the 90% of the time no one was using them, the hardware would at least be doing some useful scientific research since it’s after-all funded largely from federal scientific research funding. There was of course some bias introduced by which projects I chose to support whereas someone else may have made a more equitable choice.
I have several machines contributing to it all the time, and every now and then I run it on my 5090 at home to heat up my room a bit in winter :D It does an incredible 1M points per day, it's a monster of a GPU.
In theory yes, but in practice they usually have the speaker up far higher than they are speaking themselves so we do only hear one side clearly.
I think the high distractability is a trifecta of volume, non-naturallness of the sound (compression etc: feeling out of place in the space) and this point.
In this quote I don't think he means it from the business side. He's claiming more data allows a better product:
> ... the answers are a statistical synthesis of all of the knowledge the model makers can get their hands on, and are completely unique to every individual; at the same time, every individual user’s usage should, at least in theory, make the model better over time.
> It follows, then, that ChatGPT should obviously have an advertising model. This isn’t just a function of needing to make money: advertising would make ChatGPT a better product. It would have more users using it more, providing more feedback; capturing purchase signals — not from affiliate links, but from personalized ads — would create a richer understanding of individual users, enabling better responses.
But there is a more trivial way that it could be "better" with ads: they could give free users more quota (and/or better models), since there's some income from them.
The idea of ChatGPT's own output being modified to sell products sounds awful to me, but placing ads alongside that are not relevant to the current chat sounds like an Ok compromise to me for free users. That's what Gmail does and most people here on HN seem to use it.
The hype people are excited because they're guessing where it's going.
This is notable because it's a milestone that was not previously possible: a driver that works, from someone who spent ~zero effort learning the hardware or driver programming themselves.
It's not production ready, but neither is the first working version of anything. Do you see any reason that progress will stop abruptly here?
reply