ChatGPT is just a hyperfancy text generator with a massive corpus of text used to train it.

Because that text is so massive, you're bound to get some interesting and even accurate results for most queries, but don't mistake that for intelligence. It doesn't "understand" anything, it just compares weights internally and spits out the most probable result depending on what you asked for.

This is why ChatGPT fails the moment you ask it for domain-specific stuff that requires a bit of flexibility in interpreting it or why it produces subtle errors and presents it as functional with complete overconfidence.

To be clear, it's not useless[0], but the actual usefulness of ChatGPT as a disrupting thing is far overstated insofar as a Skynet nightmare scenario goes.

[0]: Although I'd still caution with using it to solve programming problems or similar such until any pending copyright matters have resolved, given its also willing to spit out licensed code and that can be a legal disaster.

> ChatGPT is just a hyperfancy text generator with a massive corpus of text used to train it.

> it just compares weights internally and spits out the most probable result depending

While an autoregressive language model can use probabilities from prompts provided by humans, it is not necessary for the model to do so. The model can also generate text based on its own internal state and previous generated tokens.

The latter is what you are presenting, but you and the other stochastic parrot people are missing the fact the model spends time doing the former much more frequently, especially now we have these models "hooked up" to the Internet. At Mitta.us, I've had GPT-3 discussing web pages and PDFs for over a year now, with memories of previous discussions about similar things.

LLMs have their internal (frozen) model, the corpus they are being shown (likely shredded into fragments and embedded with ada-002) and previous interactions with users and themselves. At any point someone implements memories and good search (which they have) then you have a thing that is not so much a parrot as it is an "attention entity" capable of focusing on a thing and discussing it at length.

This doesn't mean during inference that the model is "aware" of anything other than producing probabilities, but given the input is unique (user prompt + neural search for fragments + memories) then the output will also be unique. That unique output may be probabilistic, but that is no different than the way we work when we begin speaking.

> it just compares weights internally and spits out the most probable result depending on what you asked for.

Except it doesn't even do that. Sometimes it decides to go for a less probable option, and goes from there even if it's logically completely wrong. For example, I asked it to generate a sequence of topics for a course. It starts off great with intro topics, and then the topics get progressively more advanced. But then suddenly lists an intro topic it had already listed, before going back to advanced topics.

What it did was it took an alternative completion from the most likely in that moment, even though logically it was 100% not likely. That's why you can't tryst anything this technology outputs for now. If you ask it what's 1 + 1, 99% of the time it'll tell you 2, but sometimes it'll tell you 3 and then argue with you about why it's 3.

You have control of the model. You can make it always pick the most likely choice, you can also have it penalize token that it had already emitted or that are common in the corpus. Chatgpt by default and choice has some more creativity backed in, but in the chat api where you can control the tunables you're going to find what you're looking for

I do, but this thing is going from 0 to world-scale use in a matter of months. It's not being used like you imagine if you think people are choosing their completions.

Neural networks spot patterns, then patterns made up of patterns. As we have seen with chess, and go, the neural networks end up finding patterns that are beyond our comprehension. We are smart, but we cannot fathom the patterns as there are too many layers one on top of another. And you can see this in even chatGPT. I asked it to answer my questions in English, Norwegian, and phonetic Norwegian. Not perfect, but good. Scary good. All three, without missing a beat. These patterns are just too easy for neural networks. So our confidence there is nothing to worry about is founded on a lack of appreciation for how complex this beast is. But in my view the answer is not fear, but acceleration... we need aligned AI's on our side, ready to help protect us...

Thank you for verbalising the issue, overconfidence was the risk I was seeing all along. Widely accepted, shamelessly overconfident, bus still potentially gibberish generator.

> Widely accepted, shamelessly overconfident, bus still potentially gibberish generator.

I believe such a generator can be very harmful in our society that has not yet figured a way to scalably teach critical thinking.

Harmful at societal level by flooding the world with generated content therefore drowning actual information and by creating a lot of overhead.

Harmful at the personal level by increasing the difficulty of sorting information and by creating toxic dependencies (see the Replika scandal).

Harmful at the class level by further concentrating power towards plutocracy.

Harmful at the professional level by eliminating the lower levels of the professional ladder and therefore destroying the pipeline of skill.

And harmful at the interpersonal level by casting doubt on the authenticity of relationships.

Thanks for calling these things out. I didn't see at first glance how language models could be disruptive in these fashions. I am doing research for an article on the decline of civilization. I think that this topic could be a partly related to this topic. Any resources you can point to will probably help all of us.

I would like to read the article.

Unfortunately I don't really know of resources to point you to. These are mostly my thoughts. Most probably though influenced by a diet of sci-fi. I can however expand on them.

For somewhat recent and relevant content check The Orville S03E04. At the time it was just a critique of fake news but I think it is correct about what can happen if AI makes fake news easy to scale infinitely and with very low barriers to entry.

I think presently worrying about Sentient AI is misguided because Generative AI is equally serious and much more urgent. There is no need of further advancement. What I said is possible with current AIs already.

If I ever finish it, I can let you know about it somehow. I'm currently using the material for a science fiction story, but eventually I am going to want to write an article about what I learned. If you want to drop me your contact details, you can reach out to me on my hacker news profile and I will send you details if an when its finished.

> It doesn't "understand" anything, it just compares weights internally and spits out the most probable result depending on what you asked for.

And humans are different how?