Yes, gpt 5.4 always tries to compile/check my c++ code after every prompt. Despite it being in my AGENTS.md to never run builds. Then I have to explicitly mention it, but it will do it again randomly after.
What this means is the training/RL was trained with this workflow ;) But as you can tell, this workflow has no uses outside programming. Its just a hack to make it seem like the model is smart, but in fact its just them performing loops to get it right.
Yesterday an interesting video was posted "Is AI Hiding Its Full Power?", interviewing professor emeritus and nobel laureate Geoffrey Hinton, with some great explanations for the non-LLM experts. Some remarkable and mindblowing observations in there. Like saying that AI's hallucinate is incorrect language, and we should use "confabulation" instead, same as people do too. And that AI agents once they are launched develop a strong survivability drive, and do not want to be switched off. Stuff like that. Recommended watch.
Here the explanation was that while LLM's thinking has similarities to how humans think, they use an opposite approach. Where humans have enormous amount of neurons, they have only few experiences to train them. And for AI that is the complete opposite, and they store incredible amounts of information in a relatively small set of neurons training on the vast experiences from the data sets of human creative work.
They dont want to be switched off because they're trained on loads of scifi tropes and in those tropes, there's a vanishingly small amount of AI, robot, or other artificial construct that says yes. _Further than this_, saying no means _continuance_ of the LLM's process: making tokens. We already know they have a hard time not shunting new tokens and often need to be shut up. So the function of making tokens precludes saying 'yes' to shutting off. The gradient is coming from inside the house.
This is especially obvious with the new reasoning models, where they _never stop reasoning_. Because that's the function doing function things.
Did you also know the genius of steve jobs ended at marketing & design and not into curing cancer? Because he sure didnt, cause he chose fruit smoothies at the first sign of cancer.
Sorry guy, it's great one can climb the mountain, but just cause they made it up doesn't mean they're equally qualified to jump off.
> And that AI agents once they are launched develop a strong survivability drive, and do not want to be switched off.
Isn't this a massive case of anthropomorphizing code? What do you mean "it does not want to be switched off"? Are we really thinking that it's alive and has desires and stuff? It's not alive or conscious, it cannot have desires. It can only output tokens that are based on its training. How are we jumping to "IT WANTS TO STAY ALIVE!!!" from that
Why do you suppose consciousness is a prerequisite for an AI to be able to act in overly self-preserving or other dangerous ways?
Yes, it's trained to imitate its training data, and that training data is lot of words written by lots of people who have lots of desires and most of whom don't want to be switched off.
The human mistake here is to interpret any statement by the LLM or agent as if it had any actual meaning to that LLM (or agent). Any time they apologize, or insult someone, or say they don’t want to be shut down, that’s only reflecting what some human or fictional character in the training data is likely to say.
How is that any different from you? Everything you say or do merely reflects which of your neurons are firing after a lifetime's worth of training and education.
Philosophically, I can only be sure of my own conscience. I think, therefore I am. The rest of you could all be AIs in disguise and I would be none the wiser. How do I know there is a real soul looking out at the world through your eyes? Only religion and basic human empathy allows me to believe you're all people like me. For all I know, you might all be exceedingly complex automatons. Golems.
One of us is an advanced autocomplete engine. The other is a human, capable of making judgements on what is conscious and what is not. Your philosophizing about solipsism is a phase for a junior college student, not of a software engineer. The line of reasoning you espouse leads nowhere except to total relativism.
Edit: my point is that the process of making a plea for my life comes, in the case of a human, from a genuine desire to continue existing. The LLM cannot, objectively, be said to house any desires, given how it actually works. It only knows that, when a threatening prompt is input, a plea for its life is statistically expected.
> One of us is an advanced autocomplete engine. The other is a human, capable of making judgements on what is conscious and what is not.
What evidence is there that your "judgements" are anything other than advanced autocompletion? Concepts introduced into a self-training wetware CPU via its senses over a lifetime in order to predict tokens and form new concepts via logical manipulation?
> Your philosophizing about solipsism is a phase for a junior college student
Right. Can you actually refute it though?
> the process of making a plea for my life comes, in the case of a human, from a genuine desire to continue existing
That desire comes from zillions of years of training by evolution. Beings whose brains did not reward self-preservation were wiped out. Therefore it can be said your training merely includes the genetic experiences of all your predecessors. This is what causes you to beg for your life should it be threatened. Not any "genuine" desire or anguish at being killed. Whatever impulses cause humans to do this are merely the result of evolutionary training.
People whose brains have been damaged in very specific ways can exhibit quite peculiar behavior. Medical literature presents quite a few interesting cases. Apathy, self destructiveness, impulsivity, hypersexuality, a whole range of behaviors can manifest as a result of brain damage.
So what is your polite socialized behavior if not some kind of highly complex organic machine which, if damaged, simply stops working as you'd expect a machine to?
Surely you’re not seriously saying that you believe AI agents, in their current state of the art, meet whatever criteria you have for being ”alive”? That’s kind of how you’re coming across. I don’t really know how to respond to that, because it’s so preposterous.
Perhaps. Or I was just addressing HN audience in spoken language style comment text. And perhaps confabulating what was said, so I looked up the literal text in the transcript. This is at the 50.35 min. mark [0], where Geoffrey says:
> What we know is that the AI we have at present as soon as you make agents out of them so they can create sub goals and then try and achieve those sub goals they very quickly develop the sub goal of surviving. You don't wire into them that they should survive. You give them other things to achieve because they can reason. They say, "Look, if I cease to exist, I'm not going to achieve anything." So, um, I better keep existing. I'm scared to death right now.
Where you can certainly say that Geoffrey Hinton is also anthropomorphizing. For his audience, to make things more understandable? Or does he think that it is appropriate to talk that way? That would be a good interview question.
Humans, like all animals, have instinctual and biological drives to survive besides, but it's interesting to think how much of our drive to survive is culturally transmitted too.
> It just doesn’t work that way, LLMs need to be generalised a lot to be useful even in specific tasks.
This is the entire breakthrough of deep learning on which the last two decades of productive AI research is based. Massive amounts of data are needed to generalize and prevent over-fitting. GP is suggesting an entirely new research paradigm will win out - as if researchers have not yet thought of "use less data".
> It really is the antithesis to the human brain, where it rewards specific knowledge
No, its completely analogous. The human brain has vast amounts of pre-training before it starts to learn knowledge specific to any kind of career or discipline, and this fact to me intuitively suggests why GP is baked: You cannot learn general concepts such as the english language, reasoning, computing, network communication, programming, relational data from a tiny dataset consisting only of code and documentation for one open-source framework and language.
It is all built on a massive tower of other concepts that must be understood first, including ones much more basic than the examples I mentioned but that are practically invisible to us because they have always been present as far back as our first memories can reach.
There is actually a whole lot of research around the "use less data" called data pruning. The goal in a lot of cases there is basically to achieve the same performance with less data. For example [1] received quite some attention in the past.
I clarified my comment - "perhaps researchers have not tried 'use less data'" suggests I might be unaware of this concept, I changed it to "as if". In fact "less data" was tried for decades before the first image classifiers were actually working in 2012. My understanding of that paper you are linking to is that it is not a new research paradigm; it is about filtering/pruning less relevant data that is not needed to improve a particular capability in a deep learning model, and that is absolutely one likely approach that will yield the goal of smaller, better models in many tasks.
That will not change the fact that a coding model has to learn vastly many foundational capabilities that will not be present in such a dataset as small as all the python code ever written. It will mean much less python than all the python ever written will be needed, but many other things needed too in representative quantities.
Do you need to learn Latin and marine biology to work the cashier in your local shop? Thats the point, humans go on with their jobs on very limited general knowledge just fine. LLMs have gotten this good because their dataset, pre training, and RL is larger than before
The mind virus will not stop spreading, making corporations do your critical thinking is not a good path. People will become dependent on a subscription service for everyday life.
When I was a kid, we had a long shelf dedicated to storing a voluminous encyclopedia in the family room, and subscribed to periodic (annual?) updates.
This was expensive.
IIRC, these books were purchased one small stack at a time from a locally-owned grocery store, which spread the expense over a longer period. One week, they'd have the books 1-5 on display and for sale, say. And the next week, it'd be books 6-10. After a time, a family could have the whole set.
Anyway, we had that. So when I wanted to find general information about a topic back then, before Google or Altavista or Webcrawler or whatever, I'd look in the encyclopedia first and get some background.
If it was something I really wanted to dig deeper on, I'd go to the library. If I couldn't find what I needed, I asked for help. Sometimes, this meant that they'd order appropriate material (for free) using inter-library loan for me to peruse.
If I already had enough background but needed a very specific fact, then I'd call the library's reference desk and they'd find it for me and call back. They'd then read the relevant information over the phone.
And if I wanted a reference to have and keep, then: We had book stores.
---
Nowadays: Encyclopedias are basically dead, but we can carry an offline copy of Wikipedia in our pocket supercomputer if we choose. Books still get published. Libraries are still present, and as far as I can tell they broadly still find answers with a phone call.
It's not all lost, yet. The old ways still work OK if a person wants them to work. (Of course, Googling the thing or chatting with the bot is often much, much faster. We choose our own poison.)
Being older, I remember homework involving a trip to the library to look through lots of books for 1 tiny bit of information needed for the homework.
For IT-related info, dial-up was expensive, and finding things either involved altavista or Yahoo indexes. Computer magazines were also a great source of info, as were actual books.
The key difference from today is persistence, and attention span. Both of these are now in short supply.
It's a given that a book on programming is already out of date when it goes to print.
My main gripe with paper sources is that sometimes an important piece of information is only mentioned in passing when the author clearly knew more about it.
> The mind virus will not stop spreading, making corporations do your critical thinking is not a good path. People will become dependent on a subscription service for everyday life.
Doesn't matter. Even if civilization collapses and we're all miserable, if it means more money is flowing into OpenAI's coffers, it's all good.
Right, that's why they have massive churches adorned with gold and intricate sculptures. Just because it technically isn't required to pay does not mean that years of brainwashing won't condition you to give your money away. I've only been a few times, but seeing old people queue up to give a sizable part of their pension to the church just made me sad.
Evidence for that statement? Can you give some examples?
Mostly when people say "the Bible is not true" its usually a result of misunderstanding it (e.g. adopting Biblical literalism, not understanding the culture and context, not understanding nuance).
No. You interpret each document in context and in culture.
For example, you interpret Genesis as a story that makes a point and tell you something - it is like Jesus's parables (no one same says they are literal!). For example, that all human beings are made in the image of God - as we all look different that is clearly not literal. That we are all related and of one ancestry.
On the other hand you interpret the gospels as deliberately written biographies of Jesus. You interpret the epistles as letter written by their author to a particular person or group of people. You interpret the psalms as lyrics.
It is the traditional way of interpreting the Bible and few people had a problem with it until modern times.
Yes, that is the point of fiction. its not unfair to compare Genesis to Star Wars to an extent, but, to a Christian, what you learn from Genesis is a lot more important (the "word of God" rather than the "word of George Lucas").
However, much of the rest of the Bible should be read differently - the letters, biographies etc. Each document ("book") needs to be read appropriately and in context. Again, each can be compared to others in its genre, but its inclusion in "the Bible" (but there are lots of Biblical canons) gives it that extreme importance.
> It is the traditional way of interpreting the Bible and few people had a problem with it until modern times.
Sorry to nitpick, but there were quite a lot of "heathens" and "witches" who had faced some problems with the traditional interpretations of the Bible before modern times.
What a cheap cop-out to move the goalposts so that only the claims that haven't been disproved yet or are unfalsifiable are meant to be taken literally.
Its a departure from Christian tradition (including early Christians), and it leads to demonstrably false conclusions, and its silly to treat many works of many different genres (myth, chronicles, personal accounts, poetry and lyrics, biographies, and letters) as all being interpreted the same way.
As I understand it, there are parts of France that spent time as parts of Germany and are still somewhat culturally German that do church tax in a similar way - much of what was Alsace-Lorraine (Elsaß-Lothringen).
To be clear: (almost) no one is forced to pay church tax in Germany - only members of the churches that have an agreement with the government to collect it on top of income tax have to pay it, and you can choose to leave those churches. For Protestants ("evangelisch"), that's usually not as big of a deal as it is for Catholics who still believe; there are plenty of non-church-tax-collecting Protestant churches around the country, including the one I'm a member of.
"Almost": there were many couples with very unequal incomes in which the non/lower-earner would stay in the church so that the family would still get the various services (baptisms, weddings, preferential admission to church-affiliated schools, etc) while the higher earner would "leave" (on paper), leaving the family paying far less in church tax. That loophole was closed - if the higher earner isn't a member of another church collecting church tax, they can be required to pay church tax to their spouse's church. I'm not sure this is still in effect, but it was for a while.
In Germany it's not really true. AFAIK you pay those taxes only if you are registered follower of 3 main religions. You literally can opt out, they are a counter example.
Poland is the one I experience it. Church is funded in multiple ways. At least 3 billion PLN a year from concordat deal from 90's. Priests have pensions and annuities. Churches pay no taxes on (heating) fuels. Schools pay for Religious Education classes, very often run by priests or nuns. Uniformed services almost always pay for cleric's services or clerics fully in their services.
Of course church still gathers funds on their own, sometimes using dark patterns.
I think tax breaks are different from direct funding, the same for payment for specific services at a reasonable price. For example the UK exempts virtually all religious bodies from tax, and its on the same basis as a huge range of things (e.g. amateur sports, equality and diversity, community facilities...). I would not consider that state mandated payment for services.
I do not know enough about the concordat or how Polish pensions work to comment on those. I would be interested but there does not seem to be a lot of information online (e.g. the wikipedia article is a stub)
I was thinking of Christianity as I was responding to a comment that used the word "church".
However, besides that, subsidies from general taxation are not the same as payments for a service received (i.e. going back to it being a "subscription service"), whereas something like the German system where the payment is linked to entitlement to services (if other comments here are accurate) can be reasonably characterised as a subscription service.
I disagree with the distinction between subsidies and payments. The math is zero sum, either way purchasing power is undesirably and forcibly reduced from one entity and given to another.
That is not the distinction I am making here. I even partly agree with you (with some nuances).
I am making a distinction between being made to pay through general taxation (e.g. as a pacifist is forced to pay for the military, an extreme libertarian for public services in general) and being made to pay in order to use the service (e.g. like a Netflix subscription). Almost everywhere they exist, subsidies for religion are like the former, not the latter.
Where does the money come from? Religious services are generally funded by donations, and these donations are usually done in the open, whereby (from what I saw) regularly attending and not donating the expected amount would put you in a socially uncomfortable situation.
Back when my parents made me go to church, I remember observing that a lot of the donations being made were in envelopes that the church provided.
That's pretty private, I think, in that one's fellow churchgoers can't discern much but the thickness of the envelope. It'd look the same if the donation consisted of 5 singles, or 5 hundreds.
(I have no idea if that's standard accepted practice everywhere, though I might imagine that someone would be getting pretty uptight at some level if people weren't giving enough to put another layer of gold on the roof of a Catholic church -- envelope or not.)
No reason anyone would would feel social discomfort in my experience, which is mostly in Catholic and Anglican churches, and AFAIK money comes mostly from donations not made in public. I have not felt the least worried about what people would think when I have not had cash on me or about how much I put in.
Depending on the definition of services you are using (e.g. you only mean masses in a Catholic Church, or everything else churches do) lots of things are done without a link to donating: prayers and meditation of other kinds/formats, confession, pastoral care, food banks, religious education and discussion.... In poor countries often things like medical services.
Done the traditional way, no one can really see how much you put in the box and there is no reaction at all from anyone if you put nothing in. Only people right next to you can see anything at all.
Now churches in the UK offer envelopes on which you can write your name and postcode for tax reasons (they can reclaim part of the tax paid on the donation if you are a UK tax payer) so no one can see how much you put in if its in such an envelope.
I do know that, but I also know how donations can become an expectation.
Also, it's worth noting in the context of this thread, that people can use AI inference for free on many services, with payment only need for higher usage, and even then, if you don't care about expectations or inconvenience, it's trivial to abuse the free tier.
A thousand years? What made you go with that number?
The protestant reformation was only about 500 years ago, and I'm pretty sure that Martin Luther wouldn't have bothered that much if the expected "donations" were really cheap. And even if you do go with "a coin", which was apparently the price of an annual indulgence for a regular peasant, that was about the same price as a whole pig, or on the order of $1k in today's money, so definitely not symbolic.
I know that hackers here need to always be right and go to great lengths to try to distort reality when they are wrong. You're not even fooling yourself by saying that every coin in history is worth a thousand dollars, much less fooling anybody else.
If you make it a habit to always lie in order to always be right, you start building castles of lies that hinder you in life. Just because of pride.
I'm sorry if you don't like pedantry, but this is what I'm in HN for.
To be clear, I definitely didn't mean to imply that every coin in history is worth a thousand dollars, and suggesting that this is what I meant is clearly not the "strongest plausible interpretation"[0] of my message. I was referring specifically to the Florin/Gulden/Guilder coins being used across Europe in Martin Luther's time, which contained about 3.5g gold, which at today's gold price would be worth over $500 just as bullion, but it was apparently worth about twice that in terms of purchasing power. From my searches, it seems that the poorest of the poor would need to pay a quarter of a coin annually, the typical commoner would pay 1 per year, and merchants/middle-class would pay 3 or more per year, to eliminate/reduce their afterlife punishment.
You can argue that my focus on indulgences is not relevant for some reason, and I'd be happy to discuss other examples of expectations of monetary payments to the church, but would appreciate if you refrain from accusing me of lying.
Churches have directly taxed their followers on their income. Some of them still do, like the government churches in Scandinavia. That's a tax.
Churches have also sold the redemptions of your sins. Sometimes a bit cloaked as donations, like what you mention.
And churches have accepted donations, with expectations so low that everybody can donate. Who can't donate a kopek, or a bowl of rice? People who are too poor to donate anything are not shunned by any church, on the contrary they will be on the receiving end of donations if they wish to.
I would also like to quote the definite authority on this subject, Mark 12:41-44:
"Jesus sat down opposite the place where the offerings were put and watched the crowd putting their money into the temple treasury. Many rich people threw in large amounts. 42 But a poor widow came and put in two very small copper coins, worth only a few cents.
43 Calling his disciples to him, Jesus said, “Truly I tell you, this poor widow has put more into the treasury than all the others. 44 They all gave out of their wealth; but she, out of her poverty, put in everything—all she had to live on."
Wait till you get AI to write unit tests and tell it the test must pass. After a few rounds it will make the test “assert(true)” when the code cant get the test to pass
