> the paradox is, the LLMs are only useful† if you're Schwartz, and you can't become Schwartz by using LLMs.
That you can't "become Schwartz" by using LLMs is an unproven assumption. Actually, it's a contradiction in the logic of the essay: if Bob managed to produce a valid output by using an LLM at all, then it means that he must have acquired precisely that supervision ability that the essay claims to be necessary.
Btw, note that in the thought experiment Bob isn't just delegating all the work to the LLM. He makes it summarise articles, extract important knowledge and clarify concepts. This is part of a process of learning, not being a passive consumer.
There's no contradiction, the point is that Bob is able to produce valid output using LLMs, but only while he himself is being supervised; and that he doesn't develop the skills to supervise independently himself in the future.
No, this is impossible unless Bob is presenting at each weekly meeting simply the output of the LLM and feeding the tutor's feedback straight into it. For a total of 10 minutes work per week, and the tutor would notice straight away at least for the lack of progress.
No, the article specifies that Bob actually works with the LLM, doesn't just delegate. He asks the agent to summarise, to explain, and to help with bug fixing. You could easily argue that Bob, having such an AI tutor available 24/7, can develop understanding much faster. He certainly won't waste his time with small details of python syntax (though working with a "coding expert" will make his code much cleaner and more advanced).
This is the rub, Bob would not be promoted if he consistently provided unreliable LLM output. In order to get promoted, Bob needs to learn the skills that get reliable output out of an LLM. These may not be the same skills that Alice learns, but if the argument is that Schwartz's LLM output is valuable -- why are we to assume Bob's path isn't towards Schwartz?
There are flowers that look & smell like female wasps well enough to fool male wasps into "mating" with them. But they don't fly off and lay wasp eggs afterwards.
But there is a distinction we can make between flowers and wasps. If there is no distinction we can make between Schwartz and non-Schwartz, then we are susceptible to the sample problem with or without AI. And if there is a distinction then we can use that distinction to test Bob, and make him learn from his test failures.
You mistake for competence his greed and that of those who surround him. I don't think there was a plan to profit from the disaster; rather, they're so incompetent that they even lack the basic self-control to avoid publicly taking advantage of the mess they unwillingly caused, however bad and dangerous that might be.
I imagine Trump wanted to do some fun new things when he is old and will soon die. Its not many who get to experience what it feels like to start a war and kill world leaders, and when you are gonna die soon anyway why not?
I think this is the correct lense. He's a malignant narcissist on his way out, with absolutely nobody to stop him.
I'm genuinely worried that he secretly wants to go down in history as the crazy guy who set the oil fields on fire and dropped a nuke on Tehran or something.
Not sure what moves Trump- could be any of that or more. What we all know is that Netanyahu and Kushner found this and used it to get what they wanted. This is not Trump's war, he's not the initiator and he doesn't have goals of his own (though at times he might believe he does). It actually contradicts what he campaigned on for years.
It makes no sense to me. Trump is clearly desperate for the strait to be opened again (see his last tweet); the US navy is not able to escort anyone through it without taking huge risks; the insurance backstop is useless- almost no ships passed through the strait, period- the ones that do have an agreement with Iran, not the US. Increased oil prices hurt US citizens almost as much as they hurt everyone else. It all sounds like an attempt to make a quagmire look like 4d chess.
What is happening instead is that Iran is making agreements with various countries to let their ships through. These countries stand to lose it all again in case of a US attack, so they have an interest in trying to stop it.
I don't get this. It goes on and on about the conditions of a ceasefire or peace between Iran and the US, as if the US had any actual motives for this war. They do not. The real instigator of the war is Israel, the reason for decades of sanctions is Israel, and the US have already proven that they will renege any agreement if Israel tells them to.
For US motives, it's more about Trump trying to do something for his legacy, he wants to be the one who ended Iran's terror. Obviously not going according to "the plan" and Iran's regime seems to not be going anywhere. There doesn't seem to be any thought of second order effects or what comes after the shooting stops.
Short of regime change, it sounds like this is going to become an exercise in "mowing the lawn" as some people describe it. The irony of course is that it was the US doing regime change that set us on this path.
> For US motives, it's more about Trump trying to do something for his legacy
That's not a US motive, it's a Trump motive. And who used Trump's vanity as a lever? Netanyahu of course. Had it been a different president, he would have used a different lever. Humanitarian reasons, the safety of Israel, you name it. That's what I am saying: it's pointless to talk to the US because the US have no agency anymore.
I don't disagree and will only add that for people outside of the US, these details are irrelevant. The US elected their representative (twice), this is who we are, despite everyone knowing that the entire population did not choose it.
It's the same light we use when talking about Iran and Israel.
> I trust none of us would presume that the decentralized labor of pen & paper calculations somehow instantiated a “psychology”
Wrong. What you've just done is just reformulating the Chinese room experiment coming to the same wrong conclusions of the original proposer. Yes, the entire damn hand-calculated system has a psychology- otherwise you need to assume the brain has some unknown metaphysical property or process going on that cannot be simulated or approximated by calculating machines.
People go for chinese room for some reason when cartesian theater is the better fit here. What you're doing is placing yourself in the seat of the Homunculus waiting for the show to start. But anatomical investigation reveals that there's no theater at all, and in fact no central system where everything comes together. Instead, the whole design of the brain goes to great pains to tease input signals apart.
Basically, manipulating the symbols won't necessarily have any long term influence on your own state. But the variables you've touched on the paper have changed. Demonstrably; because you've written something down.
If you then act on the result of those calculations, as of course many engineers before you have done, and many after you will do; then you have just executed a functional state change in physical reality, no matter what the ivory tower folks say.
(And that's what the paper is about: Functional states)
"Chinese civilian engineers" sounds a lot like "citizen investigative journalism"- i.e. a perfectly deniable outlet to spread information originating from the military or intelligence. That said, go Iran!
Beautiful article. One thing to note though is that humans are often trained in exactly one language (with additional ones, if any, mostly acquired later). This might be relevant for the decoupling between meaning and language- if meaning is expressed in a single language, the two could end up more strongly coupled than in an LLM that has been trained across tens of languages at the same time and was under a stronger pressure to abstract its representations.
If that were the case, it's funny that the argument "it's just a language processor manipulating words" might apply more strongly to humans than to LLMs.
There is also type 3, the worst kind: the developer who enjoys writing code to "get things done", no matter what the requirements are and the quality of the code. Produces tons of spaghetti code that barely works, without much thought about architecture or quality, and duly accompanied by tons of unit tests that test the exact implementation, bugs and all.
It's worth noting that while you are annoyed by this repeated behaviour, for the LLM this is always the first conversation ever. (At least it doesn't have memory of any previous ones).
To the extent that it has any memory at all, it has memory of more conversations than any human could ever have in a single lifetime by way of its training data. That includes tons of conversations with this behavior. That's why the behavior happens in the first place.
That you can't "become Schwartz" by using LLMs is an unproven assumption. Actually, it's a contradiction in the logic of the essay: if Bob managed to produce a valid output by using an LLM at all, then it means that he must have acquired precisely that supervision ability that the essay claims to be necessary.
Btw, note that in the thought experiment Bob isn't just delegating all the work to the LLM. He makes it summarise articles, extract important knowledge and clarify concepts. This is part of a process of learning, not being a passive consumer.
reply