Strange article. The problem isn’t that everyone doesn’t know how everything works, it’s that AI coding could mean there is no one who knows how a system works.
No I think the problem is AI coding removes intentionality. And that introduces artifacts and connections and dependencies that shouldn’t be there if one had designed the system with intent. And that makes it eventually harder to reason about.
There is a difference in qualia in it happens to work and it was made for a purpose.
Business logic will strive more for it happens to work as a good enough.
The core problem is irresponsibility. Things that happen to work may stop working, or be revealed to have terrible flaws. Who is responsible? What is their duty of care?
Excellent point. The intention of business is profit, how it arrives there is considered incidental. Any product no matter what, as long as it sells. Compounding effects in computing, the internet and miniaturisation, have enabled large profit margins that further compound these effects. They think of this as a machine that can keep on printing more money and subsuming more and more as software and computers are pervasive.
Including the AI, which generated it once and forgot.
This is going to be a big problem. How do people using Claude-like code generation systems do this? What artifacts other than the generated code are left behind for reuse when modifications are needed? Comments in the code? The entire history of the inputs and outputs to the LLM? Is there any record of the design?
I have experimented with telling Claude Code to keep a historical record of the work it is performing. It did work (though I didn't assess the accuracy of the record) but I decided it was a waste of tokens and now direct it to analyze the history in ~/.claude when necessary. The real problem I was solving was making sure it didn't leave work unfinished between autocompacts (eg crucial parts of the work weren't performed and instead there are only TODO comments). But I ended up solving that with better instructions about how to break down the plan into bite-sized units that are more friendly to the todo list tool.
I have prompting in AGENTS.md that instructs the agent to update the relevant parts of the project documentation for a given change. The project has a spec, and as features get added or reworked the spec gets updated. If you commit after each session then the git history of the spec captures how the design evolves. I do read the spec, and the errors I've seen so far are pretty minor.
Is that documentation useful? I haven't seen a well-documented codebase by AI so far.
To be fair - humans also fail at that. Just look at the GTK documentation as an example. When you point that out, ebassi may ignore you because criticism is unwanted; and the documentation will never improve, meaning they don't want new developers.
I've been coding for 40 years (23 years at an S&P500) and these coding agents write better documentation than I've ever seen from my peers. You just need to work with it and not expect it to do 100% of the work in one shot.
I for one I save all conversations in the codebase. Includes both human prompts and outputs. But I’m using a modified codex to do so.
Not sure why it’s not default as it’s useful to have this info.
I am sure engineers collectively understand how the entire stack works.
With LLM generated output, nobody understands how anything works, including the very model you just interacted with -- evident in "you are absolutely correct"
Even as a collective whole, engineers will likely only understand the parts of system that are engineering problems and solutions. Even if they could understand it all, there is still no _one_ person who understands how everything works.
Just because there is someone who could understand a given system, that doesn’t mean there is anyone who actually does. I take the point to be that existing software systems are not understood by anyone most of the time.
I do not know about you all, but I need to understand the system before I can change anything, otherwise I would introduce tons of bugs. Heck, without knowing the system I do not even know what I *want* to change.
This happens even today. If a knowledgeable person leaves a company and no KT (or more likely, poor KT) takes place, then there will be no one left to understand how certain systems work. This means the company will have to have a new developer go in and study the code and then deduce how it works. In our new LLM world, the developer could even have an LLM construct an overview for him/her to come up to speed more quickly.
Yes but every time the "why" is obscured perhaps not completely because there's no finished overview or because the original reason cannot be derived any longer from the current state of affairs. Its like the movie memento: you're trying to piece together a story from fragments that seem incoherent.
I haven't been writing Rust for that long (about 2 years) but every time I see .unwrap() I read it as 'panic in production'. Clippy needs to have harder checks on unwrap.
To be fair the performance of rules or Bayesian networks or statistical models wasn't the problem (performance compared to existing practice). DeDombal showed in 1972 that a simple Bayes model was better than most ED physicians in triaging abdominal pain.
The main barrier to scaling was workflow integration due to lack of electronic data, and if it was available, interoperability (as it is today). The other barriers were problems with maintenance and performance monitoring, which are still issues today in healthcare and other industries.
I do agree the 5th Generation project never made sense, but as you point out they had developed hardware to accelerate Prolog and wanted to show it off and overused the tech. Hmmm, sounds familiar...
The paper of Ueda they cite is so lovely to read, full of marvelous ideas:
Ueda K. Logic/Constraint Programming and Concurrency: The hard-won lessons of the Fifth Generation Computer project. Science of Computer Programming. 2018;164:3-17. doi:10.1016/j.scico.2017.06.002 open access: https://linkinghub.elsevier.com/retrieve/pii/S01676423173012...
Unfortunately, I think the context rot paper [1] found that the performance degradation when context increased still occurred in models using attention sinks.
I've been using Gemini 2.5 and Claude 3.7 for Rust development and I have been very impressed with Claude, which wasn't the case for some architectural discussions where Gemini impressed with it's structure and scope. OpenAI 4.5 and o1 have been disappointing in both contexts.
Gemini doesn't seem to be as keen to agree with me so I find it makes small improvements where Claude and OpenAI will go along with initial suggestions until specifically asked to make improvements.
I have noticed Gemini not accepting an instruction to "leave all other code the same but just modify this part" on a code that included use of an alpha API with a different interface than what Gemini knows is the correct current API. No matter how I promoted 2.5 pro, I couldn't get it to respect my use of the alpha API, it would just think I must be wrong.
So I think patterns from the training data are still overriding some actual logic/intelligence in the model. Or the Google assistant fine-tuning is messing it up.
I have been using gemini daily for coding for the last week, and I swear that they are pulling levers and A/B testing in the background. Which is a very google thing to do. They did the same thing with assistant, which I was a pretty heavy user of back in the day (I was driving a lot).
When do you turn it off? I have a Mac M1 Studio and I just let it sleep. If things get weird I reboot. I think I recall using the power button about a year ago after returning from vacation after I had shut it down.
Right now I mount up to 7 HDDs to the Mac via SMB, have some Streamdeck / Pedal and the necessary external SSDs for fast storage connected. I will see if the SMB mounts come back OK after sleep (my laptop acts as server) but the Streamdeck and HDDs wake up randomly so overall it's easier to switch everything on and off depending on usage.
Everyone keeps citing idle, which is when the device is on and active but not particularly doing anything.
The standby power draw is 1W or less. I've used Mac Minis for years -- just replaced my M1 with an M4, though the M1 left me wanting for nothing -- and the number of times I've interacted with the power button is so negligible I imagine I've gone over a year without touching it. When I haven't touched it in a while it goes to standby, waking instantly when I engage it again.
Not everyone lives the same way. I am seriously considering a Mac Mini as my next upgrade yet I live in a RV and move frequently. Are there ways that I can keep the Mac mini powered while traveling.. sure, but why would/should I?
Are you not turning off entire circuits to reduce power draw when mobile? I’m actually thinking about one of these for my truck camper and its power draw seems fine, but the stumbling point for me is the additional power draw from the monitor it would require. I think I’m leaning toward an M4 MBP with nano textured screen for maximum power efficiency and ability to work outside when it’s nice, though I have not yet put much effort into researching efficient monitors
My EU mind is blown by these claims. Let’s take the lowest(1W) at sleep mode. With a thousand mac minis at sleep mode, that is already 1kW! In my country, a single person household’s yearly electricity package comes at 1400kW(+100 depending on provider) per year.
Note: intentionally keeping it simple, please don’t nitpick.
No household uses 1400kW, and kW/year doesn't make sense. Do you mean 1400kWh/year? That seems pretty low (NZ is 7000kWh/year), but if so, you're comparing power to energy, which doesn't mean much. 1W 24/7 < 9kWh/year, which is pretty small.
Personal guess from a fellow European citizen: I think they meant to say 1700 kWh/year. According to most German power utilities, the average 2-person household consumes about 2400 kWh/year.
I suspect it's not so much the "expert" instruction but the list of subjects of the expertise. These words will generate embeddings that have a better chance of activating useful pathways and relationships within the LLM's generation path.
My understanding is the goal is to prime the LLM with context that it will use when generating answers, rather than hoping it will infer connections in the feed forward layers when generating answers based on a sparse prompt.
