Let's say we could somehow train an LLM on all written and spoken language from the western Roman civilization (Republic + Western Empire, up until 476 AD/CE, just so I don't muddy the experiment with near-modern timelines). Would it, without novel information from humans, ever be able to spit out a correct predecessor of modern science like atomic theory? What about steam power, would that be feasible since Romans were toying with it? How far back do we have to go on the tech tree for such an LLM be able to "discover" something novel or generate useful new information?
My thought is that the LLM would forever be "stuck" in the knowledge of the era it was trained in. Something in the complexity of human brains working together is what drives new information. We can continue training new LLMs with new information, and LLMs might be able to find new patterns in data that humans can't see and can augment our work, but the LLM's capability for novelty is stuck on a complexity treadmill, rooted in its training data.
I don't view this ability of humans as some magic consciousness, just a system so complex to us right now that we can't fully understand or re-create it. If we're stochastic parrots, we seem to be ones that are magnitudes more powerful and unpredictable than current LLMs, and maybe even constructed in a way that our current technology path can't hope to replicate.
For your thought experiment, I'd assert that the key missing part is experimentation in the real world, as that is what acquires new information, not the complexity of human brains working together.
If you took millions of genius-level immortal humans with all the same Roman data but had them sit in a blank, empty room with their hands tied and simply discuss philosophy for eternity, I'm certain that they would not be able ever spit out a correct predecessor of modern science like atomic theory. Perhaps they could spit out billions of theories including the atomic theory as well, but they would have no data to presume that the atomic theory is more relevant than any other. Extensive information processing can squeeze out every last ounce of knowledge from some data, but anything that isn't in that data can't be acquired by mere thinking about it. On the other hand, if you gave some "LLM++" the ability to toy around with reality and attempt all kinds of experiments to test various hypotheses, then I wouldn't assume that it would not be forever stuck in the knowledge of the era it was trained in.
Yeah, I like that improvement/clarification. Good assertion. Now I wonder if it changes my stance: are the path modern LLMs are on ever going to replicate this environment for acquiring new information that humans currently operate in?
> Let's say we could somehow train an LLM on all written and spoken language from the western Roman civilization (Republic + Western Empire, up until 476 AD/CE, just so I don't muddy the experiment with near-modern timelines). Would it, without novel information from humans, ever be able to spit out a correct predecessor of modern science like atomic theory?
Yeah, I'm in the same boat. I've used YNAB for almost a decade now, but wish it were more powerful for things like investment tracking. I tried last month to get into the ledger systems (beancount, specifically), but it's a huge additional time sink. And implementing YNAB-style envelope-based budgeting on top of them is always a bit of a hack. Went back to YNAB quickly.
Cannot wait for my Intel MBP to be refreshed into an M2 at work. Running Docker turns it into a jet engine. Like having a white noise machine with an unpleasant tone that also generates a ton of heat.
I went from a Core i9 to an M1 Max and it's been a world of difference. Performance is great, it runs my dual 4K screens with scaling (3008x1692, virtual resolution 6016x3384) without a sweat, and the battery lasts forever.
Moreso than the noise, the touchbar at the back can become too hot to touch and the plastic strip around the screen, where MacBook Pro is printed, is bubbling and peeling away with the heat.
I love how apple considers my skin part of the cooling solution. If it wasn't for legal limits, they would absolutely just burn people, sell a laptop cooling mat for $100, and tell people they are using laptops wrong.
Yep, when I realized I was going deep down the money rabbit-hole of fine-tuning my homelab to grab and serve this stuff is when I just switched back to streaming. Monetarily it'll probably be a wash in the long run for me.
Huh, I had an idea recently to make an API-only game like that. Would let people build their own tools/GUIs and go as crazy as they'd like with building their own AI to control their empire. Just feels extremely niche, even if it scratches my own gameplay itch.
I'm concerned that such a game would be so much more rewarding to me as a developer than my actual software development job is, that I would become miserable at work!
Even with my racing sim rig, I find myself using my curved monitor more than the Quest 2. It's just such a pain to re-calibrate and update the software and tweak my graphics settings every time I want to hop in that I usually just don't bother.
I was thinking the same thing. I know the price points are way different between what Apple's putting forward and what Meta was shipping, but Apple's vision and tech here blows Meta's out of the water. Meta seemed to be approaching the problem with an iterative approach, where the payoff in vision was down the road, and wanted consumers to share the journey to get there - whereas Apple jumped all the way to the end.
When I was house-hunting I ended up writing a console app for my wife and I to do a few things:
- Pulled down applicable YNAB savings envelope balances and future income calculations from a Google Sheets spreadsheet (which included stock prices for determining RSU payouts) to know how much cash we'd have for down payments at any time in the next 12 months
- Allowed us to either give a house price and have it output when we could afford it, or give a month and tell us how much we could afford if we bought on that month
- Do budgetary analysis of what the monthly payment would be, given fluctuating mortgage rates and estimated insurance from scraping Zillow/Redfin
- Calculated transit times to my office and my wife's office using Google Maps
- Allowed for swappable "scenarios" for all the above to show what would happen if we wanted to sell our current place first and then buy, buy and then sell, or buy-renovate-sell, so we could evaluate all options. We ended up going the buy-renovate-sell path in reality, and it was a huge stress relief to have hard numbers showing us the money was going to be fine.
- Output several months of cash reserves for each scenario after all transactions were done, so we could know if we would cut too much into savings
- Output a yes/no decision based on all of the above to keep us grounded and help prevent over-reaching for a house we couldn't afford - basically enforcing rules on ourselves
Any chance you could share this? I made something similar for an investment decision once and i just discovered the realtor i worked with uses it all the time with his clients. I feel like others might like what you’re doing.
I just put pause on a project because i couldn’t figure out a revenue model: mappedby.com
The "hack" that a couple of the comments in this chain are referring to is only ever taking cabs _from_ a major airport into a city. Often you don't have to play the waiting-on-someone-to-show-up game, there's just a line of them right outside the airport that you can hop in immediately. In some cities there's even flat rates they have to adhere to, depending on where you're going ("$20 to midtown").
I was gleeful when Uber and Lyft came on the scene to have an alternative to cab BS, but the one place they still shine is when going from an airport into a major metro area.
Let's say we could somehow train an LLM on all written and spoken language from the western Roman civilization (Republic + Western Empire, up until 476 AD/CE, just so I don't muddy the experiment with near-modern timelines). Would it, without novel information from humans, ever be able to spit out a correct predecessor of modern science like atomic theory? What about steam power, would that be feasible since Romans were toying with it? How far back do we have to go on the tech tree for such an LLM be able to "discover" something novel or generate useful new information?
My thought is that the LLM would forever be "stuck" in the knowledge of the era it was trained in. Something in the complexity of human brains working together is what drives new information. We can continue training new LLMs with new information, and LLMs might be able to find new patterns in data that humans can't see and can augment our work, but the LLM's capability for novelty is stuck on a complexity treadmill, rooted in its training data.
I don't view this ability of humans as some magic consciousness, just a system so complex to us right now that we can't fully understand or re-create it. If we're stochastic parrots, we seem to be ones that are magnitudes more powerful and unpredictable than current LLMs, and maybe even constructed in a way that our current technology path can't hope to replicate.