This has been my experience too. That insane race condition inside the language runtime that is completely inscrutable? Claude one-shots it. Ask it to work on that same logic to add features and it will happily introduce race conditions that are obvious to an engineer but a local test will never uncover.
I have a couple of EB6s to show to my students but ever since scientific calculators were allowed in written tests, I have never used one myself. Law of cosines is good enough for wind triangles :). Worked for a commercial test as well as the ATP. It is a beautiful device though ...
Rclone is a fantastic tool, but my favorite part of it is actually the underyling FS library. I've started baking Rclone FS into internal Go tooling and now everything transparently supports reading/writing to either local or remote storage. Really great for being able to test data analysis code locally and then running as batch jobs elsewhere.
What kind of data analysis do you run with Go and do you use an open source library for it? My experience with stats libraries in Go has been lukewarm so far.
“The PDF Association operates under a strict principle—any new feature must work seamlessly with existing readers” followed by introducing compression as a breaking change in the same paragraph.
All this for brotli… on a read-many format like pdf zstd’s decompression speed is a much better fit.
yup, zstd is better. Overall use zstd for pretty much anything that can benefit from a general purpose compression. It's a beyond excellent library, tool, and an algorithm (set of).
Brotli w/o a custom dictionary is a weird choice to begin with.
Brotli makes a bit of sense considering this is a static asset; it compresses somewhat more than zstd. This is why brotli is pretty ubiquitous for precompressed static assets on the Web.
That said, I personally prefer zstd as well, it's been a great general use lib.
I made a table because I wanted to test more files, but almost all PDFs I downloaded/had stored locally were already compressed and I couldn't quickly find a way to decompress them.
Brotli seemed to have a very slight edge over zstd, even on the larger pdf, which I did not expect.
EDIT: Something weird is going on here. When compressing zstd in parallel it produces the garbage results seen here, but when compressing on a single core, it produces result competitive with Brotli (37M). See: https://news.ycombinator.com/item?id=46723158
Turns out that these numbers are caused by APFS weirdness. I used 'du' to get them which reports the size on disk, which is weirdly bloated for some reason when compressing in parallel. I should've used 'du -A', which reports the apparent size.
Here's a table with the correct sizes, reported by 'du -A' (which shows the apparent size):
I wasn't sure. I just went in with the (probably faulty) assumption that if it compresses to less than 90% of the original size that it had enough "non-randomness" to compare compression performance.
Ran the tests again with some more files, this time decompressing the pdf in advance. I picked some widely available PDFs to make the experiment reproducable.
For learnopengl.pdf I also tested the decompression performance, since it is such a large file, and got the following (less surprising) results using 'perf stat -r 5':
The conclusion seems to be consistent with what brotli's authors have said: brotli achieves slightly better compression, at the cost of a little over half the decompression speed.
maybe that was too strongly worded but there was an expectation for zstd to outperform. So the fact it didnt means the result was unexpected. i generally find it helpful to understand why something performs better than expected.
Isn't zstd primarily designed to provide decent compression ratios at amazing speeds? The reason it's exciting is mainly that you can add compression to places where it didn't necessarily make sense before because it's almost free in terms of CPU and memory consumption. I don't think it has ever had a stated goal of beating compression ratio focused algorithms like brotli on compression ratio.
I actually thought zstd was supposed to be better than Brotli in most cases, but a bit of searching reveals you're right... Brotli, especially at the highest compression levels (10/11), often exceeds zstd at the highest compression levels (20-22). Both are very slow at those levels, although perfectly suitable for "compress once, decompress many" applications which the PDF spec is obviously one of them.
Are you sure? Admittedly I only have 1 PDF in my homedir, but no combination of flags to zstd gets it to match the size of brotli's output on that particular file. Even zstd --long --ultra -22.
If that's about using predefined dictionaries, zstd can use them too.
If brotli has a different advantage on small source files, you have my curiosity.
If you're talking about max compression, zstd likely loses out there, the answer seems to vary based on the tests I look at, but it seems to be better across a very wide range.
No, it's literally just compressing small files without training zstd dict or plugging external dictionaries (not counting the built-in one that brotli has). Especially for English text, brotli at the same speed as zstd gives better results for small data (in kilobyte to a few of megabyte range).
It's correct use of Pareto, short for Pareto frontier, if the claim being made is "for every needed compression ratio, zstd is faster; and for every needed time budget, zstd is faster". (Whether this claim is true is another matter.)
brotli is ubiquitous because Google recommends it. While Deflate definitely sucks and is old, Google ships brotli in Chrome, and since Chrome is the de facto default platform nowadays, I'd imagine it was chosen because it was the lowest-effort lift.
Nevertheless, I expect this to be JBIG2 all over again: almost nobody will use this because we've got decades of devices and software in the wild that can't, and 20% filesize savings is pointless if your destination can't read the damn thing.
On max compression "--ultra -22", zstd is likely to be 2-4% less dense (larger) on text alike input. While taking over 2x times times to compress. Decompression is also much faster, usually over 2x.
Note the language: "You're not creating broken files—you're creating files that are ahead of their time."
Imagine a sales meeting where someone pitched that to you. They have to be joking, right?
I have no objection to adding Brotli, but I hope they take the compatability more seriously. You may need readers to deploy it for a long time - ten years? - before you deploy it in PDF creation tools.
You're absolutely right! It's not just an inaccurate slogan—it's a patronizing use of artificial intelligence. What you're describing is not just true, it's precise.
Every TCP proxy (that doesn't thwart this) is detectable :)
Countermeasure: pick some min-RTT >= the actual client RTT (you can do this as a TCP proxy by measuring client ping). Measure server RTT and artificially delay responses to be >= min-RTT. This will require an added delay during the handshake and ACKs, but no added delay for the response payloads.
Counter-countermeasure: the above may lead to TCP message types that don't make sense given a traditional TCP client state machine (e.g., delayed ACK would bundle ACK and PUSH but the system shows separate/simultaneous ACK and PUSH packets. Counter-counter-countermeasure is left to the reader.
I think you could also compare with TLS handshake timings, delay for client hello among other things. And you could also compare it with HTTP RTT, not to mention that you can do TCP fingerprinting and compare it with the TLS and HTTP fingerprint of the browser, you can also measure the IP TTL and ping, among many other things... What I mean is that there are a ton of things that can be done on both sides, but any company with enough people working at this and enough servers will surely make something miles away from my proof of concept, and they also have a lot of traffic to know what's baseline data and what isn't.
It’s also a little silly for the same reasons discussions of theoretical computability often are: time and space requirements. In practice the Universe, even if computable, is so complex that simulating it would require far more compute than physical particles and far more time than remaining until heat death.
Hehe yeah.. For me, its just inverted search for the God. There must be somethink behind it, if its not God, then it must be simulation! Kinda sad, I would expect more from scientist.
The big riddle of Universe is, how all that matter loves to organize itself, from basic particles to Atoms, basic molecues, structured molecues, things and finally live.. Probably unsolvable, but that doesnt mean we shouldnt research and ask questions...
>The big riddle of Universe is, how all that matter loves to organize itself, from basic particles to Atoms, basic molecues, structured molecues, things and finally live.. Probably unsolvable, but that doesnt mean we shouldnt research and ask questions...
Isn't that 'just' the laws of nature + the 2nd law of thermodynamics? Life is the ultimate increaser of entropy, because for all the order we create we just create more disorder.
Conway's game of life has very simple rules (laws of nature) and it ends up very complex. The universe doing the same thing with much more complicated rules seems pretty natural.
Yeah, agreed. The actual real riddle is consciousness. Why does it seems some configurations of this matter and energy zap into existence something that actually (allegedly) did not exist in its prior configuration.
I'd argue that it's not that complicated. That if something meets the below five criteria, we must accept that it is conscious:
(1) It maintains a persisting internal model of an environment, updated from ongoing input.
(2) It maintains a persisting internal model of its own body or vehicle as bounded and situated in that environment.
(3) It possesses a memory that binds past and present into a single temporally extended self-model.
(4) It uses these models with self-derived agency to generate and evaluate counterfactuals: Predictions of alternative futures under alternative actions. (i.e. a general predictive function.)
(5) It has control channels through which those evaluations shape its future trajectories in ways that are not trivially reducible to a fixed reflex table.
This would also indicate that Boltzmann Brains are not conscious -- so it's no surprise that we're not Boltzmann Brains, which would otherwise be very surprising -- and that P-Zombies are impossible by definition. I've been working on a book about this for the past three years...
If you remove the terms "self", "agency", and "trivially reducible", it seems to me that a classical robot/game AI planning algorithm, which no one thinks is conscious, matches these criteria.
How do you define these terms without begging the question?
If anything has, minimally, a robust spatiotemporal sense of itself, and can project that sense forward to evaluate future outcomes, then it has a robust "self."
What this requires is a persistent internal model of: (A) what counts as its own body/actuators/sensors (a maintained self–world boundary), (B) what counts as its history in time (a sense of temporal continuity), and (C) what actions it can take (degrees of freedom, i.e. the future branch space), all of which are continuously used to regulate behavior under genuine epistemic uncertainty. When (C) is robust, abstraction and generalization fall out naturally. This is, in essence, sapience.
By "not trivially reducible," I don't mean "not representable in principle." I mean that, at the system's own operative state/action abstraction, its behavior is not equivalent to executing a fixed policy or static lookup table. It must actually perform predictive modeling and counterfactual evaluation; collapsing it to a reflex table would destroy the very capacities above. (It's true that with an astronomically large table you can "look up" anything -- but that move makes the notion of explanation vacuous.)
Many robots and AIs implement pieces of this pipeline (state estimation, planning, world models,) but current deployed systems generally lack a robust, continuously updated self-model with temporally deep, globally integrated counterfactual control in this sense.
If you want to simplify it a bit, you could just say that you need a robust and bounded spatial-temporal sense, coupled to the ability to generalize from that sense.
The zombie intuition comes from treating qualia as an "add-on" rather than as the internal presentation of a self-model.
"P-zombie" is not a coherent leftover possibility once you fix the full physical structure. If a system has the full self-model (temporal-spatial sense) / world-model / memory binding / counterfactual evaluator / control loop, then that structure is what having experience amounts to (no extra ingredient need be added or subtracted).
I hope I don't later get accused of plagiarizing myself, but let's embark on a thought experiment. Imagine a bitter, toxic alkaloid that does not taste bitter. Suppose ingestion produces no distinctive local sensation at all – no taste, no burn, no nausea. The only "response" is some silent parameter in the nervous system adjusting itself, without crossing the threshold of conscious salience. There are such cases: Damaged nociception, anosmia, people congenitally insensitive to pain. In every such case, genetic fitness is slashed. The organism does not reliably avoid harm.
Now imagine a different design. You are a posthuman entity whose organic surface has been gradually replaced. Instead of a tongue, you carry an in‑line sensor which performs a spectral analysis of whatever you take in. When something toxic is detected, a red symbol flashes in your field of vision: “TOXIC -- DO NOT INGEST.” That visual event is a quale. It has a minimally structured phenomenal character -- colored, localized, bound to alarm -- and it stands in for what once was bitterness.
We can push this further. Instead of a visual alert, perhaps your motor system simply locks your arm; perhaps your global workspace is flooded with a gray, oppressive feeling; perhaps a sharp auditory tone sounds in your private inner ear. Each variant is still a mode of felt response to sensory information. Here's what I'm getting at with this: There is no way for a conscious creature to register and use risky input without some structure of "what it is like" coming along for the ride.
I have more or less the same views, although I can’t formulate them half as well as you do. I would have to think more in depth about those conditions that you highlighted in the GP; I’d read a book elaborating on it.
I’ve heard a similar thought experiment to your bitterness one from Keith Frankish:
You have the choice between two anesthetics. The first one suppresses your pain quale, meaning that you won’t _feel_ any pain at all. But it won’t suppress your external response: you will scream, kick, shout, and do whatever you would have done without any anesthetic. The second one is the opposite: it suppresses all the external symptoms of pain. You won’t budge, you’ll be sitting quiet and still as some hypothetical highly painful surgical procedure is performed on you. But you will feel the pain quale completely, it will all still be there.
I like it because it highlights the tension in the supposed platonic essence of qualia. We can’t possibly imagine how either of these two drugs could be manufactured, or what it would feel like.
Would you classify your view as some version of materialism? Is it reductionist? I’m still trying to grasp all the terminology, sometimes it feels there’s more labels than actual perspectives.
> The zombie intuition comes from treating qualia as an "add-on" rather than as the internal presentation of a self-model.
Haven't you sort of smuggled a divide back into the discussion? You say "internal presentation" as though an internal or external can be constructed in the first place without the presumption of a divided off world, the external world of material and the internal one of qualia. I agree with the concept of making the quale and the material event the same thing, (isn't that kinda like Nietzsche's wills to power?), but I'm not sure that's what you're trying to say because you're adding a lot of stuff on top.
That is not what a p-zombie is. The p-zombie does not have any qualia at all. If you want to deny the existence of qualia, that's one way a few philosophers have gone (Dennett), but that seems pretty ridiculous to most people.
1. Qualia exist as something separate from functional structure (so p-zombies are conceivable)
2. Qualia don't exist at all (Dennett-style eliminativism)
But I say that there is a third position: Qualia exist, but they are the internal presentation of a sufficiently complex self-model/world-model structure. They're not an additional ingredient that could be present or absent while the functional organization stays fixed.
To return to the posthuman thought experiment, I'm not saying the posthuman has no qualia, I'm saying the red "TOXIC" warning is qualia. It has phenomenal character. The point is that any system that satisfies certain criteria and registers information must do so as some phenomenal presentation or other. The structure doesn't generate qualia as a separate byproduct; the structure operating is the experience.
A p-zombie is only conceivable if qualia are ontologically detachable, but they're not. You can't have a physicalism which stands on its own two feet and have p-zombies at the same time.
Also, it's a fundamentally silly and childish notion. "What if everything behaves exactly as if conscious -- and is functionally analogous to a conscious agent -- but secretly isn't?" is hardly different from "couldn't something be H2O without being water?," "what if the universe was created last Thursday with false memories?," or "what if only I'm real?" These are dead-end questions. Like 14-year-old-stoner philosophy: "what if your red is ackshuallly my blue?!" The so-called "hard problem" either evaporates in the light of a rigorous structural physicalism, or it's just another silly dead-end.
You have first-person knowledge of qualia. I'm not really sure how you could deny that without claiming that qualia doesn't exist. You're claiming some middle ground here that I think almost all philosophers and neuroscientists would reject (on both sides).
> "couldn't something be H2O without being water?," "what if the universe was created last Thursday with false memories?," or "what if only I'm real?" These are dead-end questions. Like 14-year-old-stoner philosophy: "what if your red is ackshuallly my blue?!"
These are all legitimate philosophical problems, Kripke definitively solved the first one in the 1970s in Naming and Necessity. You should try to be more humble about subjects which you clearly haven't read enough about. Read the Mary's room argument.
> You have first-person knowledge of qualia. I’m not sure how you could deny that...
I don't deny that. I explicitly rely on it. You must have misunderstood... My claim is not:
1) "There are no qualia"
2) "Qualia are an illusion / do not exist"
My claim is: First-person acquaintance does not license treating qualia as ontologically detachable from the physical/functional. I reject the idea that experience is a free-floating metaphysical remainder that can be subtracted while everything else stays fixed. At root it's simply a necessary form of internally presented, salience-weighted feedback.
> This middle ground would be rejected by almost all philosophers and neuroscientists
I admit that it would be rejected by dualists and epiphenomenalists, but that's hardly "almost all."
As for Mary and her room: As you know, the thought experiment is about epistemology. At most it shows that knowing all third-person facts doesn’t give you first-person acquaintance. It is of little relevance, and as a "refutation" of physicalism it's very poor.
There is no objective evidence of anything at all.
It all gets filtered through consciousness.
"Objectivity" really means a collection of organisms having (mostly) the same subjective experiences, and building the same models, given the same stimuli.
Given that less intelligent organisms build simpler models with poorer abstractions and less predictive power, it's very naive to assume that our model-making systems aren't similarly crippled in ways we can't understand.
That's a hypothesis but the alternate hypothesis that consciousness is not well defined is equally valid at this point. Occam's razor suggests consciousness doesn't exist since it isn't necessary and isn't even mathematically or physically definable.
Yeah I guess... But such question is not really intereseting.. Answer is simple, there is nothing behind it.. and people arent confortable with that answer. Hence "How" is more interesting and scientific..
Yes, is that (obvious) point being addressed in the paper? At first skimming, it just says that a "sufficiently souped up laptop" could, in principle, compute the future of the universe (i.e. Laplace's daemon), but I haven't seen anything about the subsequent questions of time scales.
The real universe might be different and far more complex than our simulated reality. Maybe a species that can freely move within 4 or 5 dimensions is simulating our 3D + uni directional time reality just like we „simulate“ reality with Sim City and Sims.
You're predicating on particles, heat death, etc as you understand it being applicable to any potential universe. Such rules are only known to apply in this universe.
A universe is simply a function, and a function can be called multiple times with the same/different arguments, and there can be different functions taking the same or different arguments.
The issue with that in terms of the simulation argument, is that the simulation argument doesn't require a complete simulation in either space or time.
It also doesn't require a super-universe with identical properties and constraints.
There's no guarantee their logic is the same as our logic. It needs to be able to simulate our logic, but that doesn't mean it's defined or bound by it.
