I am sure places like this have their right to exist, and they would probably be a great place to consume… without actively participating. By nature of being strict and exclusionary they will essentially tend towards becoming a relatively small exclusive club perhaps slightly bordering on being an echo chamber. Whether you consider that a bad thing or good, I am not making a judgement call here.
Also the cycle/churn of software is so quick nowadays that manuals are often written as a second or third thought for majority of the projects. So I can not really blame folks for just directly probing for tribal knowledge.
jq is such an underrated tool, it is fully turing complete functional language in and of itself. It opens up an ocean of possibilities in combination with scripting languages (python has bindings), and it is ungodly fast chewing up through tons of JSON like its nothing.
It is a bit perl-ish, but being pure and functional it is a little easier to reason about when you have to revisit your queries.
PS I am certainly bookmarking your tool as well =]
There is also amusing project jqjq that implements jq in jq itself that I love to point folks at to show how expressive the language is: https://github.com/wader/jqjq
I hit some sound issues in the Linux version, for anyone else with the same problem telling Steam to use Proton to run the Windows version instead is a workaround.
My only minor peeve with the sequel so far is that the logistics of moving stuff back to the central platform is pretty fiddly and time-consuming compared to the first one. I think there's an unlockable upgrade I'm nowhere near getting which might fix that though.
Yup, I too had issues with sound, the game seems to just use the first sound interface available, but it is relatively easy to change after starting (via KDE sound panel for me, but I imagine pavucontol should also work). The issue seems to be with Unity/FMOD on linux not the actual game. Newer versions of Unity seem to not have this issue and author was saying they are trying to see if they can upgrade.
PS I was attempting to make a pipewire LUA script that would auto connect the FMOD output to default interface but got stuck on querying and linking audio ports and interfaces, if someone has more Pipewire-Fu I am all ears!
This looks totally awesome! Easy to setup, memory-efficient, streaming, real-time data aggregation, compilable to a single self contained binary, that is a dream come true.
Looks neat! It would be really nice if someone makes a fuse FS that allows exploring/interacting with dbus, in keeping with everything is a file UNIX philosophy.
Out of curiosity, what are your top reasons to pick XML over JSON(+jsonschema) or Msgpack/Protobuf, as data interchange? I have come of age as a professional software engineer around the time when industry has started switching from XML to JSON, and as a consequence in the JSON camp, but I am always curious to hear out folks with a different opinion.
I'm in the same boat, but I found XML has some nice properties that I sometimes miss in JSON, given that XML is used well ("correctly"), such as the differentiation of metadata (attributes) and data (nodes), namespaces, standard query languages, XSLT etc. (You can use XSLT on the web even.)
Think of all the custom, ad-hoc code that turns JSON into HTML vs having a declarative standardized way of doing so.
Great writeup. To add an example, I personally use JSON for most of my work, but have found myself using XML for certain AI use cases that require annotating an original text.
For example, if I wanted an AI to help me highlight to the user where in a body of text I mentioned AI, I might have it return something like:
<text>Great writeup. To add an example, I personally use JSON for most of my work, but have found myself using XML for certain <ai-mention>AI</ai-mention> use cases that require annotating an original text with segments.</text>
Msgpack is broadly fine (I shipped a parser for it a while ago, relatively few ambiguities in the spec). Json is kind of ok if you don't do numbers but schemas for it are a pain and the tooling gives me hacked together vibes.
I like being able to read and edit the data files easily in a text editor (bias against the binary formats) and for there to be a decent chance tools written by other people will interact predictably with the format (so it can't be bespoke).
I'd say the main feature is that an XML document with a schema tells you a lot about the various shapes of the file that you might need to worry about. It's essentially an extensible type system for tree shaped data.
XML has an annoying collection of spurious limitations but that's what you get with lowest common denominator / popular-cos-old systems.
JQ is an insanely powerful language, just to put to rest any of your doubts about what it is capable of here is an implementation of JQ... in JQ itself:
And also for prototyping you can also use it to tailor output of APIs to what you need in a pinch, using JQ as a library especially with something like python:
JQ + journald is great too, but 20 years of muscle memory writing bash / python / perl / awk / sql / ruby / JS / CSS selectors / xpath / xmlstarlet one-liners keep getting in my way. I keep long notes on both with examples of common tasks. I still dislike yaml (significant whitespace is my “ick” as the kids say) too much to learn whatever the equivalent is for that and still find CSV/TSV easier to slice and dice at will due to my own personal history.
I’m sure at this point that many ETL jobs in notebooks we run at $BigCo today could be reduced to jq expressions that run 100x faster and use 1/10th the memory.
The ‘nearly’ Turing complete is something I wonder about. It feels like jq might have some limitations - transformations it can’t do, due to some inherent limitation of how it handles scope or data flow. The esoteric syntax makes it hard to determine sometimes whether what you are attempting is actually possible.
As soon as jq scripts reach a certain level of complexity I break out to writing a node script instead.
And given how rapidly jq scripts acquire complexity, that level is pretty low. One nested lookup, and I’m out.
jq does often feels like a code golf language. I would say it does have some of those Perl one liner vibes, that is to say that it is often a write-only language.
Also the ‘nearly’ part is because I don’t remember if it has infinite loops or if it is more like Starlark and thus decidable. I do have vague recollections of causing infinite cycles in JQ, it quite as well could be entirely Turing complete.
So far I have not found a single task that JQ was incapable of. And I have abused it pretty bad on my spare time =], for intellectual challenge.
jq lacks coroutines, which means some tasks can be hard to accomplish in jq. It's still a very powerful language, and it is Turing complete, not just nearly.
Thank you so much for piecing together a great example (jqjq) to help open everyone’s eyes that JQ is not just a JSONpath implementation with weird syntax! I often reference it to drive home the fact that JQ is a full blown language.
The brainfuck one is also gonna be going into my notes. That implementation is quite a terse implementation.
Great to hear and that was one of my hopes! but honestly it initially came to be because i was fiddling with some jq AST-tree stuff for fq :) weirdly it was much easier to implement than i expected. Hardest part was how to handle infix operators, +/- etc, parsing without infinite recision. But once i found and managed to implement precedence climbing things got a lot easier, it's still a bit of magic to me how well it works :) the eval part had some difficulties but mostly straight forward when you can piggy-back on the "host jq", but i tried to stay away from piggy-back too much, to not piggy-back at all probably requires implement a VM somehow.
BTW your very welcome to help improve the jq documentation. Me and some other maintainers have been talking about that it probably needs an overhaul to be more approachable and also better document some nice hidden features. Join the discord if you want!
This is something very spiritually close to a concept I recently learned about from Cory Doctorow’s talk at DEFCON called Adversarial Interoperability:
As many have noted the reason this functionality is not available is because it invariably leads to commodification of services. Competitors will quickly jump in and use those APIs to interoperate to bootstrap themselves up.
Most likely for consumers that is a huge win, but for companies that is a dangerous filter that filers out companies that stagnate.
This used to be common back in the day and was keeping companies from stagnation, that was until per Cory’s arguments there have been an explosion of laws that basically outlawed Adversarial Interoperability. Which in turn led to too big to fail software giants we have today.
> As many have noted the reason this functionality is not available is because it invariably leads to commodification of services.
Exactly right, and that's why when AI-based voice assistants become wildly capable someday, some of the more brand-based companies will lose a lot of value. I'll be asking Alexa/Siri/Google Assistant for that ride/groceries/etc and as long as it's good, I won't care where it comes from. The (perceived) value will be with the voice assistant, not the fulfilling service.
There's probably a subtle business case for interoperability that you can make from within a company, but I only offer that because it's all we have the power to do ourselves, while we're waiting for regulation to change the economic motivations.
Something that was very popular back in the day, and was a great filter for companies, and something that most of the current giants did to their competitors, before declaring via laws that nobody can do it onto them to secure their market position indefinitely.
This is finally a great article demystifying non-resident keys! I got a FIDO yubikey a few months back, it came with no PIN set up, I was surprised that I could just use it like that, websites were not asking me to set it up explicitly.
I set it up via CLI. Then all websites started asking for the pin and storing the keys inside (resident keys).
Knowing that you only have limited slots, I tried deleting the pin by resetting yubikey to revert back to old behavior as there was no way of zeroizing the pin any other way (for the sake experimenting with the tech and understanding key management better, ofc pin or BIO auth is better).
And then suddenly found that I was unable to use the key to auth into my accounts I set up before. In hindsight now I know by resetting the hardware key I have reset the master key inside.
There really isn’t much well written introductions into how all of this works. Thank you for doing a great job demystifying the flow here!
Also the cycle/churn of software is so quick nowadays that manuals are often written as a second or third thought for majority of the projects. So I can not really blame folks for just directly probing for tribal knowledge.