Thank you for the shout out! I looked into your benchmark setup a bit. Two things going on:
- Ladybug by default allocates 80% of the physical memory to the buffer pool. You can limit it. This wasn't the main reason.
- Much of the RSS is in ladybug native memory connected to the python connection object. I noticed that you keep the connection open between benchmark runs. For whatever reason, python is not able to garbage collect the memory.
We ran into similar lifetime issues with golang and nodejs bindings as well. Many race conditions where the garbage collector releases memory while another thread still has a reference to native memory. We now require that the connection be closed for the memory to be released.
I am actually only vaguely familiar and I was wondering about that every time I saw the format referenced but never bothered to check, your comment is informative!
Yeah, I'm a long time user/disciple of https://zim-wiki.org ; it was basically Obsidian but 15-20 years early. To do some of the things that are now trivially easy with Obsidian I learned scripting and such, so I'm familiar with this very weird coincidence/name collision.
I really curious about what the world of archival formats is like - is there consensus? are the most-used formats actually any good and well-supported,and self documenting?
This is the same topic I had an intense argument with my coworkers at the company formerly called FB a decade ago. There is a belief that most joins are 1-2 deep. And that many hop queries with reasoning are rare and non-existent.
I wonder how you reconcile the demand for LLMs with multihop reasoning with the statement above.
I think a lot what is stated here is how things work today and where established companies operate.
The contradictions in their positions are plain and simple.
I maintain LadybugDB which implements WCOJ (inherited from the KuzuDB days). So I don't disagree with the idea. Just that it's a graph database with relational internals and some internal warts that makes it hard to compose queries. Working on fixing them.
Also an important test is the check on whether it's WCOJ on top of relational storage or is the compressed sparse row (CSR) actually persisted to disk. The PGQ implementations don't.
There are second order optimizations that LLMs logically implement that CSR implementing DBs don't. With sufficient funding, we'll be able to pursue those as well.
CSR is an array-based trie hence very costly to update. It can serve as an index for parts of the graph that basically will almost never change, but not otherwise.
For starters, LLMs themselves are a graph database with probabilistic edge traversal.
Some apps want it to be deterministic.
I'm surprised this question comes up so often.
It's mainly from the vector embedding camp, who rightfully observe that vector + keyword search gets you to 70-80% on evals. What is all this hype about graphs for the last 20-30%?
It comes from people who develop LLMs. Anthropic and Google. References below.
My other favorite quote: transformers are GNNs which won the hardware lottery.
Longer form at blog.ladybugmem.ai
You want to believe that everything probabilistic has more value and determinism doesn't? Or that the world is made up of tabular data? You have a lot of company.
The other side of the argument I believe has a lot of money.
That importing is expensive and prevents you from handling billion scale graphs.
It's possible to run cypher against duckdb (soon postgres as well via duckdb's postgres extension) without having to import anything. That's a game changer when everything is in the same process.
- Does it need index free adjacency?
- Does it need to implement compressed sparse rows?
- Does it need to implement ACID?
- Does translating Cypher to SQL count as a graph database?
- Ladybug by default allocates 80% of the physical memory to the buffer pool. You can limit it. This wasn't the main reason.
- Much of the RSS is in ladybug native memory connected to the python connection object. I noticed that you keep the connection open between benchmark runs. For whatever reason, python is not able to garbage collect the memory.
We ran into similar lifetime issues with golang and nodejs bindings as well. Many race conditions where the garbage collector releases memory while another thread still has a reference to native memory. We now require that the connection be closed for the memory to be released.
reply