The Milk-V Titan (https://milkv.io/titan) can take up to 64GB which is fine considering the number of cores and the cost of RAM. If you needed and could afford more RAM you'd be better off distributing the work across more than one board.
Unfortunately they found a bug and had to redesign the boards. I've had one of these on pre-order since last year. Latest is I think they're intending to ship them next month (April).
Ok! I will keep an eye out. It is one of the most interesting developments for me hardware wise in the last decade, and I definitely want to show my support by buying one or more of the boards. Respin is always really annoying this late in, the post mortem on that must make for interesting reading.
Ahh I thought I was the only one! One line per sentence makes the diffs so much nicer too, maybe we need git hooks to reject multiple sentences per line?
You can just do `git diff --word-diff` and then the diffs look great even with one paragraph per line.
I used to do one sentence per line, but after getting used to how Emacs handles soft-wrapping, I now do one paragraph per line—also when I use Vim. This also makes collaboration with other authors easier, since most non-vim collaborators do that.
Also IMO, sentences are a more useful logical grouping. I might want to swap sentence orders, so having the quick dd then p command is nice. Paragraphs get moved occasionally but it is rare enough that I don’t mind grabbing them as if they are a collection of sentences.
I didn't find note-taking particularly useful until I started keeping everything in a single notebook with dated pages.
This worked a lot better than (for example) trying to organise notes by category - it's often easier to remember when you were working on something than how you categorised it, and once you know roughly when, you can find it by binary search
I saw a similar (I think!) paper "Grassmannian Optimization Drives Generalization in Overparameterized DNN" at OPT-ML at neurips last week[0]
This is a little outside my area, but I think the relevant part of that abstract is "Gradient-based optimization follows horizontal lifts across low-dimensional subspaces in the Grassmannian Gr(r, p), where r p is the rank of the Hessian at the optimum"
I think this question is super interesting though: why can massively overparametrised models can still generalise?
Which repos worked well? I've had the same experience as op- unhelpful diagrams and bad information hierarchy. But I'm curious to see examples of where it's produced good output!
Really nice! Had a quick read, here's my quick summary:
- Arrays are typed `S : D` with shape S and strides D
- Each of `S` and `D` is a nested tuple (instead of the flat tuples one typically sees in a tensor framework)
- Together `S` and `D` define the layout of a tensor
- Not every layout is "tractable", but the tractable ones form a nice category
A really good exposition, my only criticism is that it's quite front-heavy- it would be nice to see a detailed example like in 2.3.8 earlier in the document; there is a lot of detail presented early that doesn't seem necessary to understand the core ideas.
Last comment: I suspect there is a connection to strictification[0], would love to know more if the authors see this!
Hellas.AI is building a compiler for categorical deep learning to power decentralised, serverless AI.
We're looking for an experienced Rust engineer. You must have demonstrated:
* Experience in designing and implementing BFT consensus protocols
* Ability to engage with academic literature on distributed systems
* Understanding of ZK proofs and proof aggregation (you don't need to design these, but you should know how to use them)
Nice-to-have: interest or experience in:
* Deep Learning / ML / AI, GPU programming
* Compilers and array programming languages
* Category Theory and string diagrams
If you have project(s) demonstrating your experience in one or more of these areas, get in touch!
Our process is simple: we read code you've written and research you've published, then interview you about it. We don't do tedious coding tests.
I've been hoping for a nice concrete example of braided monoidal categories for ages, who knew that the best one was string diagrams that represent actual string! Great post!
@the author - I assume you're aware that morphisms in symmetric monoidal categories can be represented using cospans of hypergraphs - do you know if there's a similar combinatorial representation for braided monoidal categories?
(author here) the goals of catgrad are a bit different to JAX - first of all, the "autodiff" algorithm is really a general approach to composing optics, of which autodiff is just a special case. Among other things, this means you can "plug your own language" into the syntax library to get a "free" autodiff algorithm.
Second, catgrad itself is more like an IR right now - we're using it at hellas.ai to serve as a serverless runtime for models.
More philosophically, the motto is "write programs as morphisms directly".
Rather than writing a term in some type theory which you then (maybe) give a categorical semantics, why not just work directly in a category?
Long term, the goal is to have a compiler which is a stack of categories with functors as compiler passes. The idea being that in contrast to typical compilers where you are "stuck" at a given abstraction level, this would allow you to view your code at various levels of abstractions.
So for example, you could write a program, then write an x86-specific optimization for one function which you can then prove correct with respect to the more abstract program specification.
> Among other things, this means you can "plug your own language" into the syntax library to get a "free" autodiff algorithm.
Hello, as a compiler engineer I am interested in this area. Can you expand a little bit more? How would I be able to plug in my own language for example?
> So for example, you could write a program, then write an x86-specific optimization for one function which you can then prove correct with respect to the more abstract program specification.
So, what you are saying is that catgrad alllows me to write a program and then also plug in a compiler pass? I.e., the application author can also be the compiler developer?
I get it, but once you’ve compiled your program down to a computational graph…well those are basically morphisms in a monoidal/multi category, no? And then you can have a stack of MLIR dialects that represent different levels of abstraction, where optimization passes are functors between the multicategory determined by that dialect, then you hand it off to LLVM for CPU execution, or maybe you want to run it with WebGPU, go down to CUDA, etc.
can you expand on "write programs as morphisms directly"
I'm someone super interested in category theory and ITT, but I can't quite parse what you are trying to convey even though I think I have the prereqs to understand the answer.
Hellas.AI is building a compiler for categorical deep learning to power decentralised, serverless AI.
If you're interested in any of the following, get in touch:
* Category Theory and string diagrams
* Compilers and array programming languages
* Deep Learning/ML/AI and/or GPU programming (e.g. CUDA/
WGSL)
* Building and deploying distributed systems
* Peer-to-peer software and blockchain
If you have project(s) demonstrating your experience in one or more of these areas, get in touch!
Our process is simple: we read code you've written and research you've published, then interview you about it.
We don't do tedious coding tests.
Some experience of Rust or a functional language (Haskell, OCaml, any Lisp) is preferred.
reply