You'd want to get something like a RTX Pro 6000 (~ $8,500 - $10,000) or at least a RTX 5090 (~$3,000). That's the easiest thing to do or cluster of some lower-end GPUs. Or a DGX Spark (there are some better options by other manufacturers than just Nvidia) (~$3000).
Yes, I also considered the RTX 6000 Pro Max-Q, but it’s quite expensive and probably only makes sense if I can use it for other workloads as well. Interestingly, its price hasn’t gone up since last summer, here in Germany.
I have MacStudio with 512GB RAM, 2x DGX Spark and RTX 6000 Pro WS (planing to buy a few of those in Max-Q version next). I am wondering if we ever see local inference so "cheap" as we see it right now given RAM/SSD price trends.
Good grief. I'm here cautiously telling my workplace to buy a couple of dgx sparks for dev/prototyping and you have better hardware in hand than my entire org.
What kind of experiments are you doing? Did you try out exo with a dgx doing prefill and the mac doing decode?
I'm also totally interested in hearing what you have learned working with all this gear. Did you buy all this stuff out of pocket to work with?
Yeah, Exo was one of the first things to do - MacStudio has a decent throughput at the level of 3080, great for token generation and Sparks have decent compute, either for prefill or for running non-LLM models that need compute (segment anything, stable diffusion etc). RTX 6000 Pro just crushes them all (it's essentially like having 4x3090 in a single GPU). I bought 2 sparks to also play with Nvidia's networking stack and learn their ecosystem though they are a bit of a mixed bag as they don't expose some Blackwell-specific features that make a difference. I bought it all to be able to run local agents (I write AI agents for living) and develop my own ideas fully. Also I was wrapping up grad studies at Stanford so they came handy for some projects there. I bought it all out of pocket but can amortize them in taxes.
Building AI agents for a living is what I hope to become able to do, too, I consider myself still in learning phase. I have talked with some potential customers (small orgs, freelancers) and learned that local inference would unlock opportunities that have otherwise hard to tackle compliance barriers.
That you are writing AI agents for a living is fascinating to hear. We aren't even really looking at how to use agents internally yet. I think local agents are incredibly off the radar at my org despite some really good additions as supplement resources for internal apps.
What's deployment look like for your agents? You're clearly exploring a lot of different approaches . . .
My commercial agents are just wrappers on top of GPT/Claude/Gemini so the standard deployment ways on Azure/AWS/GCP apply with integrations to whatever systems customers have like JIRA, Confluence etc. Some need to automate away some folks with repetitive work, some need to improve time to delivery with their people swamped by incoming work, hoping to accelerate cognitively-demanding tasks etc.
It's mostly RAM allocated per client. E.g. Postgres is very much limited by this fact in supporting massive numbers of clients. Hence pgbouncer and other kinds of connection pooling which allow a Postgres server to serve many more clients than it has RAM to allow connecting.
If your Node app spends vety little RAM per client, it can indeed service a great many of them.
A PHP script that does little more than checking credentials and invoking sendfile() could be adequate for the case of serving small files described in the article.
Except that it wastes 2 or 3 orders of magnitude in performance and polls all the connections from a single OS thread, locking everything if it has to do extra work on any of them.
Picking the correct theoretical architecture can't save you if you bog down on every practical decision.
I'm sure there is plenty of data/benchmarks out there and I'll let that speak for itself, but I'll just point out that there are 2 built-in core modules in Node.js, worker_threads (threads) and cluster (processes) which are very easy to bolt on to an existing plain http app.
So think of it this way: you want to avoid calling malloc() to increase performance. JavaScript does not have the semantics to avoid this. You also want to avoid looping. JavaScript does not have the semantics to avoid it.
If you haven’t had experience with actual performant code JS can seem fast. But it’s is a Huffy bike compared to a Kawasaki H2. Sure it is better than a kid’s trike but it is not a performance system by any stretch of the imagination. You use JS for convenience, not performance.
IIRC V8 actually does some tricks under the hood to avoid malocs which is why Node.js can be be unexpectedly fast (I saw some benchmarks where it was only 4x of equivalent C code) - for example it recycles objects of the same shape (which is why it is beneficial not to modify object structure in hot code paths).
JITs are a great magic trick but it's nowhere near guaranteed you'll get good steady performance out of one, especially when the workload is wide not narrow.
Hidden classes is a whoooole thing. I’ve switched several projects to Maps for lookup tables so as not to poke that bear. Storage is the unhappy path for this.
(to be fair the memory manager reuses memory, so it's not calling out to malloc all the time, but yes a manually-managed impl. will be much more efficient)
Whichever way you manage memory, it is overhead. But the main problem is the language does not have zero copy semantics so lots of things trigger a memcpy(). But if you also need to call malloc() or even worse if you have to make syscalls you are hosed. Syscalls aren’t just functions, they require a whole lot of orchestration to make happen.
JavaScript engines also are also JITted which is better than a straight interpreter but except microbenchmarks worse than compiled code.
I use it for nearly all my projects. It is fine for most UI stuff and is OK for some server stuff (though Python is superior in every way). But would never want to replace something like nginx with a JavaScript based web server.
V8 does a lot of things to prevent copies. If you create two strings and concat them and assigned to a third var, no copy happens (c = a+b, c is a rope). Objects are by reference... Strings are interned.. the main gotcha with copies is when you need to convert from internal representation (utf16) to utf8 for outputs, it will copy then.
I was a Java programmer out of school. My first specialization became performance. It’s the same problem with JavaScript. There’s at least an order of magnitude speed difference between using your head and just vomiting code, and probably more like a factor of forty. Then use some of the time you saved to do deep work to hit an even two orders of magnitude and it can be fast. If you’re not an idiot that is.
bcantril has an anecdote about porting some code to rust and it was way faster than he expected. After digging he found it was using btrees under the hood, which was a big part of the boost. B-trees being a giant pain in the ass to write in the donor language but a solved problem in Rust. It’s a bit like that. You move the effort around and it evens out.
> With JS, we have a problem that (mainstream) software hadn't faced before, which is client server architecture, that the client might be a cruddy cheap phone somewhere and/or on a dodgy link.
2000 was the peak of the dot-com boom. In the US, half of Silicon Valley was developing for client-server and nearly every person with a computer was using client-server applications.
To give a clearer example, Napster, where I worked, had 60 million users, mostly in the US and we were far from the largest.
The cruddy cheap phones one might complain today is several times more powerful than the servers we used then and the dodgy connection today are downright sublime compared to the dial-up modems over noisy phone lines.
The internet was beyond a frontier was clearly happening. But it's insane to me to pretend that most software was internet software.
Yes that was the excitement the enticing future & an amazing wave breaking upon us. Many were tuned in. But Napster is far more exception that proves the rule. Very very very consumers had any experience with networked software with client server, and I think you d have to be an absolute fucking moron to pretend that most of the world when we were thinking of computers them were thinking of networked client server systems. Get fuckign real. I appreciate that we the alpha geeks saw what was happening. But that doesn't embody what was actually happening, doesn't capture what people saw or felt.
Napster was the fucking envy. It pioneered something else. I admit it was mainstream, but as an exception that proved the rule, as something totally totally different that we all loved.
Architecture is far more important than runtime speed. (People are so easily swayed by "JS SUCKS LOL" because of experiences with terrible & careless client-server architectures, far more than js itself being "slow".)
The people ripping into js suck up the interesting energies, and bring nothing of value.
If we are discussing C10K we are by definition discussing performance. JavaScript does not enter this conversation any more than BASIC. Yes of course architecture matters. Nobody has been arguing otherwise. But the point is that if you take the best architecture and implement it in the best available JS environment you are still nowhere close to the same architecture implemented in a systems language in terms of performance. You are welcome to your wishful thinking that you do not need to learn anything besides JavaScript when it comes to this conversation. But no matter how hard you argue it will continue being wishful thinking.
We are discussing tech where having a custom userland TCP stack is not just a viable option but nearly a requirement and you are talking about using a lighter JS framework. We are not having the same conversation. I highly recommend you get off Dunning-Kruger mountain by writing even a basic network server using a systems language to learn something new and interesting. It is more effort than flaming on a forum but much more instructive.
Techempower isn't perfect, but a 33% loss of speed is really not bad versus best of the best web stacks. Your attempt to belittle is the same tired heaping scorn, when the data shows otherwise. But it's so cool to hate!
Libuv isn't the most perfect networking stack in the planet, no. It could be improved. We could figure out new ways to get io-uring or some eBPF based or even dpdk bypass userland stack running. Being JS does not limit what you do for networking, in any way. At all. It adds some overhead to the techniques you choose, requires some glue layer. So yes, some cpu and memory efficiency losses. And we can debate whether thats 20% or 50% or more. Techempower shows it's half an order of magnitude worse (10^0.5, 33%). Techempower is a decent proxy for what is available, what you get, without being extremely finicky.
Maybe that is the goal, maybe this really is a shoot for the moon effort where you abandon all existing work to get extreme performance, as c10k was, but that makes the entire lesson not actually useful or practical for almost everyone. And if you want to argue against techempower being a proxy for others, for doing extreme work & what is possible at the limit, you have to consider a more extreme js networking than what comes out of box in node.js or Deno too, into a custom engine there too.
It's serious sad cope to pretend like it's totally different domains, that js is inherently never going to have good networking, that it can't leverage the same networking techniques you are trying to vaunt over js. The language just isn't that important, the orders of magnitude (<<1 according to the only data/evidence in this thread) are not actually super significant.
Look you seem to have made up your mind and are unwilling to listen to knowledge or experience. The tech industry does not do well with willful ignorance so I wish you luck with all that.
No, I'm pointing out that you have no evidence at all & and aren't trying to speak reasonably or informatively, that you are bullying a belief you have without evidence.
I'm presenting evidence. I'm clarifying as I go. I'm adding points of discussion. I'm building a case. I just refuse to be met and torn down by a bunch of useless hot air saying nothing.
Nothing about current js ecosystem screams good architecture it’s hacks on hacks and a culture of totally ignoring everything outside of your own little bubble. Reminds me of early 2000s javabeans scene
Reminds me of early 2000s JS scene, in fact. Anything described as "DHTML" was a horrid hack. Glad to see Lemmings still works, though, at least in Firefox dev edition. https://www.elizium.nu/scripts/lemmings/
Unqualified broadscale hate making no assertions anyone can look at and check? This post is again such wicked nasty Dark Side energy, fueled by such empty vacuous nothing.
It was an incredible era that brought amazing aliveness to the internet. Most of it was pretty simple & direct, would be more intelligible to folks today than today would be intelligible & legible to folks from them.
This behavior is bad & deserves censure. The hacker spirit is free to be critical, but in ways that expand comprehension & understanding. To classify a whole era as a "horrible hack" is injurous to the hacker spirit.
Worker threads can't handle I/O, so a single process Node.js app will still have the connection limit much lower than languages where you can handle I/O on multiple threads. Obviously, the second thing you mention, ie. multiple processes, "solves" this problem, but at a cost of running more than one process. In case of web apps it probably doesn't matter too much (although it can hurt performance, especially if you cache stuff in memory), but there are things where it just isn't a good trade-off.
And I have confirmed to my own satisfaction that both PM2 and worker threads have their own libuv threads. Yes very common in node to run around one instance per core, give or take.
Libuv now supports io_uring but I’m fuzzy on how broadly nodejs is applying that fact. It seems to be a function by function migration with lots of rollbacks.
Very nice! Just a note (as the site says on bottom left side), this can vary depending on the CPU you use, would be nice to be able to select all different variations of supported CPUs as a future feature.
For instance suppose the marginal buyer would spend $20 for a product and the producer can produce for $18 and the market price is $19, resulting in $1surplus to each party. If a $3 tax was put on the product, the transaction wouldn't take place.
So you went from $2 social surplus to nothing and no tax benefit.
Any arbitrary price distortions will result in dead weight loss or the overall social surplus to go down compared to no distortion
The market price would be floored at $21 as it costs the producer $18 to produce and a $3 tax. The prior margin was $1 so realistically the producer would eat some of the increase and charge $21.50
For the person who values the good only at $20, no transaction would take place. If eventually the producer cost went down to something like $16, maybe the price would come down to $20 or below but then there's the marginal consumer that values it at $19.
Price distortions almost always result in a dead weight loss. Only in perfectly inelastic products does it not exist
> Price distortions almost always result in a dead weight loss.
Yes, some trades will not take place take after the distortion, but the the person valuing the good at 20 and still be unwilling to pay 23 after an extended time period in which they're unable to buy it is extremely rare.
Aha. So it's not the politically-connected who manage to keep IRS at arms-length. No. It is 'something but not political connections' that manages to frustrate the unjust greed of the "plitically connected" to tax rightfully earned profits of these corporations. Must be the "hand of God" ..
Housing shortage, at least in the US, isn’t really a problem of capital availability. It’s mostly due to local zoning regulations that make it very difficult to actually create housing, regardless of capital availability.
For example, New Jersey has 9 million people. Denmark 5 million. Why have other countries or states not achieved the same thing if size is the main driver?
Curious what you mean when you say strictly as an investment? In my worldview, if you are purchasing real estate as an investment, it means it generates income, meaning you plan to rent it out.
Or develop it. Or combine or subdivide parcels. Or as an alternative/hedge store of wealth.
Or even if you plan on living in it, if you want to make money when it comes time to sell, and not just break even, after accounting for total cost of ownership.
Most people who think they sell their house for a modest profit are wrong.
This is only true in places where investment capital has taken over the market, not in places where individuals are scraping together funds to buy in an economically growing neighborhood.
I meant more along the lines that people don't take into account all the costs that they should when calculating their break-even price.
This is generally true regardless of the location, with the caveat that hot markets are more likely to have more professionals participating that know how to include those costs.
Often people see those higher prices as greedy profit (or as artificially high in order to gentrify, etc.), and developers do need to make a profit, but often times a large part of the elevated price is just including hidden costs that homeowners don't account for.
But looking at it from the other side, developers/investors are only going to enter markets where they believe there's a margin for profit. Which generally means those neighborhoods are a bit underpriced to begin with.
Real estate is the highest-leverage purchase most people can make, so even if you're living in it and plan to sell in the near-ish future, a 5% appreciation will yield you more net cash than a 5% appreciation on the same amount of down payment put into a "regular" investment. The obviously-oversimplified example is something like: put 20K down, buy a 100K place. Sell for 105K, your 20K is now 25K, assuming your mortgage payments were the same as what you would've paid for rent otherwise. Which is a 25% return on your 20K, even on just 5% appreciation.
I call this speculation. In most real estate markets, a normal house is a depreciating asset that requires upkeep, maintenance, property taxes, and other liabilities.
No, not really. It's very rare for a house to depreciate, even if no repairs are being done. Moreover, property taxes are almost always going to be lower than the rise in price of the property.
And speculation and investing are the same thing, except the former is often used in a pejorative way.
reply