Writing

Where to deploy an LLM in 2026

Hard problem
Renting one GPU inside your own application boundary is the only option for workloads that cannot send data to a hosted API, and it is exactly the option platforms are withdrawing.
Approach
Read the economics from the platforms that quit. The teams who need in-boundary GPU most are the reason it does not sell, which makes this structural rather than one company misjudging a product.

Renting a GPU is getting harder to buy, and the reason is not that demand fell. It is that segregating GPUs breaks the pooling that makes a cloud work.

  • Segregated GPU fleet

    Closing

    A second, smaller pool only GPU work may enter

    Few customers, bursty demand, so headroom is large relative to the pool — and every idle unit costs roughly an order of magnitude more than an ordinary server.

  • Mixed general-purpose fleet

    Fine, and always has been

    One large pool any workload can enter

    Independent customers peak at different moments, so bursts cancel and the headroom needed grows far slower than the fleet.

Pooling is the whole story. Splitting one well-behaved pool into a big cheap one and a small expensive one puts all the volatility in the most expensive place.

Why the fleet could not pay for itself

We shipped GPUs on dedicated server hardware, so that GPU- and non-GPU workloads weren't mixed.

That is the whole mechanism. Fly's ordinary servers are fine, because pooling works: independent customers peak at different moments, bursts cancel, and the headroom a fleet must carry grows far slower than the fleet itself. Segregation replaces one well-behaved pool with two — a large cheap one, and a small expensive one that has fewer customers, burstier demand, proportionally more headroom, and idle units costing about an order of magnitude more. Hence their GPU servers being "drastically less utilized and thus less cost-effective than our ordinary servers". Dedication is not the problem in itself. Concentrating the volatility in the most expensive pool and cutting off its backfill is.

Serverless is the fix, and the exception

for most software developers, "AI-enabling" their app is best done with API calls to things like Claude and GPT, Replicate and RunPod.

The company exiting GPU rental is pointing its users at serverless GPU, and it is right to. Modal, Replicate and RunPod grow because they rebuild the pool: one card multiplexed across many customers, billed by the second. But the fix is sharing the hardware, which is the single thing a workload under an isolation requirement cannot accept. So the arrangement that repaired the economics is unavailable to exactly the work that most needs a GPU it does not own, and the demand left in the shrinking middle is almost entirely regulated. The segment that needs it most is the segment that makes it least sellable — which is why this is structural rather than one company misjudging a product, and why changing providers does not help.

What to do if you are in the exception

developers don't want GPUs. They don't even want AI/ML models. They want LLMs.

True, and the useful corollary is that you want as little GPU as possible. I have been on the wrong side of this, with a vision model reading uploads that were not permitted to leave the application's boundary. Only one stage of that pipeline runs a model: a deterministic image-quality check filters junk before any inference, and a conventional OCR pass verifies the output afterwards. Neither is an LLM and neither needs a card, so this is not the same model on cheaper hardware, it is different tools doing work the model does not have to. Shrink the GPU surface that way and what remains is portable, so keep the model behind an interface and assume renting a card gets harder. The model is the least durable decision in the system; the constraint that put it in-boundary is the most durable one.