↓Skip to main content

Local Inference Hardware Guide

·7 mins

I’ve used a pair of DGX Sparks to host various LLMs using SGLang, ollama and vLLM. I got by far the best results out of vLLM thanks to its aggressive caching. SGLang should also perform well, but I had trouble running recent models with it. For now I use vLLM to host qwen3.6-27B, which is smart enough to be useful in many real tasks.

The “best” model changes often: we will probably switch to the recently released DeepSeek V4 Flash 0731, and qwen 3.8 is on the horizon.

The DGX Spark is a great machine, but to serve an entire department, it’s easy to justify spending a bit more. I researched how the GPU spec sheet feeds into inference speed so that we can meet our needs without wasting money.

Inference primer #

Let’s begin with an intro on LLM serving. You submit a bunch of text, the GPU makes noise, text comes out. When trying to find inference bottlenecks, there are only two important computation steps:

  • prefill: Computation happens on each token of the input text. This benefits from the raw compute of the GPU - you can work on all tokens in parallel. Prefill is improved with more FLOPs. Prefill was and still is a minor part of the overall inference workload.

    Agentic coding slightly increases the role of prefill because every time the agent reads a file or curls a webpage, the resulting thousands of tokens are added to the context using a prefill step.

    Another trigger for prefill are KV cache misses. The data generated from the prefill stage has to be kept in cache for the entire duration of the agentic session. If the cache is cleared, for example because a tool took too long or because the agent was paused waiting for human input, resuming the session means repeating the prefill stage for the entire context - potentially hundreds of thousands of tokens.

  • decode: The LLM is used to generate output. This operation is usually orders of magnitude slower than prefill due to the following constraints:

    • The entire set of weights is needed to generate a token. The operation is I/O-bound - the tensor cores wait for a chunk of model weights to stream into the registers, do a couple matrix multiplications, then go back to waiting for more weights to be read from VRAM, and so on. This is why for any GPU, we first look at memory bandwidth, not FLOPs.

    • Generating any token requires knowing all tokens before it. If the response contains 1000 tokens, the GPU cannot generate all of them in one pass. If the model weights take up 10Gb, then the GPU needs to read 10Gb * 1000 = 10 Tb to produce all output tokens. Notably, we CAN easily decode multiple tokens in parallel, as long as all of their preceding tokens are known - for example when generating for 10 requests at once, or during speculative decoding.

    • On the last point, “knowing all tokens” also means holding the attention vectors computed for those tokens in memory. If we wanted to serve 100 sessions simultaneously during the same decode run, one likely bottleneck is the KV cache capacity (available VRAM). Inference engines such as ollama, SGLang and vLLM differentiate themselves based on how they optimize usage of the KV cache.

Headline metrics #

From that I derive the following headline characteristics:

  • FLOPs - prefill speed, generally the least important. It can matter at very high batch sizes, but is not the first number to look at.
  • Memory size - how many concurrent sessions and how big of a model until we regress to PCIe/RAM bandwidth or redoing the prefill stage entirely?
  • Memory bandwidth - the main driver of single-session inference speed.
  • Interconnect bandwidth - especially important for clusters of GPUs; high local bandwidth is wasted if the GPUs get stuck talking to each other over a slow interconnect.
  • Architecture optimizations - For example NVFP4 splits full precision floats into blocks of 4-bit floats with a common multiplication factor. Blackwell GPUs have hardware support for doing math in this (lossy) compressed format, allowing us to stream 1/4th as much data through the GPU with (hopefully) minimal impact on the output quality.

Hardware examples #

  • NVDA Tesla P40 - cheap old card with a lot of memory for the price. Sacrifices FLOPs (prefill) and bandwidth for VRAM size. Can host larger models and serve more long-context sessions than anything in its price range. Will be slow, may have software problems, and platform costs - CPU, RAM, electricity - will erase some of the savings.

  • Unified memory (DGX Spark, Strix Halo, Mac..) - tons of “VRAM”, variable compute, low bandwidth. Similar to the above, host large models and serve many concurrent, asynchronous sessions. DGX Spark is notable for NVFP4 support.

  • Low-end gaming GPUs - insufficient VRAM, but the right software setup could read from RAM, not unlike an unified memory machine. Note that even DDR5 RAM will be 2-3x slower than a unified memory machine unless you have 8 memory channels (threadripper). Additionally you are at risk of paying more for the platform (RAM, CPU etc) than for the GPU.

  • High-end gaming GPU (RTX 5090) - tight on VRAM but very good bandwidth. Fast inference as long as you can fit your desired model. If KV cache/weights overflow into RAM, inference speed will approach that of a cheaper card.

  • Cluster of high-end gaming GPUs - high bandwidth and decent of VRAM, but only until the GPUs need to talk to each other. Good if you can distribute a MoE model across the individual cards, so that each can compute independently. Bad for dense models as you will just replicate the weights X times, or worse, read them through PCIe every time.

  • RTX 6000 Pro - high bandwidth, high VRAM, obviously made for this workload. Clustering these has the same pitfalls as in the previous section.

  • Cluster of DGX Spark - very high VRAM, low bandwidth. Serve gigantic models slowly. By comparison, a cluster of RTX 6000 Pro will cost more, provide less VRAM, and risks degrading to PCIe speeds if you try serving dense models.

  • H100/H200 PCI - big price and bandwidth jump, VRAM slightly less/more. Useful if you are in the volume business, serving thousands of customers as fast as possible. Also good if users expect instant responses. For serving asynchronous agentic coding sessions, spending the same money on clustering cheaper cards will buy you more VRAM (at the cost of slower inference).

  • SXM/HGX cards - instead of using PCIe, high-end NVDA cards have a proprietary interconnect for faster inter-GPU communication. This is the only way to get a lot of VRAM in one place at high bandwidth (the low-bandwidth option being a DGX Spark cluster). The problem is obviously the entry price - this amortizes mainly if you can serve a consistent, high volume of requests.

  • DGX Station - 256GB VRAM at 7Tb/s is the highest amount of VRAM that you can get without crossing slow PCIe interconnects or building an HGX system. Bandwidth to the 496GB RAM is also unusually high. Upcoming product so teething software problems are to be expected, same as the DGX Spark had.

    Compared to a cluster of eight RTX 6000 Pro cards, this provides less VRAM for similar cost, but bandwidth will be higher and you won’t have to play around the slow interconnect (as long as you stay within the 256GB). The RTX cluster currently seems straightfordwardly better for serving MoE models.

  • TPUs - companies like Google have built their own custom inference chips. These probably require engineering expertise and custom software to use, but I am not aware of any that are even for sale, so the point is moot.

Summary #

Need fast responses for a small number of users? Maximize bandwidth, but you might be forced to live with a small/stupid model.

Huge models? Maximize memory, ideally use MoE distributed across a GPU cluster. If MoE is not an option, you may have to get used to slow/async inference.

Many concurrent long-context agents? Leave enough space for the KV cache. Compute may also become relevant with many concurrent batches.