Choosing a GPU server for AI is one of today’s infrastructure decisions with the greatest margin for error. The reason is simple: training a model, fine-tuning it, and running it in production are three different workloads, and each one requires a different hardware profile. Buying an 8-GPU node to serve an internal chatbot is a waste of money; trying to train a model from scratch with a single card can mean losing months. In this guide, we explain what each phase requires, how many GPUs you need and which type to choose, the role played by networking, CPU, and storage, and propose three reference configurations to ensure that the investment matches the real requirements of the project.
Training, fine-tuning, and inference: understanding each phase
Before discussing graphics cards, it is important to clarify the terminology, because many oversized configurations result from confusing these three stages.
Training: a model is built from scratch or almost from scratch. It requires processing terabytes of datasets for days or weeks, with all GPUs synchronized. It is the most demanding phase and the least common one in a business environment.
Fine-tuning: an already trained model—such as Llama, Mistral, Qwen, or a computer vision model—is adapted to a specific domain using proprietary data. Techniques such as LoRA or QLoRA significantly reduce memory requirements, while full fine-tuning is closer to a training workload.
Inference: the model is already prepared and responds to requests. The key metrics are latency and tokens per second, rather than raw TFLOPS. This phase accounts for 80% of GPU usage hours in production and is the one that is most frequently sized incorrectly.
The practical rule is this: most companies do not train models; they fine-tune them and run inference. If your project involves deploying a local LLM with private data, a RAG system, or computer vision on an industrial site, your server does not need to resemble one used by a research laboratory. We discuss this approach in detail in our guide to in-house artificial intelligence.
Hardware profile for large-scale training
Training large models is an interconnect bandwidth problem, not just a compute problem. Every optimization step requires all GPUs to exchange gradients, so the bottleneck lies in how they communicate with each other and with other nodes.
The industry standard is the NVIDIA HGX platform: 8 H100 or H200 GPUs in SXM format, connected via NVLink and NVSwitch, providing 900 GB/s of bandwidth between cards—far more than PCIe 5.0 can offer. The H200 provides 141 GB of HBM3e and 4.8 TB/s, compared with 80 GB and 3.35 TB/s on the H100. This allows larger models to be loaded onto each GPU and reduces partitioning requirements. If you are unsure which one to choose, read our dedicated NVIDIA H100 vs H200 comparison.
At node level, a serious training server includes 2 high-core-count CPUs (EPYC 9005 or Xeon 6), 2 TB of RAM or more, 8 × 400 Gb/s NICs—one per GPU—for scaling across multiple nodes, tens of terabytes of local NVMe storage, and a power requirement of 10 to 12 kW per chassis. That last figure alone rules out many server rooms and is the first point we check in any project. To learn more about the architecture of these platforms, read our article about NVIDIA HGX.
Hardware profile for fine-tuning
Fine-tuning occupies the middle ground and is also the most common scenario in business environments. The amount of memory required depends on the size of the model and the technique used:
LoRA/QLoRA on 7B to 13B models: fits on a single GPU with 48 to 96 GB of memory, such as an L40S or RTX PRO 6000 Blackwell. This is the most reasonable entry point.
LoRA on 70B models: requires between 2 and 4 GPUs with NVLink to avoid communication bottlenecks between cards.
Full fine-tuning of 70B or larger models: requires 4 to 8 H100/H200 GPUs, placing it within the training hardware profile.
This is where NVLink stops being a luxury. With two or more GPUs working on the same model, a standard PCIe platform forces data to pass through the CPU and system memory. An NVLink bridge or a motherboard with SXM/NVL support can significantly increase effective performance without changing the GPUs.
Hardware profile for inference—the real 80% of the market
An AI inference server is sized in the opposite way to a training server: first, calculate the VRAM required to host the model, and then adjust the number of GPUs according to the expected number of concurrent users. As a quick reference for LLMs quantized to 4 bits, an 8B model occupies approximately 6 GB, a 32B model around 20 GB, and a 70B model approximately 40 GB. You must also account for the KV cache, which grows according to the number of active conversations and the context length.
This explains why the most cost-effective cards for inference are not necessarily H100s. The NVIDIA L40S (48 GB, 350 W) and the RTX PRO 6000 Blackwell Server Edition (96 GB GDDR7, FP4 engines) can serve 70B models using one or two cards at a fraction of the cost. For lighter workloads and high-density deployments, the L4 (24 GB, 72 W) allows four or more GPUs to be installed in a 1U server without thermal issues. We examine these scenarios in our L40S vs L4 vs T4 comparison.
Software also changes the equation. Inference servers such as vLLM, TensorRT-LLM, or SGLang use continuous batching and paged attention, and can multiply the number of users served by a single GPU by four or five. Before adding a second card, it is worth checking that the first one is being fully utilized.
GPU: how many and which type for each phase
The following table summarizes GPU selection by project stage. It is a general guide; for specific projects, we adjust the model, quantization, and expected concurrency.
Phase
Recommended GPU
Number of GPUs
Key design consideration
Training
H100/H200 SXM (HGX)
8 per node
NVLink + NVSwitch, 400G networking per GPU, cooling, and power
Fine-tuning
H100/H200/RTX PRO 6000
2 to 8
NVLink between cards, total VRAM ≥ model in FP16 + optimizer
70B LLM inference
L40S/RTX PRO 6000 Blackwell
1 to 2
VRAM for the model + KV cache, quantization, and vLLM
Lightweight inference/computer vision
L4/RTX PRO 4500
1 to 4
Density, power consumption per card, and latency
One important distinction is that, in PCIe format—such as the H100 NVL, L40S, or RTX PRO—the chassis is a conventional server and can be upgraded more easily. In SXM format—HGX—the GPU baseboard is integrated and the complete node is purchased as a single platform. Positioned between these two options is NVIDIA MGX, a modular architecture that allows manufacturers to combine CPUs, GPUs, and networking in different form factors.
Networking and interconnection: NVLink, InfiniBand, and 400G Ethernet
Inside the server, the interconnect is NVLink. Between servers, the choice is between InfiniBand NDR (400 Gb/s) and 400G Ethernet with RoCE. InfiniBand remains the standard for training clusters because of its low latency and congestion management, while Ethernet is gaining ground in environments that already operate IP networks and want to avoid maintaining a second infrastructure. None of this is necessary for a single inference node: a 25 or 100 Gb/s network connection to users and storage is sufficient.
CPU, RAM, and storage: the frequently overlooked components
GPUs receive most of the attention, but an unbalanced server leaves them waiting for data. These are three rules we apply to every configuration:
CPU: at least 4 physical cores per GPU and, most importantly, enough PCIe 5.0 lanes for every card to operate at x16. EPYC 9005 processors, with 128 lanes per socket, can provide this in a single-socket system. We explain it in our guide to AMD EPYC 9005 Turin.
RAM: at least twice the total VRAM in the system, allowing data preparation and offloading without overwhelming the server. Training workloads require even more.
Storage: local NVMe storage for active datasets and checkpoints, with a read bandwidth of several GB/s. Serving a dataset from a 1 Gb/s NAS can turn an 8-GPU node into the equivalent of a single-GPU node.
Three reference configurations
These three configurations cover the scenarios we encounter most frequently. The budget ranges are indicative and depend on availability and current market prices.
Configuration 1 · Business inference and RAG—from approximately €15,000
A 2U server with 1 × 32-core EPYC 9005 CPU, 256 GB of DDR5 memory, 2 × NVIDIA L40S GPUs or 1 × RTX PRO 6000 Blackwell, 2 × 3.84 TB NVMe drives, and 25 Gb/s networking. It can serve quantized models of up to 70B to dozens of concurrent users and is a typical foundation for a private LLM or an internal documentation system.
Configuration 2 · Fine-tuning and development—from approximately €40,000
A 4U server with 2 × 32- to 48-core CPUs, 512 GB to 1 TB of DDR5 memory, 4 × H100 NVL GPUs with NVLink bridges or 4 × RTX PRO 6000 Blackwell GPUs, 4 × 7.68 TB NVMe drives, and 100 Gb/s networking. It can fine-tune 70B models using LoRA, train computer vision models from scratch, and serve production workloads on the same system.
Configuration 3 · Training and clustering—from approximately €250,000 per node
An HGX H200 node with 8 SXM GPUs, 2 × 64-core CPUs, 2 TB of DDR5 memory, 8 × NVMe drives, 8 × 400 Gb/s InfiniBand NICs, and planned cooling and power infrastructure requiring 10 to 12 kW per chassis. It is the basic building block of a training cluster and only makes sense with a continuous utilization plan.
On-premises vs cloud: when and why
The cloud is unbeatable for occasional peaks in demand and for testing before purchasing hardware. However, once a GPU is operating continuously, the economics change: an on-premises inference server can often pay for itself within 9 to 14 months compared with an equivalent rental. When sensitive data is involved—in healthcare, legal services, industry, or public administration—data sovereignty and compliance with GDPR and NIS2 can tip the balance. The hybrid model we recommend most frequently is on-premises inference and fine-tuning, with occasional large-scale training in the cloud. To learn more about the available GPU nodes, visit our AI servers section, where we describe different platforms and configuration options.
Frequently asked questions
Which GPU do I need to run a 70B LLM locally?
When quantized to 4 bits, a 70B model occupies approximately 40 GB, so it can fit on one 48 GB L40S with a limited context window or, with greater headroom, on one 96 GB RTX PRO 6000 Blackwell. A second card is recommended when serving many concurrent users.
Is a server with 8 smaller GPUs better than one with 2 larger GPUs?
It depends on whether the workload can be distributed or must be shared. For inference involving many small models, several independent cards—such as the L4 or L40S—provide greater density. For a single large model, fewer GPUs with more VRAM and NVLink generally perform better.
Can I start with one GPU and expand later?
Yes, provided that the chassis, power supply, and CPU are selected with future expansion in mind. You will need available PCIe 5.0 x16 slots, sufficient power capacity, and front-to-back cooling. This is exactly what we plan when configuring a custom server.
What is the difference between HGX and MGX?
HGX is an 8-GPU SXM baseboard with NVLink designed for training. MGX is a modular reference architecture that manufacturers use to build servers with PCIe GPUs, Grace processors, or other accelerators in different form factors.
Do you need a GPU server for your AI project?
We size training, fine-tuning, and inference infrastructure according to your model, users, and server room requirements. Configure your custom GPU server.
