When a company decides to run an AI model in-house, the first question is usually: “Which graphics card should we buy?” But the graphics card is only part of the story. A card that does not fit the server chassis, a power draw the power supply or UPS cannot handle, or a server room that cannot remove the heat will stop the whole project.
Why a GPU?
AI models essentially perform huge numbers of matrix multiplications, and a graphics card does this in parallel, far faster than a CPU. Small models can run on a CPU, but responses are slow and usually not enough for several users at once.
GPU memory: the most important number
For running a model (inference), the card’s VRAM usually matters more than its raw speed, because the whole model has to fit in memory. A rough rule:
- Memory needed ≈ number of model parameters × bytes per parameter.
- At 16-bit precision each parameter takes 2 bytes, so a 7-billion-parameter model needs about 14 GB for the weights alone.
- With 4-bit compression the same model needs roughly a quarter of that.
- Long input text and concurrent users need additional memory.
Training or fine-tuning a model takes several times more memory than running it. Most small and mid-sized companies only need to run models.
Before buying, try your chosen model on a cloud server or test system with the same amount of memory. This keeps you from buying a card that is too small, or far larger than your model needs.
Desktop card or data center card?
Data center cards usually offer more memory, cooling designed for server chassis and official support for virtualization, but they cost more. Desktop cards are cheaper, but they may not fit a server chassis, their fans and airflow may not be designed for a rack, and they may lack features such as vGPU. Check the driver and data center usage license terms for each product separately. Parts availability and warranty in your market are also important criteria.
CPU, RAM and storage
- CPU and PCIe slots: The motherboard needs enough slots and PCIe lanes for the cards. The CPU does not need to be very powerful, but it must not become a bottleneck.
- System RAM: Plan for at least as much as the total GPU memory, since the model is loaded into RAM first.
- Storage: NVMe drives to store and quickly load model files; if you connect many documents to the model, account for their capacity and backup too.
- Network: A standard network is fine for a single server; multiple servers or shared storage call for a high-speed network.
Power and cooling: the part people forget
Powerful graphics cards draw a lot of power under load and turn nearly all of it into heat. Before installation:
- Compare the server power supply’s rating with the combined draw of the cards and other components, and leave a safety margin.
- Recalculate UPS capacity and runtime after adding the new server.
- Check the rack’s electrical circuit capacity and cabling.
- Make sure the server room’s cooling can remove the extra heat and that the rack’s hot and cold airflow is correct.
Checklist
- You defined the model and use case before buying hardware.
- You estimated the VRAM needed for the model, text length and number of users.
- The card’s size, cooling and compatibility with the server chassis and motherboard are confirmed.
- The need for GPU Passthrough or vGPU, and its licensing, is clear.
- The power supply, UPS and electrical circuit have been reviewed.
- Server room cooling is sufficient for the new load.
ITPlus724 engineers can assess and prepare the servers, racks, power, UPS, cooling and virtualization your company needs to host AI services in your server room or data center.