NVIDIA DGX H100: specifications, costs, and cloud alternatives

NVIDIA DGX H100 is an integrated server with eight H100 GPUs and 640 GB of total GPU memory. It is designed for AI training, inference, and other accelerated workloads that can use several GPUs together. Its value depends on the workload, the connections between GPUs, and the team operating it.

If you are evaluating DGX H100, start with a configuration-specific quote and a representative benchmark. An advertised GPU count or peak performance figure does not tell you how quickly your application will run, how much it will cost, or whether a smaller system would meet the same target.

What is inside a DGX H100?

The NVIDIA DGX H100/H200 user guide documents the hardware. H100 and H200 configurations share a manual, so check the model attached to each specification.

Component	DGX H100 specification
GPUs	8 NVIDIA H100 GPUs; 80 GB per GPU, 640 GB total
GPU interconnect	NVLink and NVSwitch; published GPU-to-GPU bandwidth of 900 GB/s
Processors	2 Intel Xeon Platinum 8480C CPUs, 56 cores each
System memory	2 TB
Operating-system storage	2 × 1.92 TB NVMe SSDs in RAID 1
Data cache	8 × 3.84 TB NVMe SSDs in RAID 0
Cluster networking	8 ConnectX-7 interfaces, up to 400 Gb/s each
Form factor	8U rackmount; maximum weight 130.45 kg
System power	10.2 kW maximum

The memory total is spread across eight GPUs. Software must distribute the model or work to use that capacity effectively. System RAM is separate from GPU memory, and the local data cache should not be treated as your only durable copy of a dataset or checkpoint.

Why NVLink and NVSwitch matter

Some jobs divide one model across GPUs and exchange data repeatedly. Others run independent tasks on separate devices. The first group can be much more sensitive to interconnect performance. A comparison between two eight-GPU machines should therefore include their topology and communication behavior, not just their accelerator count.

For example, tensor parallelism splits model calculations across devices. Whether that helps depends on the implementation and the time spent exchanging intermediate results. Measure useful throughput and latency with the actual model rather than assuming that eight GPUs produce an eightfold speedup.

Which workloads should consider DGX H100?

A DGX H100 can be worth evaluating when a team needs substantial GPU memory, frequent communication between devices, or a shared system for sustained demand. Training large models is one use case, but inference, analytics, and scientific workloads can also justify a multi-GPU system. The job’s requirements decide the fit.

• Distributed training: measure time to the same quality target, including data loading, checkpointing, and recovery.
• Large-model inference: test memory use, context length, concurrency, and response latency together.
• Independent batch jobs: compare the cost of scheduling many jobs on a shared server with renting smaller instances.
• Scientific computing: verify the numerical precision, libraries, and communication pattern required by the application.

There is no reliable rule that every model below a particular parameter count fits one GPU, or that fine-tuning always needs less than 48 GB. Precision, optimizer state, activations, batch size, and sequence length all affect memory demand. An inference result cannot establish a training requirement.

When a smaller system may be enough

Start with the smallest configuration that can run the complete job within its memory and time limits. If one GPU meets the target and demand is intermittent, an eight-GPU server may leave substantial capacity idle. If several teams can keep it busy, or the model needs fast exchanges across GPUs, the calculation changes.

Record queue time as well as execution time. A fast shared machine can still delay a project when access is constrained. Conversely, renting capacity only when needed is useful only if the required configuration is available when the work must run.

How much does DGX H100 cost?

Use a current supplier quote for the exact system, support coverage, software entitlements, location, and delivery terms. This guide does not give a universal purchase price: an old reseller listing or an unsupported range is not a dependable budget for a new procurement.

Ask the supplier to separate the server price from installation, support renewals, networking, storage, taxes, and facility work. For used equipment, also verify warranty transfer, service eligibility, configuration, and replacement-part availability. A lower acquisition price can come with different operating responsibilities.

Calculate total cost of ownership

Build a model over the period you expect to use the system. Include:

1. Hardware acquisition, financing if applicable, and any expected resale value.
2. Network switches, cables, shared storage, and backup capacity.
3. Electricity, cooling, rack space, and installation work.
4. Support, software renewals, maintenance, and staff time.
5. Idle capacity, failed jobs, planned maintenance, and recovery time.

Divide that total by the useful work completed over the same period. For an hourly comparison, keep the units consistent: a whole-server hour is different from one GPU-hour. An eight-GPU server used for one hour represents eight allocated GPU-hours, even if the application leaves some devices idle.

Utilization matters because fixed costs continue when the system is unused. As an illustrative calculation, spreading the same fixed annual cost over 2,000 productive server-hours instead of 4,000 doubles that fixed cost per productive hour. Electricity and other variable costs need their own assumptions.

Power, cooling, and deployment requirements

NVIDIA’s DGX H100 deployment guide specifies air cooling and a 10.2 kW maximum system power requirement. That maximum is a planning limit, not a claim that every application continuously draws 10.2 kW.

Have the facility team check rack loading, electrical distribution, airflow, maintenance access, and redundancy before ordering. A server fitting into a rack does not establish that the room can power and cool it. Use measured workload consumption for energy estimates and the manufacturer’s requirements for installation planning.

For several servers, size storage and networking with compute. Slow dataset reads or checkpoint writes can leave GPUs waiting. The GPU cluster guide covers the relationship between worker nodes, shared storage, networks, and scheduling.

DGX H100, DGX A100, and newer systems

DGX H100 belongs to NVIDIA’s Hopper generation. It should not be described as the current flagship: the DGX documentation catalog also covers newer systems. Compare the exact models available to you, including support terms and delivery dates.

For an existing A100 installation, use the DGX A100 guide as background, then benchmark the proposed replacement against your current software and workload. Separate improvements from the hardware, numerical precision, and software changes. A vendor’s best-case multiplier is not a forecast for every application.

Moving to newer hardware may increase memory or throughput, but it can also change facility requirements and the software you need to validate. Include migration effort in the comparison. Keeping a working system can be reasonable when it still meets the project’s requirements.

Cloud alternatives to owning a DGX H100

Renting H100 capacity can avoid purchasing and operating a physical server. Compare the instance’s GPU model, memory, interconnect, CPU resources, network, storage, and support. A cloud instance containing H100 GPUs is not automatically a DGX H100 system. Our H100 on AWS guide explains one provider-specific route.

Request a cost estimate for the same workload and service level. Include runtime, data movement, persistent storage, idle resources, capacity reservations, and operational work. Check quotas, regional capacity, interruption terms, and how checkpoints survive an instance failure.

Where Compute with Hivenet fits

The Hivenet GPU reference lists RTX 5090 instances with 32 GB of VRAM per GPU and configurations up to eight GPUs when capacity is available. Hivenet’s RTX 4090 fleet is retired. These RTX instances have different memory and interconnect characteristics from DGX H100.

Evaluate Compute with Hivenet for workloads that fit the selected preset and run well in a representative test. Do not assume equivalent performance from the GPU count alone, or promise that an RTX instance can replace a DGX system for every task. Consult the current pricing page and confirm the exact preset, location, availability, and active price in the console before budgeting.

A practical evaluation checklist

• Define the model, dataset, precision, batch size, concurrency, and required completion time.
• Measure memory use and performance on a representative configuration.
• Test data loading, checkpointing, failure recovery, and sustained demand.
• Compare complete ownership and rental costs using the same units and assumptions.
• Confirm installation requirements, support coverage, software compatibility, and who owns daily operations.

A useful purchase decision explains which requirement the system meets and what evidence supports that conclusion. If the benchmark exposes a memory, network, or storage limit, address that specific constraint before adding more hardware.

Frequently asked questions

Does DGX H100 have 640 GB of shared GPU memory?

It has 640 GB in total across eight GPUs. Applications need an appropriate distribution strategy to use multiple devices; the total should not be interpreted as one ordinary 640 GB GPU allocation.

Does DGX H100 require liquid cooling?

NVIDIA documents DGX H100 as air-cooled. The facility still needs to meet its airflow, temperature, power, and installation requirements.

Is DGX H100 only useful for training?

No. Inference and other accelerated workloads can also benefit. Evaluate memory, communication, throughput, latency, and utilization for the actual application.

Is cloud GPU rental always cheaper?

No. The result depends on workload performance, demand, capacity terms, and the full operating cost. Compare completed work over a realistic period, including time when owned or reserved capacity sits idle.