Cloud GPU pricing flipped: Blackwell ships, Hopper declines, cost-per-token inverts

The 2026 GPU Pricing Crossover

Cloud GPU pricing has been climbing in a near-straight line since the H100 ramp in 2023. In early 2026 that line bent. Three things happened in close succession: Blackwell — NVIDIA’s B200 and GB200 — shipped at scale across the major specialized providers and a growing slice of the hyperscaler fleet, Hopper-generation H100 and H200 capacity entered a clear price-decline cycle as the supply curve loosened, and the cost-per-token leaderboard flipped in a way that changes the workload-class decision matrix for almost every team running AI workloads in production.

Three numbers anchor the picture:

• B200 on-demand at $0.42 per million tokens edged out H100 PCIe at $0.47 per million despite a higher hourly rate. Memory bandwidth and FP8 throughput finally moved the per-token economics in Blackwell’s favor for inference workloads where the tokens-per-second-per-dollar metric is the right metric.
• H100 hourly rates start at $2.00 / hr at specialized providers (CoreWeave, Lambda, RunPod, GMI Cloud, Genesis Cloud, Spheron) — a 40–70% discount on hyperscaler list pricing for the same SKU.
• B200 spot at $2.12 / hr lands cost-per-token at roughly $0.15 per million for checkpoint-friendly workloads.

The hyperscaler-list-vs-specialized-provider gap is not new. The cost-per-token Blackwell-vs-Hopper crossover is. So is the magnitude of the gap.

What’s Actually Driving the Decline

Hopper generation H100 and H200 are not getting cheaper because demand collapsed — demand for inference and training capacity has continued to grow at double-digit percent quarter-over-quarter. They are getting cheaper because the supply curve loosened in three places at once:

• Specialized providers finished the multi-year buildout of Hopper-class capacity that started in 2023, and that capacity is now running. Once the capex is sunk, on-demand-rate pricing competes for utilization, not for the capex recovery.
• Hyperscalers moved a slice of their reserved-capacity pricing down on Hopper SKUs to compete with specialized providers on enterprise migration deals.
• Blackwell ramped as a higher-throughput, more energy-efficient alternative for the workloads that benefit from FP8 and the larger memory hierarchy. The teams that were paying premium prices for H100 capacity to serve large-context inference moved a portion of that workload to B200, and the H100 capacity that freed up went to spot and on-demand.

The combination is the pattern that always shows up in semiconductor cycles: the previous generation gets cheaper as the next generation gets to scale, and the workloads partition by the cost-per-output-unit math rather than the headline hourly rate.

The 20–40% Hidden-Cost Overhead

Headline hourly rates are still the wrong number to compare. Hyperscaler egress, storage, networking, and reserved-capacity overhead consistently add 20–40% to the bill above the per-GPU-hour line item. The breakdown:

• Egress fees. AWS, GCP, and Azure charge for outbound data transfer at rates that compound when a workload moves checkpoints between regions, exports inference outputs to a CDN-backed serving layer, or syncs with a multi-cloud setup.
• Storage. EBS, GP3, and NVMe-class block storage rates on hyperscalers run materially above specialized-provider pricing, and the cost compounds with checkpoint-size growth on training workloads.
• Networking. VPC peering, NAT gateway, and inter-AZ traffic charges show up across nearly every multi-tier inference architecture.
• Reserved capacity overhead. The discount on a 1-year or 3-year reserved instance is real, but the lock-in cost when the workload-mix shifts (which it always does in AI workloads) eats most of the discount.

Specialized providers don’t charge egress fees on the same scale and don’t require minimum commitments. The sticker-price comparison ($2.00/hr H100 at GMI vs $4.50/hr p4d at AWS, for example) systematically understates the actual gap.

Workload-Class Decision Matrix

Not every workload should run on the cheapest GPU. The five workload classes that show up in 2026 production deployments and the GPU choice that actually saves money for each:

• Large-scale pretraining. B200 / GB200 in a tightly coupled NVLink-class network. The throughput-per-dollar wins, and the network topology dominates the cost more than the per-GPU rate.
• Fine-tuning runs. H100 / H200 spot capacity. Fine-tunes are checkpoint-tolerant, run-time-bounded, and the hourly-rate decline on Hopper is the largest single cost lever.
• Batch inference. B200 spot at $2.12/hr is the cost leader for high-throughput, latency-tolerant batch jobs. The cost-per-token math wins decisively.
• Latency-sensitive serving. B200 on-demand. The latency floor and the per-token throughput together justify the on-demand premium for user-facing serving where p99 latency matters.
• Checkpoint-friendly serving. B200 spot with intelligent failover to on-demand. The $0.15-per-million-tokens economics are real but require an inference engine that handles preemption gracefully (vLLM, SGLang, and LMDeploy all do, with caveats on how the KV cache is preserved across preemptions).

Outside those five, the workload-class decision is usually the wrong question — the right question is whether the workload should be running on a GPU at all, or whether it belongs on a CPU, an inference accelerator (Inferentia, Trainium, TPU), or a managed model API.

See the IAN team run on your cloud. We connect to your AWS account via a scoped read-only role, run the Observe-tier agents, and leave you with a concrete audit report — cost waste, security exposure, compliance gaps, and a labor-offset estimate. You keep the findings regardless of next steps. Get a free infrastructure audit →

What Cost Tooling Helps and Where It Falls Short

The 2026 GPU-cost tooling landscape covers parts of the picture well. CloudHealth, Vantage, Apptio Cloudability, FinOut, and the FOCUS-conformant rollups (per the FinOps Foundation FOCUS 1.3 spec ratified December 5, 2025) handle the per-account aggregation and the per-team chargeback question across hyperscalers. Provider-native tools (AWS Cost Explorer, GCP Recommender, Azure Cost Management) cover the per-resource view inside a single cloud.

Where the tooling is thin is on the workload-class decision itself. The cost dashboards tell you what you spent. They do not tell you whether the H100 capacity you reserved last quarter would be cheaper as B200 spot today, or whether the latency-tolerant batch job running on H200 on-demand should be moved to B200 spot for a 60% reduction.

Three things in particular are hard to wire up with off-the-shelf cost tooling:

• Continuous workload-class classification of every GPU job against the five-class matrix.
• Continuous price comparison across specialized providers, hyperscaler list, hyperscaler reserved, and hyperscaler spot for every workload class.
• Reversible-action automation to actually move the workload — provision the new capacity, migrate the checkpoint, drain the old capacity — when the cost delta exceeds the migration overhead.

Those three live in the gap between FinOps reporting and operational execution. They are the work of an active operational layer, not a passive cost dashboard.

Capability Tiers Mapped to GPU-Cost Optimization

The Observe / Operate / Administer capability-tier model applies cleanly to GPU-cost optimization:

• Observe. Continuous GPU inventory across every connected cloud account, FOCUS-normalized cost rollup, workload-class classification, cross-provider price comparison, and waste flagging (idle GPUs, oversized GPUs, mis-classed workloads). Auto-execute, fully audited.
• Operate. Reversible cost actions: provisioning new capacity at a cheaper provider tier, draining idle capacity, moving checkpoint-tolerant workloads to spot, right-sizing GPU SKUs against workload class. Auto-execute when in policy and reversible; gated when out-of-policy or hard to reverse.
• Administer. Reserved-capacity commitments, multi-year contracts with specialized providers, cross-cloud workload migrations that touch data residency, and GPU-budget allocation across teams. Always requires explicit approval; separation-of-duties enforced.

The mapping is what makes a cost agent deployable inside an organization with real procurement constraints. Agents do the continuous work; humans approve the structural commitments.

How IAN Helps: The Cost-Agent on the Active Operational Layer

IAN is the AI DevOps team for cloud infrastructure, delivered as a coordinated team of specialized agents on the active operational layer. The cost agent is one of those agents, and GPU-cost optimization is one of its primary specializations.

The cost agent runs continuous GPU inventory and workload-class classification across every connected cloud account. FOCUS-normalized billing data lands in the audit trail with cross-provider price comparisons attached. When the cost delta on a workload exceeds the configured threshold, the agent surfaces the recommendation in the team’s chat ops surface with the migration plan, the projected savings, and the rollback path attached.

Reversible Operate-tier actions — moving checkpoint-tolerant workloads to spot, draining idle capacity, right-sizing GPU SKUs — execute automatically when policy permits. Reserved-capacity commitments and multi-year contract changes escalate to the Administer tier with separation-of-duties enforced. Every action lands in the immutable audit trail, which doubles as the evidence pack for the FinOps and finance teams.

Pricing is BYOK and usage-based with a monthly minimum. Customers bring their own model keys (Claude, OpenAI, or another provider) and pay inference cost directly to their model vendor. IAN charges for the orchestration layer, per agent action, per cloud account, per operation class. Usage scales with the volume of GPU spend under management, which is itself a strong signal of where the team should invest in cost optimization.

The Three-Phase Rollout

Phase 1 — Stand up the GPU-cost Observe layer. Wire FOCUS-normalized billing ingestion, GPU inventory, and workload-class classification into the cost-agent’s read path across every connected cloud account. Two-to-four weeks for a team with reasonable cloud-cost capacity.

Phase 2 — Promote scoped Operate-tier actions. Let the cost agent move checkpoint-tolerant workloads to spot, drain idle capacity, and right-size GPU SKUs under capability-tier governance. Measure the cost-saving delta against the Phase 1 baseline. Two-to-three months.

Phase 3 — Run the cross-provider migration loop. Promote the cost agent to recommend cross-provider migrations (specialized provider vs hyperscaler, hyperscaler reserved vs hyperscaler spot) with the migration cost amortized into the recommendation. Keep the procurement decision gated to Administer-tier approval. Continue running the reversible Operate-tier actions in parallel.

The pattern compounds. By the time Phase 3 lands, the FinOps team is doing different work — fewer minutes producing dashboards on what was spent last month, more minutes setting policy on which workload classes should run on which provider tier — and the cost agent has accumulated enough audit data to make the next round of policy tuning a data exercise rather than a debate.

Get a free infrastructure audit → | See pricing →