Cheap GPU hours do not settle the rent-or-buy decision

An advertised GPU hour is a unit of available capacity. A business needs completed work, at an acceptable delay and quality, under terms it can use. The rent-or-buy calculation becomes useful only when it connects those two things through utilization, capacity requirements and operating responsibility.

Eugene Cheah's 11 October 2024 essay in Latent Space argued that falling H100 rental rates weakened the case for buying new hardware. His observations included short resale opportunities at $1–$2 an hour in August 2024. Those are dated observations from an inference-provider cofounder, not a verified current quote, a universal market average or terms available to every buyer. This November 2026 retrospective examines the decision structure rather than endorsing an investment forecast.

Separate observation from projection

Cheah's investment model included an assumed five-year hardware life and a rental-price decline across that period. Such assumptions can reveal sensitivity, but they do not become realized outcomes because a spreadsheet calculates a return. His market explanation is an interested participant's account, not an independent audit of every provider's economics.

A buyer should keep three columns conceptually separate: a dated offer with its conditions, a forecast about future prices, and a hypothetical workload. A cheap historical resale offer belongs in the first category. Expecting similar offers to continue belongs in the second. Assuming the company's jobs will occupy the GPU for most of the month belongs in the third.

Combining them into a single confident savings number hides where the uncertainty sits. If the calculation fails when one assumption changes, the decision needs that condition stated. An hourly price can be factual for a specific offer while the proposed annual savings remain entirely hypothetical.

Start with the work capacity must serve

Define the workload before comparing machines. A short batch job that can wait for a free slot has different requirements from a customer-facing service whose response time matters throughout the day. The first may tolerate interruption or irregular availability. The second may need spare capacity even when average demand is low.

Useful utilization is also different from a device reporting that it is busy. Time spent retrying a failed job or producing output that the business rejects still consumes capacity. Compare completed acceptable work over the period, with queueing and failure recovery included. Otherwise the apparently cheaper configuration may simply be delivering less.

For an illustrative internal batch process, write down the input size, completion deadline and permitted data location. Measure a representative job before extrapolating to a month. This is a proposed evaluation method; no GPU was rented, model benchmarked or business workload measured for this article.

The hardware label alone is insufficient. Memory, surrounding CPU and storage, interconnect and allocation rules can change whether a configuration fits the work. Ask for the actual available service and permitted configuration. Avoid assuming that two offers naming H100 provide interchangeable operating conditions.

Put idle time on the worksheet

Ownership commits money to capacity whether or not the work arrives. Include the acquisition and installation cost over an explicit planning period, plus power, cooling, space, maintenance and the people needed to operate it. Treat an assumed resale value as a scenario, not a guaranteed recovery of capital.

For rental, include the hours actually billed rather than only useful computation. Setup, data transfer, checkpoint recovery and time before shutdown may matter. Storage and traffic can be separate charges. A reserved contract can preserve a rate while leaving the buyer responsible for unused capacity. Read the actual conditions rather than assuming all rental is pay only when used.

A simple sensitivity exercise needs no invented market price. Hold the quoted terms fixed and compare low, expected and high workload cases using clearly labelled internal assumptions. Then vary the future workload or price separately. This reveals whether ownership depends on near-continuous demand or whether a rental commitment has recreated the same idle-capacity exposure.

Keep the time horizon consistent. Comparing a purchase spread over several years with one unusually cheap rental day gives a misleading answer. Compare the same required service period and include the possibility that the workload or suitable model changes before the purchased capacity is fully used.

Predictable demand can justify ownership

The strongest case for buying is sustained work on a known configuration, backed by the skills and facilities to run it. Existing infrastructure can change the incremental cost. Operational control or restrictions on where data may be processed can also narrow the permitted choices. Ownership may therefore be rational even when a casual hourly comparison favors rental.

Those constraints deserve their own assessment. Owning a device does not by itself establish confidentiality, and renting under a company account does not automatically meet a data-location requirement. Define the allowed infrastructure and access boundary first. Compare costs only among options that can actually serve the workload within it.

Conversely, rental can buy information as well as capacity. A bounded trial can establish whether the planned workload needs that hardware at all before a long commitment. Its value depends on measuring a representative task and keeping the trial's terms distinct from the terms of sustained operation.

Neither a historical price collapse nor a supplier's forecast establishes universal savings. The missing inputs for a real decision are the dated offer, enforceable conditions, measured workload and complete operating estimate. Without them, a recommendation to buy or rent is premature.

For a custom software project, connect infrastructure spending to the job the product must complete and the person responsible for operating it. Ask which assumption would reverse the decision, then keep that assumption visible as demand develops. Cheap capacity is useful when it fits the work; it is not a substitute for knowing how much work the business can use.