English
What a New Chip Benchmark Actually Tells You
Speed, throughput, elapsed time and energy are different measures. Read the system disclosure before translating a chart into a purchase.
IN THIS ARTICLE+
One chip completes a single job quickly. Another finishes more jobs over an hour. Both can lead a benchmark chart, depending on what the chart measures. When a processor announcement arrives, identify the metric before comparing the bars.
Speed and throughput answer different questions
SPEC CPU 2017 distinguishes speed and rate measures. A speed result focuses on time to complete a workload; a rate result describes work completed per unit time using copies of the workload. Check base versus peak tuning as well. A rate result from a many-core system does not establish that one interactive request will finish faster.
Read the disclosed system configuration: processor count, memory population, compiler, operating system and tuning flags. A comparison between a fully populated server and a memory-constrained baseline may be useful for a system purchase but cannot isolate the CPU's contribution.
Translate a percentage into the right unit
Suppose a hypothetical benchmark falls from 100 seconds to 80 seconds. That is 20 percent less elapsed time and a speedup of 1.25 times. Those are compatible descriptions, but neither establishes a 25 percent reduction in electricity use. Energy depends on power over time and the boundary of the measurement.
Look for performance at a comparable power limit or measure total system energy for the same completed work. Check whether the published result is measured or estimated. Record uncertainty where the disclosure does not answer your question.
Match the benchmark to your decision
If your service is limited by database locks or network round trips, a CPU-intensive benchmark is a poor predictor of end-to-end latency. Use the launch result to select a candidate, then run representative jobs with the same data, concurrency and service objective. Compare purchase and operating costs against the outcome you actually need.
Further reading
SPEC: CPU 2017 Run and Reporting Rules
Related guide: Measure Storage Latency Before Buying Faster Disks.
No comments yet. Share the first perspective.