CPU inference · PyTorch vs. Intel® OpenVINO™
How much faster does AI inference get when you compile it for the Intel stack?
Three production models, exactly as published, run on one CPU three ways. Nothing is retrained and the weights never change; only how the graph is compiled and what precision it runs in. Every number here is measured: latency, throughput, RAM, CPU load, accuracy, memory bandwidth and the machine kernels each layer executes on.
Three ways to run the same model
Every configuration uses identical inputs, identical preprocessing and the same thread budget, so the differences are purely the software stack.
Intel® oneAPI · OpenVINO™
Where Intel comes in
Everything on the optimized side of this comparison is Intel software, and the INT8 gains come from Intel instruction-set extensions. The baseline is stock PyTorch. These are the four pieces the study exercises.
OpenVINO™
Intel's open-source inference toolkit. It converts the PyTorch graph to OpenVINO IR, fuses convolution, bias and activation into single ops, folds constants, chooses blocked memory layouts and runs the result through its CPU plugin. The entire optimized column exists because of it.
oneDNN · oneAPI
Intel's Deep Neural Network Library, part of the oneAPI toolkit. It JIT-generates the machine kernels every layer executes on; the kernel profile on the results page names them (brgconv, brgemm, AVX-512). PyTorch links it for convolutions only; OpenVINO uses it for everything.
NNCF
Intel's Neural Network Compression Framework. Post-training quantization calibrated on 300 real samples turns FP32 weights and activations into INT8, cutting weight bytes 4× and unlocking the 8-bit dot-product instructions. It is the only step that can change predictions, so accuracy is re-measured.
VNNI & AMX
Intel instruction-set extensions for 8-bit math. AVX-512 VNNI does four multiply-adds per lane per instruction; AMX on 4th-gen Xeon® Scalable adds tiled matrix units at 16 per lane. The INT8 speedup measured here runs on the VNNI path.
The pipeline
One command runs the whole study; each stage is a separate module you can run on its own.
- 1
Load the published model
torchvision / Hugging Face weights, FP32, eval mode. This is also the baseline that gets benchmarked as-is.
benchmark/baseline.py - 2
Compile to OpenVINO IR
ov.convert_modelcaptures the graph, fuses Conv+bias+activation and matmul chains, folds constants, picks blocked memory layouts. Executed by oneDNN JIT kernels.optimized/convert.py - 3
Quantize to INT8 with NNCF
300 real training samples calibrate 8-bit ranges for weights and activations. The CPU plugin lowers them to VNNI / AMX dot products. Weights shrink 4x.
nncf.quantize - 4
Measure in isolation
Every model / configuration / batch runs in its own process: p50 / p95 / p99 latency, throughput, peak RSS, CPU time, accuracy on a fixed set.
benchmark/worker.py - 5
Profile and publish
Per-layer kernel profile, STREAM memory bandwidth, hardware detection, then one JSON file that the results page renders.
results/*.json
Which configuration should you use?
A recommendation per model, derived from the measurements. The same reasoning applies to any model of the same shape.
Models under test
One convolutional network, one Transformer, one edge network: the three shapes of workload a CPU inference stack has to handle.
Memory: the part of the story people skip
Inference on a CPU is usually bound by how fast bytes move, not by arithmetic. This section measures the machine's RAM and shows what each model asks of it.
What gets measured
Each number on the results page comes from one of these instruments.
Hardware at a glance
Detected automatically at benchmark time and written into the results file. The results page has the live view.
Results
See the full measurements
Latency, throughput, memory and CPU charts, accuracy after quantization, kernel profiles, and every raw number.