At inference time, a neural network is just a graph of operations plus a set of weights. Training needs the whole PyTorch ecosystem. Running, in theory, doesn't. So the question is simple: if I compile that graph into a native artifact and call it from C++, how much faster does it get, what dependencies are left, and do the outputs stay the same?
I had two reasons to dig into this. The first is Enty, a language I'm building and will write more about in another post. It will need a runtime to run models, and I wanted to know, with numbers, which paths exist before picking one. The second is plain curiosity: can you optimize a network by writing the engine by hand, in C++, and get close to the off-the-shelf runtimes?
The full experiment, with code, scripts and the report, is in the compiled-cnn repository.
2.94 ms
TensorRT FP16 on the RTX 3050: 340 FPS, 1.90× PyTorch
14.91 ms
OpenVINO INT8 on the CPU: 67 FPS, losing 0.31 mAP point
93%
of OpenVINO's speed with my own engine in FP32, no libraries at all
149 of 152
validations passed; the 3 failures have known causes
What was tested
The model is Ultralytics' YOLOv10s. I picked this version for a detail that makes comparison easier: its head already outputs the final detections, with no NMS. So every backend does exactly the same work, from pixel to box.
There were four rounds with existing compilers and runtimes, plus an engine of my own:
- Round 0, webcam: Ultralytics
predict, which is how most people run the model. The starting point. - Round 1, AOTInductor: PyTorch's own ahead-of-time compiler. It produces a
.pt2package that C++ loads through libtorch, with no Python. - Round 2, TensorRT: NVIDIA's compiler. ONNX becomes a
.planengine, and the C++ runner doesn't even use libtorch. - Round 3, CPU: PyTorch eager, ONNX Runtime and OpenVINO, in FP32 and INT8.
- Round 4, ncnn: over Vulkan, to see the path that doesn't depend on the GPU vendor.
- Own engine: a code generator that reads the ONNX and writes C++ with AVX2 kernels and INT8 with AVX-VNNI. No inference library, no BLAS.
Everything ran on one laptop: an RTX 3050 6 GB Laptop GPU and a Core i5-13450HX. Absolute numbers will differ on another machine. The comparisons, and the reasons behind them, carry over to hardware with the same features.
A few terms
- Eager vs compiled: in eager mode, PyTorch runs the graph one operation at a time, from Python. A compiler sees the whole graph up front, so it can fuse operations, pick the best kernel for each shape and drop the interpreter.
- ONNX: an open format to describe the graph and the weights. It's the bridge between PyTorch and almost every other path tested here.
- FP32, FP16 and INT8: ways to represent the numbers. FP16 uses half the memory and, on hardware that supports it, doubles throughput. INT8 uses 8-bit integers and requires quantization.
- Quantization and calibration: mapping real values to integers with a scale. The activation scales come from calibration: the network runs over hundreds of real images and records the value range of each tensor.
- AVX-VNNI: CPU instructions that compute 8-bit dot products in hardware. It's what makes INT8 fast on this machine.
How I know the outputs are the same
Measuring speed is easy. The hard part is making sure the fast backend computes the same thing. A runtime that's 2× faster and gets boxes wrong is useless.
- Fixed reference. Before any benchmark, I generated 8 reference images (two classic Ultralytics samples and six frames from my webcam), preprocessed once and run through PyTorch eager with TF32 off. Every backend reads the same files.
- Strict tolerances. FP32: boxes within 0.5 px and scores within 0.001. FP16: boxes within 2 px and scores within 0.005. In both, detections with score ≥ 0.25 must match in count, class and IoU ≥ 0.95.
- Pairing by IoU. Detections are paired by overlap, not by position in the list, because TopK may order near-ties differently.
- INT8 judged two ways. By agreement with FP32 (IoU ≥ 0.80 and a score difference ≤ 0.05) and by accuracy: COCO mAP50-95 may drop by at most 1 point, measured on 500 images that weren't used for calibration.
- Preprocessing checked. Each runner's
detectcommand reads the JPEG and does the letterbox in C++, so preprocessing gets tested too.
Tolerances were fixed before each round and never loosened. Every C++ runner has the same three commands (validate, bench and detect), and validate only exits with code 0 if every image passes.
How I measured
One forward pass, batch 1, a static 640×640 input already in device memory. The model alone, with no camera and nothing drawn on screen.
- GPU: 50 warm-up iterations and 500 measured ones.
- CPU: 20 and 200, with a 60 s cooldown before every run, 3 rounds with the order rotated between them and the median as the result. A load monitor discards and retries a run if another process uses the CPU.
- Threads pinned to the 6 P-cores with
taskset. On a hybrid processor, a thread that lands on an E-core makes the others wait at every layer.
The thermal protocol came from a real problem: the laptop heats up. In a back-to-back sequence, the P-core clock dropped from 4.2 to 3.1 GHz and penalized whatever ran last by up to 25%.
GPU: TensorRT leads
- FP32
- FP16
Latency in ms per inference, lower is better
One back-to-back session, unlocked clocks (±3% between runs). In FP16, TensorRT gives 1.90× over eager, AOTInductor 1.43× and ncnn 0.58×.
TensorRT in FP16 runs at 340 FPS and leaves the leanest runtime: the final binary only needs libnvinfer and the CUDA runtime. The cost is about a minute to build the engine, and that engine is tied to this GPU and this TensorRT version.
AOTInductor removes Python, but not libtorch. It's a good middle ground when the project already lives in PyTorch and the build needs to stay light.
Vulkan is portable, but it was far behind CUDA on this card, even when using the tensor cores through cooperative matrix. On the Intel integrated GPU it was even worse: about 230 ms in FP16. I stopped that benchmark on purpose, because it made no sense for this model.
CPU: INT8 is the big lever
FPS, higher is better (6 threads on the P-cores, median of 3 rounds)
FP16 doesn't help on this CPU. It has F16C, which only converts between FP16 and FP32; the math stays in FP32. There's no AVX-512 FP16 and no AMX.
What changes the game is AVX-VNNI. The vpdpbusd instruction does 32 8-bit multiply-adds per instruction, against 8 for an FP32 FMA.
The own engine
After measuring the ready-made runtimes, came the curiosity part: does an engine written from scratch, specialized for this graph, get close to OpenVINO?
How it works
The approach is a compiler's, not an interpreter's. An interpreter reads the graph on every run and decides what to do. My generator does that once, before compiling:
- it computes each tensor's shape and checks it against ONNX shape inference;
- it fuses convolution, SiLU and the residual add into a single call;
- it turns 15 concatenations into direct writes to the destination, with no copy;
- it plans every buffer in a single arena, reusing the space of tensors that won't be read again (370 down to 25 MB);
- it writes a
model.cppwith one call per operation and fixed shapes.
The weights come out already in the layout the kernels read and are embedded in the executable with .incbin. The C++ compiler optimizes each call for that layer's exact sizes. The final binary depends only on libc, libstdc++, libm and libgcc_s.
- Convolution as GEMM. Every convolution becomes a matrix multiplication. An AVX2 micro-kernel computes a block of 6 channels by 16 positions entirely in registers.
- Own threads. A pool with one thread per P-core, pinned by affinity. Between layers, they wait for each other by spinning instead of sleeping.
- INT8. 8-bit weights with one scale per channel, activations calibrated on 300 COCO images and the math done with
vpdpbusd. - Reference as oracle. The direct-loop version is still in the code and is the baseline for the tests of every optimized kernel.
From 21 seconds to 47 ms
The work went through five milestones, all validated with the same references and strict FP32 tolerances.
Latency in ms, log scale, lower is better
M1 to M4 are quick measurements without thermal control. M5 is the final benchmark.
| Milestone | What went in | Threads | Latency |
|---|---|---|---|
| M1 | reference direct loops | 1 | 21,692 ms |
| M2 | convolution as GEMM, AVX2/FMA micro-kernel | 1 | 578 ms |
| M3 | bias, SiLU and residual in the epilogue, copy-free concat, arena | 1 | 348 ms |
| M4 | thread pool pinned to the P-cores | 6 | 79 ms |
| M5 | packed panels, 6×16 kernel, per-layer grid | 6 | 46.67 ms |
Going from direct loops to GEMM gave 37×. Fusions, threads and packing took it from 79 to 47 ms. The last 10% to OpenVINO are the most expensive: they need convolution kernels at oneDNN's level.
Against OpenVINO
In the final benchmark, the engine ran in the same session as OpenVINO, so the comparison would be fair. I couldn't close the browser and the chat app for that measurement, so the absolute numbers are a bit higher than in the CPU section.
- Own engine
- OpenVINO
Latency in ms, lower is better (median of 3 rounds)
| Own engine | OpenVINO | Relative speed | mAP loss | |
|---|---|---|---|---|
| FP32 | 46.67 ms | 43.71 ms | ~93% | — |
| INT8 | 20.94 ms | 17.01 ms | ~81% | −0.26 vs −0.31 |
What this means for Enty
The best path changes with the hardware. TensorRT on an NVIDIA GPU, OpenVINO INT8 on an x86 CPU, ncnn when CUDA isn't an option. For Enty, that points to a backend built around an intermediate format (ONNX) and a native runtime that can be swapped per target, instead of tying the language to a single runtime.
And the own engine showed that a single executable, with no dependencies at all, reaches ~93% of OpenVINO in FP32. To close the INT8 gap, the next step is keeping activations in u8 between consecutive INT8 convolutions, quantizing in the producer's epilogue instead of in the consumer's packing.
Limits
- One machine, one GPU, unlocked clocks. Batch 1 and a static 640×640 input only.
- Equivalence, not accuracy. The 8 validation images check numerical equivalence. mAP was only measured for the CPU INT8 comparisons.
- Artifacts tied to the hardware. TensorRT engines and AOTInductor packages are specific to the GPU architecture.
- External load on the engine. The own engine was measured with the browser open. OpenVINO ran in the same session, which keeps the comparison fair, but the absolute numbers are a bit higher than they would be on an idle machine.
Code and reproduction
The whole experiment is at github.com/nicolasfmelo/compiled-cnn:
- C++ runners for AOTInductor, TensorRT, ncnn and the own engine, all with
validate,benchanddetect. - The engine generator, AVX2 and AVX-VNNI kernels, INT8 calibration and the tests for each kernel.
- Benchmark scripts with the thermal protocol, and the validation against the fixed reference.
- The full report in PDF and HTML (in Portuguese), plus a technical report per round.
The README has a step-by-step guide to rebuild everything from scratch, and you can run only the backends you care about. The CPU rounds and the own engine don't need any GPU-specific package.