Same chip conditions.
ESP32-D0WD-V3 and ESP32-S3, one core at 240 MHz. Latency uses device timing and medians; the main README describes 1,000 inferences.
Benchmarks
Evidence over promises.
Tradeoffs included.
Explore repository-reported measurements on ESP32-family hardware. Compare latency, memory and task score together, then inspect the method and source.
Inference latency
ms · lower is better01 / SYMBEX · BINARY40.7 ms
02 / TFLITE MICRO · INT852.0 ms
SYMBEX ACCURACY71.28 ± 0.81 %
INT8 ACCURACY87.0 %
Methodology review is ongoing. A review found test-set selection in some CDT runs and sizing studies. Fixes and affected re-measurements are ongoing. These are repository-reported results, not independently reproduced website measurements. Read the methodology.
Exact values & accessible data table
| Metric | SYMBEX · binary | TFLite Micro · INT8 |
|---|---|---|
| Latency (ms) | 40.7 | 52.0 |
| Model flash (KB) | 24.3 | 96.2 |
| Working RAM (KB) | 20 | 80.5 |
| Accuracy (%) | 71.28 ± 0.81 | 87.0 |
| PC agreement (%) | 100 | 98.1 |
Download versioned benchmark dataset ↓ · Missing measurements remain unavailable. Energy has not been measured.
HOW THE NUMBERS WERE PRODUCED
Conditions are
part of the result.
These measurements come from the project’s repository at revision b34f328. The website has not independently repeated the chip experiments.
Independent INT8.
TensorFlow Lite’s converter and TensorFlow Lite Micro produce the INT8 baseline in a separate environment. Matched architectures are identified; wider dense models are labeled.
Accuracy alongside speed.
Seed means and standard deviations appear where reported. CIFAR-10 uses classification accuracy; FOMO uses center-detection F1. They measure different tasks.
Definitions stay visible.
Model flash is the exported model footprint. Working RAM and a TFLite tensor arena are runtime-specific quantities. Weight payload is not total flash; table memory and placement may add overhead.
Check the integer model.
SYMBEX device results must match its integer PC simulation. INT8 agreement is measured against its TFLite PC interpreter. Neither is a dataset accuracy guarantee.
Keep the open questions.
Energy is not measured. Current work includes validation-split corrections, sensor workloads, more chip families and distillation. Missing measurements are never estimated here.
Validation-split review. Some historical CDT runs and sizing studies selected checkpoints or variants using the test set. The project documents this and is applying a held-out validation split with affected figures being re-measured. Fashion-MNIST and Digits are explicitly under re-measurement; resized convolutional studies carry their selection caveats.
ADDITIONAL STUDIES / KEEP THEIR CONTEXT
Beyond one model shape.
| Binary variant | Accuracy | ESP32-S3 latency | RAM |
|---|---|---|---|
| 16–32–64 · 2–2–2 blocks | 71.28 ± 0.81% | 40.7 ms | 20 KB |
| 16–32–128 · 1–1–2 blocks | 73.10 ± 0.23% | 39.6 ms | 15 KB |
| 16–64–128 · 1–1–2 blocks | 74.45 ± 0.23% | 46.6 ms | 16 KB |
These are binary-only variants from a sizing study, not additional matched INT8 comparisons. Selection methodology is under review. See the full study.
Raw arithmetic is a separate experiment.
The count benchmark reports 11–17× binary-vs-INT8 raw arithmetic on classic ESP32, 1.05–2.1× on ESP32-S3, and 1.5–3.8× on x86 with AVX2 / VNNI. These are kernel comparisons, not end-to-end model speedups.
REPRODUCE / DON’T JUST READ THE CHART
Start with the source.
BENCHMARK DIRECTORY ↗
CIFAR-10 / ResNet-8
Configuration, methodology, scripts and available results.
BENCHMARK DIRECTORY ↗Pascal VOC / FOMO
Configuration, methodology, scripts and available results.
BENCHMARK DIRECTORY ↗Fashion-MNIST / MLP
Configuration, methodology, scripts and available results.
BENCHMARK DIRECTORY ↗Digits / MLP
Configuration, methodology, scripts and available results.
BENCHMARK DIRECTORY ↗Human activity / 1D · in preparation
Configuration, methodology, scripts and available results.
BENCHMARK DIRECTORY ↗Kernel experiments
Configuration, methodology, scripts and available results.