# Inside Edge Inference — technical review

Reviewed 7 September 2026. Baseline: Git commit `f748c12` and the live Vercel presentation.

## Conclusion

The revised presentation is a defensible **conceptual introduction to a Llama-style decoder-only LLM using a selected Qualcomm NPU execution path**. The model mathematics, distinction between system RAM and local accelerator memory, cache-size arithmetic, and basic deployment explanation are supported by the references below. Several animation and wording ambiguities were corrected during this review.

This is a documentation and source-code review, not a Qualcomm certification or a measurement of a particular Snapdragon device. No model artifact, QNN operator trace, device benchmark, or numerical output was executed or validated. Exact placement, fusion, precision, cache layout, and performance remain properties of a specific model/export/runtime/SoC combination.

## Coverage and method

- Reviewed all 14 chapter descriptions, facts, takeaways, inline notes, inspector text, diagram labels, and animation implementation.
- Stepped through the live presentation's 88 numbered animation beats, including the 21-beat final overview. Reviewed processor colors, connections and the token/cache state sequence against the source.
- Reviewed all 14 full speaker scripts and the overview's 21 narration cues.
- Read Qualcomm publications, Google AI Edge documentation, ExecuTorch and ONNX Runtime documentation, and pinned implementation sources. Recalculated the cache example independently.
- Rechecked the corrected browser diagrams and synchronized generated phone notes. The animation is explanatory, not a representation of measured timing or physical compute-unit count.

## Chapter-by-chapter result

| # | Chapter | Result and qualification |
|---|---|---|
| 01 | Silicon | Supported at the architectural level. CPU, Adreno and Hexagon have distinct roles; shared LPDDR is not NPU-local TCM. The three devices are not obligatory consecutive stages. [1, 2] |
| 02 | Edge / cloud | Supported as a comparison of typical deployments. Offline use assumes provisioned assets and an application that does not require remote services. Lower end-to-end latency and privacy are not automatic guarantees. [1, 3] |
| 03 | Package | The three software routes are alternatives. Corrected processor-coded colors and connected every route to device storage. GENIE's color must not imply GPU execution. Artifact compatibility remains target-dependent. [3, 4, 5] |
| 04 | Load | Supported. Restoring a compatible AOT context and compiling on-device are alternative supported preparation paths; mapping files does not establish full physical residency. No per-token model-file reload is implied. [4, 5] |
| 05 | Partition | Corrected sequencing language: partitioning occurs during export/compilation, not necessarily after loading. Corrected the custom-op connector. Supported-op mappings do not prove every shape/type works, and fallback requires a configured implementation. [4, 6] |
| 06 | Buffers | Supported with runtime-specific caveats. Shared allocations can avoid copies, but binding, visibility and completion still matter. Local tile movement remains separate from host/accelerator sharing. [2, 7, 8] |
| 07 | Prompt | Supported. Made host embedding consistent with the final overview; other exports may delegate it. Token IDs, chat formatting and mask/position inputs are illustrative and model-specific. [5, 6, 9] |
| 08 | Prefill | Supported for the stated Llama-style architecture. Clarified both residual bypasses and their reveal timing. Final norm/head occur after the block stack. Grouped beats do not reorder work across layers. [9] |
| 09 | Kernels | Supported as a conceptual tiling example. The Hexagon-MLIR paper supports memory-hierarchy and reuse principles, not a claim that this exact compiler or kernel schedule is used by every QNN deployment. [2, 6] |
| 10 | KV cache | Arithmetic verified: 48 KiB/position; 2,048 positions = 96 MiB; 8,192 = 384 MiB. Clarified capacity versus valid occupancy and the current input position's state. Static allocations need not grow on every token. [9] |
| 11 | Decode | Corrected the animation to emit one illustrative new text piece per pass, made embedding/final norm explicit, and removed the late cache-append caption. Added text-buffering caveats. This is conventional batch-one decoding, excluding speculative variants. [9, 10, 11] |
| 12 | Hand-offs | Supported as a boundary-cost explanation. Qualified the additive timing formula as a serial-path model; overlapped execution requires critical-path accounting. Conversion is conditional, not required at every boundary. [4, 7] |
| 13 | Measure | The proposed evaluation dimensions are appropriate. Removed processor colors from measurement categories to avoid implying that steady decode is CPU-only or thermal behavior is GPU-only. There are no measured results here. [7] |
| 14 | Full flow | Token/cache sequence checked: four illustrative prompt positions, then two consumed generated positions; the third generated token is displayed and the request stops before consuming it. Reinforced full-block ordering and separated session retention from request-state reuse. [9, 10, 11] |

## Material corrections

1. **Software-stack colors and connections.** The global legend gives blue/orange/green processor meanings. Package routes now use neutral colors, and all three routes connect to deployment. Neutral measurement categories prevent the same ambiguity in Chapter 13.
2. **One decode pass, one sampled ID.** The original local-stream animation revealed four successive sentence prefixes after a single sampler step. The revised view adds one illustrative piece. Multiple generated IDs require further forward passes in the conventional loop shown.
3. **KV timing.** Current input K/V belongs to that input's transformer pass. The next sampled ID gets its own K/V when consumed later. The final decode caption no longer puts an unspecified append after text output; attention also explicitly includes the current permitted position.
4. **Residuals and omitted operations.** The first skip path explicitly bypasses the first normalization and attention. The second skips the normalized FFN. Decode now labels embedding and final normalization, while the full-flow bracket clarifies that the entire block repeats.
5. **Preparation order and timing.** Partitioning is not a mandatory step after restoring an AOT context. The hand-off time sum is explicitly serial, rather than a universal wall-clock identity.
6. **Memory and text granularity.** The cache calculator estimates dense capacity, not necessarily live allocated growth. Token output may need incremental byte buffering or special-token suppression.
7. **Stable evidence.** Implementation links point to reviewed commits; vendor/framework documentation remains rolling. The public scope text identifies the conceptual model family and variable placement.

## Primary evidence

1. [Qualcomm AI Engine](https://www.qualcomm.com/processors/ai-engine): heterogeneous Snapdragon compute.
2. [Hexagon-MLIR, Qualcomm-affiliated authors, v1](https://arxiv.org/html/2602.19762v1): local TCM, shared DDR, tiling, vector/matrix work, fusion and asynchronous transfers. This is a research compiler paper, not a QNN kernel trace.
3. [Qualcomm GENIE integration](https://www.qualcomm.com/developer/blog/2024/12/how-qualcomm-gen-ai-inference-extensions-enable-npu-gen-ai-acceleration-ai-hub): generative model orchestration through AI Engine Direct; [Qualcomm ExecuTorch article](https://www.qualcomm.com/developer/blog/2025/10/bringing-edge-ai-performance-to-pytorch-developers-with-executorch-1-0).
4. [ExecuTorch Qualcomm backend](https://docs.pytorch.org/executorch/stable/backends-qualcomm.html): compilation, selected backends, deployment and compatibility.
5. [LiteRT Qualcomm compilation](https://developers.google.com/edge/litert/next/qualcomm) and [LiteRT-LM NPU deployment](https://developers.google.com/edge/litert/next/litert_lm_npu): AOT/on-device alternatives and SoC-specific model/runtime assets.
6. [LiteRT QNN compiler mappings, reviewed revision](https://github.com/google-ai-edge/LiteRT/blob/761d99cb90e20c67efcb3fe1119a60c92381bd1a/litert/vendors/qualcomm/compiler/Qualcomm_QNN_Compiler.md): operator/type mappings; not universal hardware support for every mapped type.
7. [ONNX Runtime QNN provider](https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html): shared-memory allocator, graph-I/O quantization and profiling. Its options describe that integration, not every Qualcomm stack.
8. [LiteRT Qualcomm dispatch, reviewed revision](https://github.com/google-ai-edge/LiteRT/blob/761d99cb90e20c67efcb3fe1119a60c92381bd1a/litert/vendors/qualcomm/dispatch/litert_dispatch_invocation_context.cc): tensor-buffer types and graph binding.
9. [Meta Llama model, reviewed revision](https://github.com/meta-llama/llama/blob/689c7f261b9c5514636ecc3c5fefefcbb3e6eed7/llama/model.py): architecture, residuals, position handling and cache behavior. This supports model mathematics, not Qualcomm placement.
10. [Meta generation loop, reviewed revision](https://github.com/meta-llama/llama/blob/689c7f261b9c5514636ecc3c5fefefcbb3e6eed7/llama/generation.py): forward/sample/feedback/stop sequence.
11. [LiteRT-LM NPU executor, reviewed revision](https://github.com/google-ai-edge/LiteRT-LM/blob/007e7c760efc3dae5c1d628ee8f28053ae149468/runtime/executor/llm_litert_npu_compiled_model_executor.cc) and [settings](https://github.com/google-ai-edge/LiteRT-LM/blob/007e7c760efc3dae5c1d628ee8f28053ae149468/runtime/executor/llm_executor_settings.h): conventional decode, greedy-sampling option, masks/cache configuration. Options and signatures do not establish all-model placement.

The direct Qualcomm HTP backend documentation endpoint did not remain reliably retrievable during this review. No unsupported device-specific conclusions were inferred from it; memory principles and integration claims were checked against the accessible sources above.

## How to introduce it to the Qualcomm group

“This is an independent visual explanation of a Llama-style inference request on a selected Qualcomm NPU path. The diagrams illustrate logical dependencies and memory behavior. They are not a captured trace of a particular model or Snapdragon build; exact placement and performance require that build's profiling data.”