# Inside Edge Inference — Speaker Notes

Presenter script for all 21 chapters of [Inside Edge Inference](https://inside-edge-inference.vercel.app).

**Suggested pace:** allow 45–55 minutes for the full talk with animation pauses. For a shorter slot, cover the seven request phases and select the edge chapters relevant to your audience. The text is a talk track, not a script to rush through at the animation’s default speed.

**Opening seven phases:** use the original presentation’s phase tabs and Next / Previous controls. Select a diagram block for the original focus animation. The five narrative cues in these notes are teaching checkpoints, not a displayed five-step counter.

**Edge presenter controls:** pause the animation while explaining; use **Space** or **Step →** to advance; **R** replays. Chapter 21 has 21 beats; its numbered cues below match the displayed animation counter.

**Framing to establish once:** Part I is a conceptual cloud inference request. Part II follows a conceptual decoder-only LLM on a Qualcomm device. The transformer example is Llama-style. Exact processor placement and kernels depend on the model, export, runtime and Snapdragon target. The animations show dependencies, not measured execution times. Complete one transformer block before proceeding to the next; grouped animation beats do not imply doing QKV for all layers before attention. No device-specific kernel trace was captured.

---

## 01 — Query: A request enters the system.

[Open chapter](https://inside-edge-inference.vercel.app/#request-query)

### Talk track

Start with a user action: someone asks a question. The client packages the conversation, model selection and generation settings. In this cloud example, HTTPS carries that payload to a service. Gateway and admission policies vary, but the system must decide whether and where to execute the work. A busy replica can introduce queueing before any model arithmetic starts.

The worker can already be warm with weights resident. Sending a prompt does not retrain the model or require reopening the weights for every request. We will follow one admitted request first, then zoom out to concurrent serving. Later, we will move this same computation onto a Snapdragon device.

**Animation cues:** 01: The client packages messages and generation settings. 02: HTTPS carries the request to a service boundary. 03: The gateway checks credentials, limits and request format. 04: Routing selects a compatible model worker; admission may queue the request. 05: The admitted request enters the inference runtime.

**Transition:** Text becomes model input.

## 02 — Tokens: Text becomes model input.

[Open chapter](https://inside-edge-inference.vercel.app/#request-tokens)

### Talk track

The model does not consume the chat interface directly. A template converts role-tagged messages into the sequence expected by this model, including any special markers. Its tokenizer then converts the formatted sequence into vocabulary IDs. Token boundaries may span parts of words, punctuation or bytes. The pieces and IDs shown here are teaching examples, not a tokenizer trace.

The runtime prepares the appropriate tensor representation, positions and attention information. These details may be implicit in an optimized runtime. Embedding lookup selects learned vectors for token IDs. That is the bridge from integer symbols to the numerical hidden states consumed by the transformer.

**Animation cues:** 01: Begin with structured conversation messages. 02: Apply the model’s chat template and special tokens. 03: Tokenize the formatted sequence into vocabulary pieces. 04: Map pieces to integer token IDs; prepare positions and mask as needed. 05: Embedding lookup turns token IDs into model-width vectors.

**Transition:** Read the prompt before answering.

## 03 — Prefill: Read the prompt before answering.

[Open chapter](https://inside-edge-inference.vercel.app/#request-prefill)

### Talk track

Prefill is the prompt-processing phase. Within a transformer block, many prompt positions can be computed together using large matrix operations. This does not mean all layers execute independently. Each block receives the hidden states produced by the preceding block. Causal attention prevents a position from attending to future positions.

A Llama-style block includes normalization, QKV projections, attention, an output projection, residual updates and a feed-forward network. Kernels may fuse or reorganize these logical operations. Each layer produces keys and values for reuse. The final prompt position provides logits for the first generated token. Prefill is often compute intensive, but queueing, prompt length, chunking and hardware also affect time to first token.

**Animation cues:** 01: Embed the complete formatted prompt. 02: Project hidden states into queries, keys and values within a block. 03: Apply causal attention and its output projection. 04: Run the feed-forward network and residual updates; continue through later blocks. 05: Retain per-layer K/V state and use the final prompt position to score the first generated token.

**Transition:** Keep attention history, not answers.

## 04 — K/V: Keep attention history, not answers.

[Open chapter](https://inside-edge-inference.vercel.app/#request-cache)

### Talk track

Keys and values are intermediate attention state, distinct from the learned weights. Each layer maintains its own state for the positions it retains. During standard autoregressive decode, the new query attends to those retained keys and values. Processing the latest token adds its K/V state; the token selected from its logits will be processed on the next iteration.

This illustration includes a paged allocation strategy used by systems such as vLLM. Logical blocks need not be physically adjacent. Not every runtime pages its cache, and sliding-window or compressed caches change the retained state. The approximate dense-cache formula scales with layers, KV heads, positions, head dimension and bytes per element; actual allocations add overhead and may reserve extra capacity.

**Animation cues:** 01: Each attention layer contributes its own key and value vectors. 02: Store those vectors for the prompt’s retained positions. 03: A paged runtime maps logical token blocks to physical cache blocks. 04: Decode reads retained K/V and appends state for the token being processed. 05: Cache capacity limits concurrency; policies may evict, recompute or reuse eligible prefixes.

**Transition:** Generate one token, then repeat.

## 05 — Decode: Generate one token, then repeat.

[Open chapter](https://inside-edge-inference.vercel.app/#request-decode)

### Talk track

Prefill has already produced the scores used to choose the first output token. Decode begins by feeding that token into the model. The stack uses cached attention history and learned weights to produce scores for the next token. Each processed token contributes new keys and values to its layer caches.

The output head produces logits over the vocabulary. Selection can be greedy or use a configured sampling strategy. Temperature, top-k and top-p are options rather than mandatory operations. Once selected, the token can be sent to the output pipeline. Generation continues until a configured stop condition, cancellation or resource limit. For a small batch, moving weights and cache data often dominates; the animation timing is not a benchmark.

**Animation cues:** 01: Take the first generated token selected from prefill logits. 02: Embed it and run the sequential transformer stack. 03: Read retained K/V, append the processed token’s K/V, and reuse learned weights. 04: The output head produces vocabulary logits; greedy or sampling logic selects the next token. 05: Emit the selected token and repeat unless a stop condition has been reached.

**Transition:** Tokens become visible text.

## 06 — Stream: Tokens become visible text.

[Open chapter](https://inside-edge-inference.vercel.app/#request-stream)

### Talk track

A generated token ID must be converted back into text. Pieces can represent partial words or bytes, so an incremental decoder may buffer until it can emit valid text. The transport can also combine or delay chunks. Avoid promising one token, one word and one event.

A cloud API can use server-sent events, with JSON payloads containing content deltas. HTTP versions and protocol details vary. The browser renders arriving content while the model continues generating. Time to first model token, first response byte and first visible content are related but distinct measurements. Later, when we move onto the device, this network transport can become a local callback to the application.

**Animation cues:** 01: Generated IDs enter an output queue. 02: The incremental detokenizer joins vocabulary pieces into valid text. 03: The server frames and flushes content chunks through the configured protocol. 04: The client appends content as chunks arrive. 05: A completion signal closes the generation stream; local apps can use callbacks instead.

**Transition:** One request becomes a shared service.

## 07 — Serving: One request becomes a shared service.

[Open chapter](https://inside-edge-inference.vercel.app/#request-serving)

### Talk track

So far we followed one request. A service must manage many at once. Routing selects healthy compatible capacity. Admission prevents uncontrolled overload. The runtime scheduler balances prompt work, decode work and cache capacity. Continuous batching can let completed requests leave and new ones enter without waiting for an entire fixed batch to finish.

Quantization, prefix caching, parallelism, speculative decoding and prefill/decode disaggregation are possible optimizations, not a mandatory pipeline. Measure end-to-end latency, throughput, errors, cost and quality together. Now transition to the edge: the mathematics remains recognizable, but the runtime sits inside a device with shared RAM, battery and thermal limits. We will meet that hardware next.

**Animation cues:** 01: Route incoming work toward a healthy compatible replica. 02: Admit requests within memory and latency budgets. 03: A scheduler combines eligible work and manages cache allocations. 04: Model workers produce output while completed requests release capacity. 05: Measure queues, TTFT, token gaps, errors and quality; now move the request onto the device.

**Transition:** Next, meet the device hardware and its shared memory.

## 08 — Silicon: Meet the device

[Open chapter](https://inside-edge-inference.vercel.app/#silicon)

### Talk track

“Before we follow a request, let’s meet the hardware inside the device.

The CPU runs the application and coordinates execution. It handles general-purpose work, prepares inputs, and asks accelerators to run supported computation. The Adreno GPU handles graphics and can also execute model computation through an appropriate backend. The Hexagon NPU is designed to accelerate supported neural-network workloads.

These are different participants in one system. A request does not have to visit all three processors in sequence.

The large working set is backed by shared LPDDR: the phone’s system RAM. That includes model weights, temporary activations, attention-cache state, and memory used by other applications. Inside the accelerator, smaller local memories hold pieces of the work currently being processed.

Keep three terms separate: **weights** are the learned model parameters; **activations** are intermediate results for the current inputs; **KV cache** is attention state retained across token positions. Putting a model on the NPU does not mean the entire model permanently fits inside the NPU.”

**Animation cues:** reveal CPU, GPU and NPU individually. Pause when shared RAM appears. Point to local TCM last to contrast the large backing store with the smaller working area.

**Transition:** “Now let’s compare where this same request would run in the cloud.”

## 09 — Edge / cloud: Same model math. Different system

[Open chapter](https://inside-edge-inference.vercel.app/#edge-cloud)

### Talk track

“The transformer computation can be similar in both environments. What changes is the system around it.

For cloud inference, the app sends input over a network. A service receives and schedules that request, runs the model on server hardware, and sends output back. Servers can have larger accelerator memory, multiple devices, and opportunities to batch requests from different users.

For local inference, the app calls a runtime on the device. Once the required assets are available, generation can work without a server round trip. That supports offline experiences and can keep prompts on the device, provided the application itself is designed to do so.

The tradeoff is a tighter local budget. The model shares memory with other apps, uses battery power, and must stay within thermal limits. Removing network latency does not guarantee a faster overall answer if local computation is slower.

The choice depends on the required model quality, responsiveness, privacy, connectivity and resource budget.”

**Animation cues:** follow the upper network path first. Then follow the lower local path. Pause at the device’s budget block before discussing tradeoffs.

**Transition:** “To run locally, we first need to turn a trained model into something the device can execute.”

## 10 — Package: A model becomes a deployable artifact

[Open chapter](https://inside-edge-inference.vercel.app/#package)

### Talk track

“A file of weights is only part of a working language-model application.

We also need the computation graph, tensor shapes, tokenizer, configuration and compatible runtime assets. Export captures the operations we want to run. Optimization can change how those operations are represented. Quantization uses lower-precision representations where supported, potentially reducing storage and memory traffic, but it also requires quality validation.

The three branches here are alternative software routes. LiteRT-LM provides an LLM pipeline and uses packaged model assets, including target-specific NPU examples. ExecuTorch starts from a PyTorch export and can delegate supported work through its Qualcomm backend. Qualcomm’s GENIE library orchestrates optimized generative-model binaries through AI Engine Direct.

We choose a deployment stack; we do not pass the model through all three in sequence.

Compatibility matters. The model artifact, runtime libraries, SDK and device target must agree. A familiar file extension alone does not tell us that an artifact will run on every Snapdragon device.”

**Animation cues:** move from trained assets to export and optimization. Pause when the three branches appear and say “choose one route.” Finish at device storage.

**Transition:** “The assets are on the phone. What happens when the app opens them?”

## 11 — Load: From storage to a ready session

[Open chapter](https://inside-edge-inference.vercel.app/#load)

### Talk track

“This is preparation work before we generate tokens.

The CPU opens and validates the assets, initializes the runtime and backend, and establishes graph execution state. Weight and tensor data are read or mapped into the session’s memory. Mapping a file does not necessarily mean every page is immediately resident in RAM.

There are two possible preparation paths shown here. An ahead-of-time artifact already contains compatible compiled graph state that can be restored. Some runtime paths support compiling on the device instead. These are alternatives, not two mandatory steps that every model performs.

The runtime also prepares reusable input, output and cache buffers. Once the session is ready, we want to reuse it across generation steps.

That distinction matters for latency: cold start includes preparation costs. A warm request can reuse a prepared model session. We should not describe loading the model from flash as something that happens for every generated token.”

**Animation cues:** trace storage → CPU runtime → memory. Pause at the AOT/compile fork. Finish at “Session ready,” emphasizing reuse.

**Transition:** “A ready graph still leaves an important question: which processor executes each operation?”

## 12 — Partition: Who runs which operation?

[Open chapter](https://inside-edge-inference.vercel.app/#partition)

### Talk track

“Saying ‘this model runs on the NPU’ can hide several different execution arrangements. Partitioning belongs to export or compilation. With an AOT artifact, that work normally happened before loading; this chapter is not an extra mandatory step after restoring a context.

A transformer layer contains many operators. Backend support depends on the operator, tensor shapes, data types and implementation constraints. The compiler or delegate identifies compatible regions and turns them into accelerator subgraphs.

If an operation is unsupported, the runtime needs an explicitly available alternative path. That can split accelerator work around a CPU operation. In some configurations, export or execution fails instead of falling back.

Notice the embedding block. We have placed it on the CPU in this example to illustrate a boundary. That is not a universal rule: embedding lookup can be included in a delegated graph in other exports.

Also distinguish three levels: a **layer** is part of the model architecture, an **operator** is a mathematical operation, and a **kernel** is a target-specific implementation of work. Fusion or decomposition means these do not map one-to-one.”

**Animation cues:** inspect the original operator chain. Highlight the unsupported operation, then the separate HTP regions and the CPU boundary.

**Transition:** “Once processors share the work, they need a reliable way to share the data.”

## 13 — Buffers: Shared memory is a contract

[Open chapter](https://inside-edge-inference.vercel.app/#buffers)

### Talk track

“Shared physical memory does not mean every processor can use every buffer immediately, with no coordination.

The CPU has a view of an allocation. The accelerator runtime needs a suitable representation of that memory, often through registration and tensor handles. Different processors may refer to the same physical storage using different addresses or handles.

Where supported, shared buffers can avoid a separate host-to-accelerator copy. That is valuable, but ownership, visibility and completion still matter. The CPU must not consume output before the accelerator has finished producing it.

There can also be layout or cache-maintenance work, depending on the runtime’s memory contract.

Finally, sharing system RAM does not eliminate the accelerator’s internal memory hierarchy. The working tiles still move into smaller local memory for efficient computation. We need to think about both coordination between processors and data movement within the accelerator.”

**Animation cues:** follow CPU view → physical allocation → QNN handle. Then follow the tiles into local TCM. Pause at the completion boundary before returning to the CPU.

**Transition:** “With the session and buffers ready, we can finally accept a user’s prompt.”

## 14 — Prompt: Text becomes model input

[Open chapter](https://inside-edge-inference.vercel.app/#prompt)

### Talk track

“The model does not directly consume the string that the user typed.

The application first applies the model’s chat format, including any required role markers and special tokens. The tokenizer converts that formatted text into vocabulary IDs. A token can be a word, a word fragment, punctuation, or another tokenizer-specific unit.

The runtime then prepares the model inputs: token IDs and the sequence information required by the exported graph, such as positions and attention masks.

Embedding lookup selects learned vectors corresponding to those IDs. This converts discrete symbols into the numerical hidden representations used by the transformer. This animation selects host embedding, matching the final overview. The lookup can instead be part of a delegated graph in other exports.

The token numbers in the picture are examples, not actual output from a particular tokenizer. In a real deployment, using the correct tokenizer and chat template is part of model correctness.”

**Animation cues:** reveal the user message, tokenizer pieces, input tensors and embedding rows in order. Pause before dispatching prefill.

**Transition:** “Those input vectors now pass through the transformer stack. This first pass is prefill.”

## 15 — Prefill: Through one transformer layer

[Open chapter](https://inside-edge-inference.vercel.app/#prefill)

### Talk track

“Prefill processes the prompt positions through the model. Within a layer, many positions can be handled together, while causal attention prevents a position from using future tokens. The layers themselves still form a sequence.

In this Llama-style block, RMSNorm first rescales the hidden representation. Learned projections produce queries, keys and values. Queries and keys are used to determine attention scores; values carry the information combined using those scores.

RoPE adds positional information by rotating components of queries and keys. Attention then combines the scores with the causal mask, applies softmax, and uses the resulting weights to combine values.

The attention result passes through an output projection and a residual connection. Another normalization feeds a gated feed-forward network, followed by another residual connection. We repeat the complete block through the stack.

After the final layer, final normalization and the language-model head produce vocabulary scores. For generation, the final prompt position supplies the scores used to select the first output token.

Prefill also establishes the prompt’s per-layer KV state. Processing multiple positions can offer more weight reuse than single-token decode, though the actual bottleneck depends on the workload.”

**Animation cues:** pause at each of the six beats: normalization/QKV; RoPE/cache; attention; output/residual; FFN; repeat/final head. Give attention the longest pause. Explain the causal triangle rather than reading the formula aloud.

**Transition:** “Those are logical model operations. Let’s zoom in on how the accelerator implements the work.”

## 16 — Kernels: Inside the NPU—move less, reuse more

[Open chapter](https://inside-edge-inference.vercel.app/#kernels)

### Talk track

“A large matrix operation usually cannot keep all of its inputs and intermediates in local accelerator memory at once. The implementation breaks it into smaller tiles.

Weight and activation tiles are brought from shared RAM into the local working area. Matrix execution performs multiply-accumulate operations: multiply corresponding values, then accumulate partial results. Reusing data already nearby reduces repeated traffic to the larger memory pool.

Other supported work, such as elementwise operations and reductions, uses appropriate vector or scalar execution. The compiler may fuse neighboring operations or overlap data transfers with computation when the implementation permits it.

This is why accelerator performance is not only about arithmetic capacity. A compute unit cannot help much while it is waiting for data.

The animated grid illustrates those principles. It does not represent the physical number of units in a particular Snapdragon chip, or a measured QNN kernel schedule. For exact kernel names, placement and timings, we need the compiled model and a device profile.”

**Animation cues:** follow tiles into TCM, then watch the active matrix cells. Pause on reuse before following the writeback path.

**Transition:** “One kind of data has to survive beyond this computation: the attention cache.”

## 17 — KV cache: Remember the past. Append the present

[Open chapter](https://inside-edge-inference.vercel.app/#kv-cache)

### Talk track

“The KV cache stores keys and values from earlier positions, separately for the relevant layers. Decode can use that state instead of recomputing the earlier positions’ K/V from scratch.

It is not the model’s weights, and it is not a store of every intermediate activation. It is specific attention state that grows or is managed according to the model and cache policy.

For a conventional dense cache, a useful estimate is two times the number of layers, batch size, KV heads, cached positions, head dimension and bytes per element. The factor of two represents keys and values. With grouped-query attention, use the KV-head count, not the larger query-head count.

Our illustrative configuration needs 48 KiB per cached position: at 2,048 positions, that is 96 MiB before overhead. Move the slider to 8,192 and it becomes 384 MiB.

That shows why fitting the weights is not the whole memory budget. Longer contexts also require cache capacity and attention reads. These numbers estimate payload for the chosen cache capacity. A static cache may allocate its full capacity up front, so more valid tokens do not necessarily cause a new allocation. Padding, quantization, sliding windows and allocation policy can change the actual footprint.”

**Animation cues:** point to separate layer rows and K/V pairs. Show earlier positions being read, then a new position appended. Open the notes and move the context slider from 2,048 to 8,192.

**Transition:** “Now we can generate new tokens while reusing that stored state.”

## 18 — Decode: One new token. Then again

[Open chapter](https://inside-edge-inference.vercel.app/#decode)

### Talk track

“The first output token comes from the prefill result. Once that token is selected, we feed its ID into the next model pass to predict the following token.

In this conventional batch-one decode loop, each iteration processes the newest position through the layers. Attention reads earlier KV state and adds the current position’s state. The final head produces vocabulary scores, and the sampler chooses the next token.

Sampling may choose the highest-scoring token or use a probability-based policy. The selected ID is incrementally decoded and any available text is delivered to the app. Some tokens produce no immediately displayable characters; byte fragments can require buffering, and special tokens may be suppressed. This animation adds one illustrative piece per pass.

Notice the memory behavior. We reuse the loaded model session, but the computation still reads weight data through the memory hierarchy. With one new position at a time, weight reuse is often lower than during prefill, so memory bandwidth can become especially important.

There is also a timing detail: a newly sampled token can be displayed immediately. Its own K/V is produced when it is fed through the next model pass. Sampling and computing that token’s internal state are different events.”

**Animation cues:** pause at the weight/cache inputs, then sampling and local text output. Follow the return wire. Emphasize that it does not return to model loading.

**Transition:** “That loop is repeated many times, which makes repeated processor hand-offs worth examining.”

## 19 — Hand-offs: The cost lives at the boundary

[Open chapter](https://inside-edge-inference.vercel.app/#handoffs)

### Talk track

“Compare the same logical work in two execution arrangements.

The upper path is a contiguous supported NPU region. The compiler can potentially optimize across that region. In the lower path, a CPU operation splits the accelerator work into two parts.

At a boundary, the runtime may need to submit work, wait for completion, establish memory visibility, and possibly convert the tensor’s layout or precision. Sharing the underlying allocation can avoid a copy, but it does not automatically remove these other costs.

A precision conversion is required only when the formats on the two sides require it. It is not an unavoidable tax on every CPU/NPU crossing.

The timing sum on this slide describes a serial path. If execution and transfers overlap, wall-clock latency follows the critical path rather than the sum of all measured work. The practical question is whether the complete accelerated path saves time and energy after including its overhead. More NPU execution does not automatically mean a better result. For a particular small model or short workload, another processor can be competitive.”

**Animation cues:** show the contiguous path first. Pause at each red boundary in the fragmented path. Point to the total-time expression at the end.

**Transition:** “That is why we need to measure the whole request, not just its fastest kernel.”

## 20 — Measure: Ship the experience. Measure the system

[Open chapter](https://inside-edge-inference.vercel.app/#measure)

### Talk track

“Let’s turn the explanation into a practical evaluation plan.

Separate cold-start initialization from a warm request. Define exactly where your time-to-first-token measurement starts; it may include prompt preparation and queueing as well as model computation. Then measure decode speed and the gaps between visible tokens.

Track peak memory for the complete application, not just the weights. Look at energy use and temperature over repeated requests. A short burst on a cool device may not describe sustained behavior.

Record the model revision, tokenizer, quantization, runtime, SDK, device and input/output lengths. When comparing CPU, GPU and NPU, also compare output quality rather than assuming different precision choices are equivalent.

Finally, inspect the partitioned graph and profile the actual device execution. That is how we learn which operations stayed on the host, which kernels ran, and where time was spent.

The goal is a model that fits, responds consistently and remains useful under realistic device conditions.”

**Animation cues:** step through cold start, first token, decode, resources and profiling. Ask the audience to name one metric they would miss by reporting only tokens per second.

**Transition:** “Let’s finish by following one complete request, with all these pieces connected.”

## 21 — Full flow: See the whole request

[Open chapter](https://inside-edge-inference.vercel.app/#overall)

### Opening talk track

“This is the complete journey. The upper row prepares the session and shared memory. The middle row shows the selected CPU and Hexagon path. The bottom row shows the prompt, request state and streamed response.

We will produce three illustrative pieces: ‘on’, ‘ device’, and ‘.’. Watch what changes between prefill and decode, and what stays resident. The cache cells summarize state across layers; they are not an exact model layout.”

### Cue-by-cue narration

Use **Step →** to keep the explanation aligned with the animation counter. For a short recap, group the cues as **01**, **02–08**, **09–14**, **15–21**.

| Beat | What to say and point to |
| --- | --- |
| 01 / 21 | “Prepare the session once. The CPU opens compatible model assets, prepares graph execution and binds shared buffers. The GPU is shown as an optional participant, not a required hop.” Point across the upper row. |
| 02 / 21 | “Apply the chat template and tokenize this prompt.” Follow the user input into the tokenizer. |
| 03 / 21 | “Look up the prompt’s embedding vectors and prepare the prefill inputs. This example shows host embedding; placement can differ in another export.” |
| 04 / 21 | “Normalize and compute Q, K and V using the loaded weights.” Point to both the weight-memory path and the first NPU block. |
| 05 / 21 | “Apply position information and causal attention. Retain the prompt’s keys and values for each layer.” Point out the cache filling. |
| 06 / 21 | “Finish attention’s output projection and residual, then the normalized gated FFN and its residual. Repeat the complete block through all layers.” Trace the layer bracket; it does not mean only the FFN repeats. |
| 07 / 21 | “After the stack, final normalization and the LM head produce vocabulary scores for the next token.” |
| 08 / 21 | “The CPU samples and displays the first piece: ‘on’. Prefill has produced the first visible output.” Pause at the response. |
| 09 / 21 | “Feed that sampled token back into embedding. We do not tokenize the original prompt again, and we do not reload the model.” Follow the blue return wire. |
| 10 / 21 | “Run projections for the newest position using the same learned weights.” |
| 11 / 21 | “Read earlier KV state and append this position’s K/V. The previously sampled token now has its own computed attention state.” Point to the added cache column. |
| 12 / 21 | “Complete the layer stack, including attention projection, residuals and feed-forward work.” |
| 13 / 21 | “The final head produces the next set of vocabulary scores.” |
| 14 / 21 | “Sample and display ‘ device’. The app now shows ‘on device’.” |
| 15 / 21 | “Begin another decode pass with the newest token ID. The loaded weights and earlier cache state remain available.” Trace the return wire again. |
| 16 / 21 | “Within each layer, compute this position’s Q, K and V. Finish its attention and FFN before moving to the next layer.” |
| 17 / 21 | “Attention reuses the earlier cache and retains another position’s state.” Point to the next cache column. |
| 18 / 21 | “Finish the remaining transformer work through the stack.” |
| 19 / 21 | “Produce scores for the next output token.” |
| 20 / 21 | “Sample and display the period. The three illustrative pieces form ‘on device.’” |
| 21 / 21 | “The configured three-token limit ends this example. A real request can stop on EOS or another rule. The model session can stay prepared, while request-specific state is managed for the next interaction.” |

### Closing talk track

“The CPU coordinates the application, the NPU accelerates supported model computation, and shared memory holds the larger working set. The model is prepared once; generation repeatedly combines new input with loaded weights and cached attention state.

That is what makes edge inference an engineering problem across the whole system. We need the model to fit, the operations to map well, the data to move efficiently, and the experience to stay responsive within the device’s power and thermal budget.”

---

## Presenter reference links

Opening request phases: [Chat templates](https://huggingface.co/docs/transformers/main/en/chat_templating), [Text generation](https://huggingface.co/docs/transformers/main/en/llm_tutorial), [Paged attention](https://docs.vllm.ai/en/latest/design/paged_attention/), [Chat API](https://developers.openai.com/api/reference/resources/chat).


These are the primary references already associated with the presentation. Keep them for questions; they do not need to be read aloud.

- **Hardware and execution principles:** [Qualcomm AI Engine](https://www.qualcomm.com/processors/ai-engine); [Hexagon-MLIR paper](https://arxiv.org/html/2602.19762v1).
- **Deployment stacks and loading:** [Qualcomm GENIE](https://www.qualcomm.com/developer/blog/2024/12/how-qualcomm-gen-ai-inference-extensions-enable-npu-gen-ai-acceleration-ai-hub); [ExecuTorch Qualcomm backend](https://docs.pytorch.org/executorch/stable/backends-qualcomm.html); [LiteRT-LM on Qualcomm NPU](https://developers.google.com/edge/litert/next/litert_lm_npu); [LiteRT Qualcomm compilation](https://developers.google.com/edge/litert/next/qualcomm).
- **Operator placement, buffers and profiling:** [LiteRT QNN compiler mappings](https://github.com/google-ai-edge/LiteRT/blob/761d99cb90e20c67efcb3fe1119a60c92381bd1a/litert/vendors/qualcomm/compiler/Qualcomm_QNN_Compiler.md); [ONNX Runtime QNN execution provider](https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html).
- **Transformer and cache behavior:** [Meta Llama model implementation](https://github.com/meta-llama/llama/blob/689c7f261b9c5514636ecc3c5fefefcbb3e6eed7/llama/model.py); [Attention Is All You Need](https://arxiv.org/abs/1706.03762); [LiteRT-LM NPU executor](https://github.com/google-ai-edge/LiteRT-LM/blob/007e7c760efc3dae5c1d628ee8f28053ae149468/runtime/executor/llm_litert_npu_compiled_model_executor.h).

**Acknowledgements:** Animation direction inspired by Rudraksh Karpe. Earlier presentation inspiration from Suman Debnath and Linda Haviv.