FROM THE API TO THE SCREEN
Watch one real request.
Send a prompt. Follow the connection. Watch the answer arrive.
Checking local companion…Ready
Local Ollama endpoint; model inference runs in Ollama’s cloud.
REQUEST PATHWaiting for a request
Solid connections show the app’s configured route. Moving packets mark observed stream frames, not individual model tokens.
First content at bridge—
Elapsed at bridge—
Content chunks received0
Generated tokens · API reported—
Generation rate · API reported—
Observed events
What happens inside the model?
Conceptual explanation; the chat API does not reveal tokenizer IDs, individual layer execution, KV allocations, GPU utilization or exact prefill/decode boundaries.
Format + tokenize→Prefill + retain K/V→Decode + select↻Detokenize + stream
Prompt positions can be processed together within a layer. Layers remain sequential. Standard decode reuses attention history; gpt-oss uses a mixture-of-experts architecture. Stream chunks need not equal tokens. Runtime-reported durations may be absent for cloud models.
Run the same request in Python
No dependencies. With Ollama running and signed in, save this as sample_request.py and run python3 sample_request.py.
Presenter notes / live setup
Start with the route: local browser, local bridge, local Ollama, cloud model. Run the default prompt, point to first-content latency, then show the stream chunks and final API counters. Switch to the installed SmolLM2 model to demonstrate truly local model execution. Compare routes and observed timings without treating the two models as a quality-equivalent benchmark.
Launch the companion from the repository: python3 edge-inference/demo/server.py, then open http://127.0.0.1:8800/demo/. The public page can replay a captured real request; live requests require this localhost companion.