docs(ollama): explain tool prompt cache reuse

2026-07-20 17:47:04 +08:00
parent 8423cf3eeb
commit 9d830fb6b6
4 changed files with 254 additions and 1 deletions
@@ -0,0 +1,239 @@
+# How to Improve Ollama Tool-Calling Prompt Cache Reuse in nanobot
+
+Some Ollama model templates move or remove tool definitions as a conversation
+switches between user, assistant, and tool messages. nanobot can send a correct
+append-only chat request while the model template still renders a different token
+prefix. On slower local hardware, re-evaluating that prefix can add tens of seconds
+to an otherwise simple tool-using turn.
+
+This guide shows how to diagnose that specific pattern and create a derived
+`llama3.1:8b` tag with a prefix-stable tool template. It does not modify nanobot or
+overwrite the original Ollama model.
+
+## What you will build
+
+- a repeatable two-turn cache check
+- an optional derived `llama3.1:8b-prefix-stable-v1` Ollama tag
+- a nanobot model preset that uses the derived tag
+
+## When to use this
+
+Use this guide when all of the following are true:
+
+- direct Ollama responses are reasonably fast;
+- nanobot becomes slow after the model calls a tool;
+- Ollama logs show a long main prompt, a much shorter tool follow-up, and low
+  initial cache reuse on the next main prompt;
+- the model is `llama3.1:8b` with a template that renders concrete tools only for
+  the final user message.
+
+Do not apply this template to another model family without checking that model's
+tool-call format first.
+
+## Diagnose the rendered prompt
+
+Stop any existing Ollama process, then start a single-slot debug server. A single
+slot makes the cache sequence easier to read.
+
+**macOS or Linux**
+
+```bash
+OLLAMA_CONTEXT_LENGTH=16384 \
+OLLAMA_NUM_PARALLEL=1 \
+OLLAMA_DEBUG=1 \
+ollama serve
+```
+
+**Windows PowerShell**
+
+```powershell
+$env:OLLAMA_CONTEXT_LENGTH = "16384"
+$env:OLLAMA_NUM_PARALLEL = "1"
+$env:OLLAMA_DEBUG = "1"
+ollama serve
+```
+
+In another terminal, use a fresh session and explicitly request a tool so both
+turns exercise the agent loop:
+
+```bash
+nanobot agent --session cli:ollama-cache-check \
+  --message "Use the exec tool to calculate 2+2, then answer"
+nanobot agent --session cli:ollama-cache-check \
+  --message "Use the exec tool to calculate 4+7, then answer"
+```
+
+In the Ollama output, find each `new prompt` line and the first
+`cached n_tokens` line that follows it. Later increasing `cached n_tokens` lines
+are prompt-evaluation progress, not additional initial cache hits.
+
+A cache-unfriendly tool template may produce a pattern like this:
+
+```text
+turn 1 main:             2 / 8460 initially cached
+turn 1 tool follow-up: 3713 / 3758 initially cached
+turn 2 main:          3767 / 8519 initially cached
+```
+
+The cache is working, but the next main request can reuse only the shorter prompt.
+Hardware throughput determines how expensive the remaining evaluation is.
+
+To inspect the API request bodies as well, add
+`OLLAMA_DEBUG_LOG_REQUESTS=1` before starting Ollama. These logs can contain system
+prompts, workspace context, and user messages. Keep them local and disable request
+logging after diagnosis.
+
+## Why this happens with the stock template
+
+The tested `llama3.1:8b` template conditionally expands the tool definitions inside
+a user message:
+
+```gotemplate
+{{- if and $.Tools $last }}
+  ... render tool definitions ...
+{{- end }}
+```
+
+The first request ends with a user message, so the tools are rendered there. After
+nanobot appends an assistant tool call and its result, that user message is no
+longer last, so the same API request history renders without the concrete tool
+block. On the next user turn, the tools reappear at a new position.
+
+This is a model-template behavior. At the API boundary, nanobot continues to append
+the assistant tool call and tool result and sends the same tool definitions.
+
+## Create a prefix-stable derived model
+
+Create `PrefixStable.Modelfile` with the content below. The template keeps concrete
+tool definitions in the system block, where they remain in the same position across
+user and tool messages.
+
+```dockerfile
+FROM llama3.1:8b
+
+TEMPLATE """{{- if or .System .Tools }}<|start_header_id|>system<|end_header_id|>
+{{- if .System }}
+
+{{ .System }}
+{{- end }}
+{{- if .Tools }}
+
+Cutting Knowledge Date: December 2023
+
+When you receive a tool call response, use the output to format an answer to the original user question.
+
+You are a helpful assistant with tool calling capabilities.
+
+Given the following functions, respond with a JSON function call with the proper arguments when a tool is needed.
+
+Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.
+
+{{ range .Tools }}
+{{- . }}
+{{ end }}
+{{- end }}<|eot_id|>
+{{- end }}
+{{- range $i, $_ := .Messages }}
+{{- $last := eq (len (slice $.Messages $i)) 1 }}
+{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|>
+
+{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>
+
+{{ end }}
+{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|>
+{{- if .ToolCalls }}
+{{ range .ToolCalls }}
+{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }}
+{{- else }}
+
+{{ .Content }}
+{{- end }}{{ if not $last }}<|eot_id|>{{ end }}
+{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|>
+
+{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|>
+
+{{ end }}
+{{- end }}
+{{- end }}"""
+```
+
+Create the new tag:
+
+```bash
+ollama create llama3.1:8b-prefix-stable-v1 -f PrefixStable.Modelfile
+ollama list
+```
+
+Ollama reuses the existing model layers. The new tag adds a small template and
+manifest instead of copying the base weights.
+
+## Select the derived model in nanobot
+
+Merge this preset into `~/.nanobot/config.json` and select it:
+
+```json
+{
+  "providers": {
+    "ollama": {
+      "apiBase": "http://localhost:11434/v1"
+    }
+  },
+  "modelPresets": {
+    "ollamaPrefixStable": {
+      "label": "Ollama Llama 3.1 prefix-stable",
+      "provider": "ollama",
+      "model": "llama3.1:8b-prefix-stable-v1",
+      "maxTokens": 2048,
+      "contextWindowTokens": 16384,
+      "temperature": 0.1
+    }
+  },
+  "agents": {
+    "defaults": {
+      "modelPreset": "ollamaPrefixStable"
+    }
+  }
+}
+```
+
+Verify the selected model and repeat the two-turn check:
+
+```bash
+nanobot status
+nanobot agent --session cli:ollama-stable-check \
+  --message "Use the exec tool to calculate 2+2, then answer"
+nanobot agent --session cli:ollama-stable-check \
+  --message "Use the exec tool to calculate 4+7, then answer"
+```
+
+In one controlled test with Ollama 0.32.1, `llama3.1:8b`, and one slot, the second
+main request improved from `3767 / 8519` initially cached (44.22%) to
+`8505 / 8520` (99.82%). The number of re-evaluated tokens fell from 4752 to 15.
+Treat these numbers as a diagnostic example, not a performance guarantee.
+
+## Roll back
+
+Switch `agents.defaults.modelPreset` back to the original preset. When no config
+uses the derived tag, remove it with:
+
+```bash
+ollama rm llama3.1:8b-prefix-stable-v1
+```
+
+Removing the derived tag does not remove `llama3.1:8b`.
+
+## Limitations
+
+- The template above is specific to the tested `llama3.1:8b` tool-call format.
+- Ollama or the model publisher may update the stock template in a later release.
+- Validate multiple tool calls, tool errors, parallel calls, and long conversations
+  before using a custom template for unattended workloads.
+- A higher cache ratio reduces prompt evaluation, but model generation, tool
+  execution, process startup, and storage can still dominate end-to-end latency.
+- Multiple Ollama slots change cache scheduling and may produce different results.
+
+## Related nanobot docs
+
+- [Provider Cookbook: Ollama Local Model](../provider-cookbook.md#recipe-ollama-local-model)
+- [Providers and Models: Ollama](../providers.md#ollama)
+- [Troubleshooting](../troubleshooting.md)