diff --git a/docs/guides/README.md b/docs/guides/README.md index 81595618..a21c3951 100644 --- a/docs/guides/README.md +++ b/docs/guides/README.md @@ -44,6 +44,7 @@ Use **Settings → Channels** in the WebUI for guided setup. These guides explai | Enable web search | [Configure web search](./configure-web-search.md) | | Add model fallback | [Configure model fallback](./configure-model-fallback.md) | | Add an OpenAI-compatible provider | [Configure an OpenAI-compatible provider](./configure-openai-compatible-provider.md) | +| Improve Ollama tool prompt-cache reuse | [Configure Ollama prompt caching](./configure-ollama-prompt-cache.md) | | Add Langfuse tracing | [Configure Langfuse observability](./configure-langfuse-observability.md) | | Secure local tools | [Secure a local AI agent](./secure-local-ai-agent.md) | | Deploy the gateway | [Deploy nanobot gateway](./deploy-nanobot-gateway.md) | diff --git a/docs/guides/configure-ollama-prompt-cache.md b/docs/guides/configure-ollama-prompt-cache.md new file mode 100644 index 00000000..6845cdbd --- /dev/null +++ b/docs/guides/configure-ollama-prompt-cache.md @@ -0,0 +1,239 @@ +# How to Improve Ollama Tool-Calling Prompt Cache Reuse in nanobot + +Some Ollama model templates move or remove tool definitions as a conversation +switches between user, assistant, and tool messages. nanobot can send a correct +append-only chat request while the model template still renders a different token +prefix. On slower local hardware, re-evaluating that prefix can add tens of seconds +to an otherwise simple tool-using turn. + +This guide shows how to diagnose that specific pattern and create a derived +`llama3.1:8b` tag with a prefix-stable tool template. It does not modify nanobot or +overwrite the original Ollama model. + +## What you will build + +- a repeatable two-turn cache check +- an optional derived `llama3.1:8b-prefix-stable-v1` Ollama tag +- a nanobot model preset that uses the derived tag + +## When to use this + +Use this guide when all of the following are true: + +- direct Ollama responses are reasonably fast; +- nanobot becomes slow after the model calls a tool; +- Ollama logs show a long main prompt, a much shorter tool follow-up, and low + initial cache reuse on the next main prompt; +- the model is `llama3.1:8b` with a template that renders concrete tools only for + the final user message. + +Do not apply this template to another model family without checking that model's +tool-call format first. + +## Diagnose the rendered prompt + +Stop any existing Ollama process, then start a single-slot debug server. A single +slot makes the cache sequence easier to read. + +**macOS or Linux** + +```bash +OLLAMA_CONTEXT_LENGTH=16384 \ +OLLAMA_NUM_PARALLEL=1 \ +OLLAMA_DEBUG=1 \ +ollama serve +``` + +**Windows PowerShell** + +```powershell +$env:OLLAMA_CONTEXT_LENGTH = "16384" +$env:OLLAMA_NUM_PARALLEL = "1" +$env:OLLAMA_DEBUG = "1" +ollama serve +``` + +In another terminal, use a fresh session and explicitly request a tool so both +turns exercise the agent loop: + +```bash +nanobot agent --session cli:ollama-cache-check \ + --message "Use the exec tool to calculate 2+2, then answer" +nanobot agent --session cli:ollama-cache-check \ + --message "Use the exec tool to calculate 4+7, then answer" +``` + +In the Ollama output, find each `new prompt` line and the first +`cached n_tokens` line that follows it. Later increasing `cached n_tokens` lines +are prompt-evaluation progress, not additional initial cache hits. + +A cache-unfriendly tool template may produce a pattern like this: + +```text +turn 1 main: 2 / 8460 initially cached +turn 1 tool follow-up: 3713 / 3758 initially cached +turn 2 main: 3767 / 8519 initially cached +``` + +The cache is working, but the next main request can reuse only the shorter prompt. +Hardware throughput determines how expensive the remaining evaluation is. + +To inspect the API request bodies as well, add +`OLLAMA_DEBUG_LOG_REQUESTS=1` before starting Ollama. These logs can contain system +prompts, workspace context, and user messages. Keep them local and disable request +logging after diagnosis. + +## Why this happens with the stock template + +The tested `llama3.1:8b` template conditionally expands the tool definitions inside +a user message: + +```gotemplate +{{- if and $.Tools $last }} + ... render tool definitions ... +{{- end }} +``` + +The first request ends with a user message, so the tools are rendered there. After +nanobot appends an assistant tool call and its result, that user message is no +longer last, so the same API request history renders without the concrete tool +block. On the next user turn, the tools reappear at a new position. + +This is a model-template behavior. At the API boundary, nanobot continues to append +the assistant tool call and tool result and sends the same tool definitions. + +## Create a prefix-stable derived model + +Create `PrefixStable.Modelfile` with the content below. The template keeps concrete +tool definitions in the system block, where they remain in the same position across +user and tool messages. + +```dockerfile +FROM llama3.1:8b + +TEMPLATE """{{- if or .System .Tools }}<|start_header_id|>system<|end_header_id|> +{{- if .System }} + +{{ .System }} +{{- end }} +{{- if .Tools }} + +Cutting Knowledge Date: December 2023 + +When you receive a tool call response, use the output to format an answer to the original user question. + +You are a helpful assistant with tool calling capabilities. + +Given the following functions, respond with a JSON function call with the proper arguments when a tool is needed. + +Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables. + +{{ range .Tools }} +{{- . }} +{{ end }} +{{- end }}<|eot_id|> +{{- end }} +{{- range $i, $_ := .Messages }} +{{- $last := eq (len (slice $.Messages $i)) 1 }} +{{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|> + +{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|> + +{{ end }} +{{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|> +{{- if .ToolCalls }} +{{ range .ToolCalls }} +{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }} +{{- else }} + +{{ .Content }} +{{- end }}{{ if not $last }}<|eot_id|>{{ end }} +{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|> + +{{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|> + +{{ end }} +{{- end }} +{{- end }}""" +``` + +Create the new tag: + +```bash +ollama create llama3.1:8b-prefix-stable-v1 -f PrefixStable.Modelfile +ollama list +``` + +Ollama reuses the existing model layers. The new tag adds a small template and +manifest instead of copying the base weights. + +## Select the derived model in nanobot + +Merge this preset into `~/.nanobot/config.json` and select it: + +```json +{ + "providers": { + "ollama": { + "apiBase": "http://localhost:11434/v1" + } + }, + "modelPresets": { + "ollamaPrefixStable": { + "label": "Ollama Llama 3.1 prefix-stable", + "provider": "ollama", + "model": "llama3.1:8b-prefix-stable-v1", + "maxTokens": 2048, + "contextWindowTokens": 16384, + "temperature": 0.1 + } + }, + "agents": { + "defaults": { + "modelPreset": "ollamaPrefixStable" + } + } +} +``` + +Verify the selected model and repeat the two-turn check: + +```bash +nanobot status +nanobot agent --session cli:ollama-stable-check \ + --message "Use the exec tool to calculate 2+2, then answer" +nanobot agent --session cli:ollama-stable-check \ + --message "Use the exec tool to calculate 4+7, then answer" +``` + +In one controlled test with Ollama 0.32.1, `llama3.1:8b`, and one slot, the second +main request improved from `3767 / 8519` initially cached (44.22%) to +`8505 / 8520` (99.82%). The number of re-evaluated tokens fell from 4752 to 15. +Treat these numbers as a diagnostic example, not a performance guarantee. + +## Roll back + +Switch `agents.defaults.modelPreset` back to the original preset. When no config +uses the derived tag, remove it with: + +```bash +ollama rm llama3.1:8b-prefix-stable-v1 +``` + +Removing the derived tag does not remove `llama3.1:8b`. + +## Limitations + +- The template above is specific to the tested `llama3.1:8b` tool-call format. +- Ollama or the model publisher may update the stock template in a later release. +- Validate multiple tool calls, tool errors, parallel calls, and long conversations + before using a custom template for unattended workloads. +- A higher cache ratio reduces prompt evaluation, but model generation, tool + execution, process startup, and storage can still dominate end-to-end latency. +- Multiple Ollama slots change cache scheduling and may produce different results. + +## Related nanobot docs + +- [Provider Cookbook: Ollama Local Model](../provider-cookbook.md#recipe-ollama-local-model) +- [Providers and Models: Ollama](../providers.md#ollama) +- [Troubleshooting](../troubleshooting.md) diff --git a/docs/provider-cookbook.md b/docs/provider-cookbook.md index afa3e178..742a2b33 100644 --- a/docs/provider-cookbook.md +++ b/docs/provider-cookbook.md @@ -431,7 +431,13 @@ curl -sS http://localhost:11434/v1/models nanobot agent -m "Hello!" ``` -If you see `connection refused`, Ollama is not running or `apiBase` points to the wrong port. If the response is very slow, try a smaller local model or lower `contextWindowTokens`. +If you see `connection refused`, Ollama is not running or `apiBase` points to the wrong port. If every response is slow, try a smaller local model or lower `contextWindowTokens`. + +If direct Ollama responses are fast but tool-using nanobot turns repeatedly evaluate +thousands of prompt tokens, the model's chat template may be moving its tool +definitions between requests. See +[Improve Ollama Tool-Calling Prompt Cache Reuse](./guides/configure-ollama-prompt-cache.md) +for a diagnostic procedure and an optional model-specific workaround. ## Recipe: vLLM or LM Studio diff --git a/docs/providers.md b/docs/providers.md index e287fcfb..9ce3135e 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -331,6 +331,13 @@ Start Ollama separately, then point nanobot at the OpenAI-compatible endpoint. Most Ollama setups do not require an API key. +Ollama renders the OpenAI-compatible messages and tools through each model's chat +template. If ordinary model responses are fast but tool-using turns show low prompt +cache reuse, diagnose the rendered template before changing nanobot's context or +memory settings. The +[Ollama prompt-cache guide](./guides/configure-ollama-prompt-cache.md) explains the +log pattern and a tested `llama3.1:8b` workaround. + ### vLLM or Other Local OpenAI-Compatible Server ```json