Add an `extra_body` field to `ProviderConfig` that merges arbitrary
key-value pairs into every OpenAI-compatible request body. This is the
escape hatch for provider-specific features that nanobot does not have
first-class fields for.
Real-world use cases this unblocks via config alone (no code changes):
- vLLM/TGI `chat_template_kwargs` (e.g. `enable_thinking: false`)
- vLLM guided decoding (`guided_json`, `guided_regex`)
- Local model sampling params (`repetition_penalty`, `top_k`, `min_p`)
- Any future provider-specific param without a new PR each time
The config extra_body is applied last via recursive deep-merge, so it
can extend or override provider-specific defaults (e.g. thinking
params) without clobbering sibling keys set by internal logic.
Changes:
- Add `extra_body: dict[str, Any] | None` to `ProviderConfig`
- Pass it through `factory.py` to `OpenAICompatProvider.__init__`
- Deep-merge into `_build_kwargs` after all internal extra_body entries
- Add `_deep_merge` helper (recursive dict merge, does not mutate inputs)
- 21 tests: deep-merge semantics, provider init, _build_kwargs
integration, thinking coexistence, real-world patterns (guided_json,
repetition_penalty), and schema validation
The non-streaming parse path unconditionally promoted the `reasoning`
response field to `content` when content was empty. This was intended
for StepFun (whose API returns the actual answer in `reasoning`), but
it applied to every OpenAI-compatible provider — causing internal
thinking chains from models like Xiaomi MIMO to be leaked as formal
replies.
Add `reasoning_as_content: bool` to ProviderSpec (default False) and
set it only for StepFun. The fallback now requires this flag rather
than running globally.
Fixes#3443
Parse the endpoint host before disabling keepalive so public hostnames that merely contain private-network substrings keep the default connection pool behavior.
Made-with: Cursor
Local model servers (Ollama, llama.cpp, vLLM) often close idle HTTP
connections before the client-side keepalive timer expires. When two
LLM calls happen seconds apart — for example the heartbeat _decide()
phase followed immediately by process_direct() — the second call grabs
a now-dead pooled connection, causing a transient APIConnectionError
on every first attempt.
The fix detects local endpoints via:
- ProviderSpec.is_local (Ollama, LM Studio, vLLM, OVMS)
- Private-network URL patterns (localhost, 127.x, 192.168.x, 10.x,
172.16-31.x, host.docker.internal, [::1])
For these endpoints, the AsyncOpenAI client is created with a custom
httpx.AsyncClient that sets keepalive_expiry=0, forcing a fresh TCP
connection for each request. This is cheap on LAN (sub-5ms connect)
and eliminates the stale-connection retry tax entirely.
Cloud providers (OpenAI, Anthropic, OpenRouter, etc.) keep the default
5-second keepalive, which is fine for high-frequency API usage.
The private-network heuristic also covers the common case where users
configure provider='openai' but point apiBase at a LAN IP running
llama.cpp — the spec says is_local=False, but the URL clearly is.
Two issues with DeepSeek V4 thinking mode support:
1. Missing thinking parameter injection.
DeepSeek V4 requires `extra_body: {"thinking": {"type": "enabled/disabled"}}`
— identical to VolcEngine/BytePlus. The code had this for volcengine,
byteplus, dashscope, minimax, and kimi but not DeepSeek. This means
`reasoning_effort=minimal` (thinking off) silently has no effect.
Root cause: the thinking-style→wire-format mapping was an if/elif chain
on provider *names*. DeepSeek was forgotten.
Fix: make the mapping declarative via `ProviderSpec.thinking_style`:
- "thinking_type" → {"thinking": {"type": "..."}} (DeepSeek, Volc, BytePlus)
- "enable_thinking" → {"enable_thinking": bool} (DashScope)
- "reasoning_split" → {"reasoning_split": bool} (MiniMax)
`_build_kwargs` now does a single dict lookup. Adding a new provider
with an existing wire format requires zero changes to the function.
2. Legacy session messages crash thinking-mode requests.
When a session was started without thinking mode (or with a different
model), assistant messages lack reasoning_content. DeepSeek V4 in
thinking mode rejects these with 400:
"The reasoning_content in the thinking mode must be passed back to the API."
This affects ALL assistant messages, not just those with tool_calls
(despite the docs only mentioning the tool_calls case).
Fix: `_build_kwargs` backfills `reasoning_content: ""` on every
assistant message missing it, but only when thinking mode is active.
This is semantically neutral — the model treats empty reasoning_content
as "no thinking happened on that turn". The backfill only touches the
in-memory request copy; session files on disk are untouched.
Tests: +5 (3 thinking toggle, 2 backfill). Full suite: 2377 passed.
Made-with: Cursor
Calling GitHub Copilot with `gpt-5.*` / `o*` models (e.g.
`github_copilot/gpt-5.4`, `github_copilot/gpt-5.4-mini`) failed with a
chain of misleading errors:
1. `Unsupported parameter: 'max_tokens' is not supported with this
model. Use 'max_completion_tokens' instead.`
2. `model "gpt-5.4-mini" is not accessible via the /chat/completions
endpoint` (`unsupported_api_for_model`).
3. `The requested model is not supported.` (`model_not_supported`)
even after routing to /responses.
Root causes (each one masked the next):
* The `github_copilot` ProviderSpec did not opt into
`supports_max_completion_tokens`, so `_build_kwargs` always sent the
legacy `max_tokens` parameter that GPT-5/o-series reject.
* `_should_use_responses_api` was hard-gated to
`spec.name == "openai"` plus a direct-OpenAI base URL, so the
GitHub Copilot backend always went through /chat/completions even
for models the Copilot gateway exposes only via /responses
(e.g. `gpt-5.4-mini`).
* When /responses did fail on github_copilot, the existing
"compatibility marker" heuristic silently fell back to
/chat/completions — which can never succeed for these models — so
the real upstream error was hidden.
* `_build_responses_body` did not honour `spec.strip_model_prefix`,
so the request body sent `model="github_copilot/gpt-5.4-mini"`
(with the routing prefix), which the Copilot gateway rejects with
`model_not_supported`. (`_build_kwargs` already stripped it; this
branch was missed.)
Fix:
* registry.py: set `supports_max_completion_tokens=True` on the
`github_copilot` spec so requests use `max_completion_tokens`.
* openai_compat_provider.py:
- `_should_use_responses_api` now also allows the
`github_copilot` spec, and skips the direct-OpenAI base check
for it (the Copilot gateway is its own base URL).
- `_build_responses_body` now strips the model routing prefix
when `spec.strip_model_prefix` is set, matching `_build_kwargs`.
- `chat` / `chat_stream` no longer fall back from /responses to
/chat/completions on the `github_copilot` spec: the fallback
cannot succeed for GPT-5/o-series and would mask the real
gateway error.
Tests:
* tests/cli/test_commands.py: switched the
`test_github_copilot_provider_refreshes_client_api_key_before_chat`
fixture model from `gpt-5.1` to `gpt-4` so it continues to exercise
the /chat/completions code path it was designed for (gpt-5.1 now
correctly routes to /responses on github_copilot).
* `pytest tests/providers/ tests/cli/test_commands.py` — 314 passed.
* Verified end-to-end against the live Copilot gateway with both
`github_copilot/gpt-5.4` and `github_copilot/gpt-5.4-mini`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DashScope rejects the OpenAI-style value "minimal" with
`'reasoning_effort.effort' must be one of: 'none', 'minimum', 'low',
'medium', 'high', 'xhigh'`, but nanobot was passing the string through
verbatim. Users who tried the documented "minimal" to disable thinking
got a 400; users who tried the DashScope-native "minimum" to work
around it got `enable_thinking=True` because the internal comparison
was a hard string match on "minimal".
Introduce a semantic/wire split in `_build_kwargs`:
- `semantic_effort` is the internal canonical form (OpenAI vocabulary).
"minimum" on the way in is normalized to "minimal" here so both
spellings share one meaning.
- `wire_effort` is what we actually serialize. For DashScope with
semantic_effort == "minimal" we translate to "minimum" on the way
out; other providers are unchanged.
- `thinking_enabled` and the Kimi thinking branch now compare on
`semantic_effort`, so either user spelling correctly disables
provider-side thinking.
Tests:
- Strengthen `test_dashscope_thinking_disabled_for_minimal` to assert
the wire value is "minimum" in addition to the extra_body signal;
the original version only checked extra_body and let the
invalid-value bug slip through.
- Add `test_dashscope_thinking_disabled_for_minimum_alias` so a user
who read the DashScope docs and configured "minimum" still gets
thinking off.
- Add `test_non_dashscope_minimal_not_retranslated` to pin down that
the DashScope-specific translation does not leak to OpenAI et al.
When the Responses API fails repeatedly (3 consecutive compatibility
errors), skip it and fall back directly to Chat Completions. Unlike a
permanent disable, the circuit re-probes after 5 minutes so recovery
is automatic when the API comes back. Success resets the counter.
Keyed per (model, reasoning_effort) so a failure with one model does
not affect others.
- Inject `thinking={"type": "enabled|disabled"}` via extra_body for
Kimi thinking-capable models (kimi-k2.5, k2.6-code-preview).
- Add _is_kimi_thinking_model helper to handle both bare slugs and
OpenRouter-style prefixed names (e.g. moonshotai/kimi-k2.5).
- reasoning_effort="minimal" maps to disabled; any other value enables it.
- Add tests for enabled/disabled states and OpenRouter prefix handling.
Ensure assistant tool-call function.arguments is always emitted as valid JSON text so strict OpenAI-compatible backends (including Alibaba code models) do not reject requests. Add regressions for dict and malformed-string argument payloads in message sanitization.
Made-with: Cursor
Keep tool-call assistant messages valid across provider sanitization and avoid trailing user-only history after model errors. This prevents follow-up requests from sending broken tool chains back to the gateway.
Resolved conflict in azure_openai_provider.py by keeping main's
Responses API implementation (role alternation not needed for the
Responses API input format).
Made-with: Cursor
- Add reasoning_content fallback from reasoning in _parse dict branch
- Add content fallback from msg.reasoning in _parse SDK object branch
- Add reasoning_content fallback in _parse SDK object branch
- Add reasoning fallback in _parse_chunks dict branch
- Add reasoning fallback in _parse_chunks SDK object branch
This ensures StepFun Plan API works correctly in both streaming and
non-streaming modes, for both dict and SDK object response formats.
StepFun Plan API returns response content in the 'reasoning' field when
the model is in thinking mode and 'content' is empty. OpenAICompatProvider
previously only checked 'content' and 'reasoning_content', missing this field.
This patch adds a fallback: if content is empty and 'reasoning' is present,
extract text from reasoning to populate content, ensuring StepFun models
(step-3.5-flash, step-3.5-flash-2603) work correctly with tool calls.
Co-authored-by: moranfong <moranfong@gmail.com>
Merge the three retry-after header parsers (base, OpenAI, Anthropic)
into a single _extract_retry_after_from_headers on LLMProvider that
handles retry-after-ms, case-insensitive lookup, and HTTP date.
Remove the per-provider _parse_retry_after_headers duplicates and
their now-unused email.utils / time imports. Add test for retry-after-ms.
Made-with: Cursor
- Add byteplus and byteplus_coding_plan to thinking param providers
- Only send extra_body when reasoning_effort is explicitly set
- Use setdefault().update() to avoid clobbering existing extra_body
- Add 7 regression tests for thinking params
Made-with: Cursor
- Use import importlib.util (not bare importlib) for find_spec
- Warn and fall back to standard openai instead of crashing with
ImportError when LANGFUSE_SECRET_KEY is set but langfuse is missing
Made-with: Cursor
The streaming path in OpenAICompatProvider.chat_stream() was passing
reasoning_content deltas through on_content_delta(), causing model
internal reasoning to be displayed to the user alongside the actual
response content.
reasoning_content is already collected separately in _parse_chunks()
and stored in LLMResponse.reasoning_content for session history.
It should never be forwarded to the user-facing stream.
Enable GPT-5 models (gpt-5, gpt-5.4, gpt-5.4-mini, etc.) to work
correctly with the OpenAI-compatible provider by:
- Setting `supports_max_completion_tokens=True` on the OpenAI provider
spec so `max_completion_tokens` is sent instead of the deprecated
`max_tokens` parameter that GPT-5 rejects.
- Adding `_supports_temperature()` to conditionally omit the
`temperature` parameter for reasoning models (o1/o3/o4) and when
`reasoning_effort` is active, matching the existing Azure provider
behaviour.
Both changes are backward-compatible: older GPT-4 models continue to
work as before since `max_completion_tokens` is accepted by all recent
OpenAI models and temperature is only omitted when reasoning is active.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reasoning_content and extra_content were accidentally dropped from
_ALLOWED_MSG_KEYS.
Also fix session/manager.py to include reasoning_content when building
LLM messages from session history, so the field is not lost across
turns.
Without this fix, providers such as Kimi, emit reasoning_content in
assistant messages will have it stripped on the next request, breaking
multi-turn thinking mode.
Fixes: https://github.com/HKUDS/nanobot/issues/2777
Signed-off-by: Lingao Meng <menglingao@xiaomi.com>
Extract reasoning_content from both non-streaming and streaming responses
in OpenAICompatProvider. Accumulate chunks during streaming and merge into
LLMResponse, enabling reasoning chain display for models like MiMo and DeepSeek-R1.
Signed-off-by: Lingao Meng <menglingao@xiaomi.com>
hussein1362andXubin Ren
hanyuanlingandXubin Ren
Xubin RenandXubin Ren
chengyongruandXubin Ren
Peixian GongandCopilot
hlgandXubin Ren
kandXubin Ren
Mohamed ElkholyandXubin Ren
razzhandXubin Ren
Michael-lhhandXubin Ren
haosenwang1018andXubin Ren
moranfongandXubin Ren
PlayDustinDBandXubin Ren
lang07123andXubin Ren
chengyongruandXubin Ren
Lingao MengandXubin Ren
pikaxinge
chengyongruandXubin Ren
Tejas1KoliandXubin Ren