Symptom
-------
LLM requests intermittently fail with:
'utf-8' codec can't encode characters in position N-N+1: surrogates not allowed
when messages contain emoji-heavy content (e.g. HTML with mixed emoji + JSON round-trips).
This blocks the affected session until the session file is quarantined.
Root cause
----------
Surrogate sanitization was only applied at the CLI entry point
(nanobot/cli/commands.py: _sanitize_surrogates). Requests entering
the LLM provider layer through other channels (Feishu, cron, webui,
tool results, memory injection) had no defensive cleaning, so any
message that happened to carry unpaired UTF-16 surrogates (from an
upstream JSON round-trip with ensure_ascii=True on ill-formed input,
memory rehydration, or third-party content) would blow up at
json.dumps -> HTTP encode time inside the provider client.
Fix
---
1. Extract sanitize_surrogates() and sanitize_surrogates_deep() into
nanobot/utils/helpers.py as the single source of truth. Both use
utf-16-le round-tripping with errors='surrogatepass' / 'replace',
so paired surrogates reconstruct back into their real code point
and lone surrogates collapse to U+FFFD.
2. Make nanobot/cli/commands.py:_sanitize_surrogates a thin wrapper
that re-exports the shared helper (backward compatible).
3. Add defense-in-depth at the LLM provider boundary in
nanobot/providers/base.py:_sanitize_empty_content by running
sanitize_surrogates_deep over each message and its content blocks
right before requests are serialized to JSON.
Non-goals
---------
- truncate_text() is intentionally left untouched. Python str slicing
cannot split a single code point into surrogate halves, so it is
not the source of lone surrogates.
- session/manager storage layer is untouched. Archived sessions
reproduced the failure only through the request path, not through
storage.
Verification
------------
- New regression suite tests/providers/test_sanitize_surrogates.py
covers: paired surrogate reconstruction, lone surrogate replacement,
identity return on clean input (zero allocation), deep recursion on
dict/list/tuple, provider _sanitize_empty_content integration, and
full utf-8 encodability of the sanitized request body.
- 14/14 new tests pass; full existing test module also green.
- Replayed 58 archived real session messages plus adversarial
lone-surrogate injection through the provider path with no encode
errors after the fix.
Impact
------
- No behaviour change for clean inputs (sanitize_surrogates_deep is
an identity return when no surrogate is present).
- Fails-safe: unpaired surrogates degrade to U+FFFD instead of
aborting the entire request.
jobs.json can store everyMs/atMs and next/last run timestamps as strings.
Loading left them as str, so _compute_next_run compared str to int and raised TypeError.
Coerce with an optional-int helper at from_store_dict, matching runHistory int() paths.
config.loader.load_config() intentionally returns the raw config with ${VAR}
references intact — env interpolation is a separate, explicit step
(resolve_config_env_vars) so that settings read/edit/save paths never
materialize secrets to disk or to the UI.
The transcription config path does not apply that step: both
channels/base.py (channel voice notes) and webui/transcription_ws.py (WebUI
recording) build their effective config via
resolve_transcription_config(load_config()). As a result a configured
api_key of "${GROQ_API_KEY}" (the documented way to reference secrets) is
passed to the provider verbatim, which fails with 401 Invalid API Key. No
amount of rotating the real key helps, because the literal placeholder
string is what gets sent.
Resolve the reference at the single choke point both callers share —
_resolve_transcription_api_key / _resolve_transcription_api_base — using a
new lenient loader.resolve_env_refs() helper (unset var -> empty string, so
a missing variable degrades to "not configured" rather than raising or
leaking). This fixes both entry points at once and cannot drift the way a
per-call-site fix does. Resolving inside load_config() was rejected: the
~20 settings-UI callers depend on it returning raw ${VAR} placeholders.
Literal keys are unaffected; the settings API only reads the derived
`configured` flag (never the key), which now reflects the resolved value.
Claude-Session: https://claude.ai/code/session_01Q3HuVaJAAQJA3kgVQVJ2Zt
save_config truncated config.json in place on crash mid-write.
Route through _write_text_atomic like the pairing store so a failed write leaves the prior file intact.
Maintainer edit: keep a top-level trailing '&' in the segment being matched so background execution cannot be checked as if the ampersand were absent. Redirection forms like 2>&1 and &> remain untouched.
Maintainer edit: keep the single-ampersand guard behavior, but fold the redirect exceptions into one condition instead of carrying temporary previous/next character variables.
Maintainer edit: single '&' backgrounds the preceding command and starts another top-level shell segment, so allowPatterns must split it the same way as ';', '|', '&&', and '||'. Keep fd redirections such as 2>&1 and &> intact.
Fixes chengyongru's review concern: re.search is more permissive
than the original re.fullmatch behavior for single-segment commands.
Using re.fullmatch per segment preserves backward compatibility while
still fixing the chained-command bypass.
Guard against shell-chain bypass where an attacker appends '&& malicious'
after an allowlisted prefix. The allowlist check now splits the command
on top-level chaining operators (&&, ||, ;, |) and requires every segment
to match at least one allowPattern independently.
Fixes#4521
Bind SessionManager saves to the existing raw archive path so SDK imports and other bypass saves cannot persist more than the file cap without archiving unconsolidated overflow.
Add an SDK regression test that exercises the real ingest path.
Refs #4787
The tool execution path caught BaseException, which includes
KeyboardInterrupt, SystemExit, MemoryError, and GeneratorExit.
These should never be caught and converted into conversational
error messages. CancelledError is already handled separately.
Change except BaseException to except Exception so fatal signals
propagate instead of being swallowed.
Adds parametrized regression test for KeyboardInterrupt and
SystemExit propagation.
Fixes#4788
The QQ channel's _run_bot() used a fixed 5-second reconnect interval with
no backoff. When the network is unavailable (e.g., DNS failure), this
produces excessive botpy SDK error tracebacks every 5 seconds, flooding
logs.
botpy's Client.bot_connect() catches ws_connect() exceptions internally
and calls BotWebSocket.on_error(), which logs a full traceback and
immediately re-queues the session. The outer _run_bot() except never
fires for the reported DNS failure path.
Override bot_connect() on the _Bot subclass to:
- Apply exponential backoff (5s -> 300s cap) before re-queuing the session
- Log network errors (ClientConnectorDNSError, ClientConnectorError,
OSError) compactly without traceback
- Reset backoff on successful connection
- Still call traceback.print_exc() for non-network errors
The outer _run_bot() loop retains exponential backoff as a fallback for
exceptions that escape start() entirely. The botpy library logging
redirect is elevated to ERROR to suppress redundant connection tracebacks.
Consistent with patterns already used in matrix.py and napcat.py.
Add 7 regression tests covering:
- DNS error applies backoff and re-queues session
- No traceback printed for network errors
- ClientConnectorError also triggers backoff
- Backoff doubles and caps at 300s
- Successful connection resets backoff
- Non-network errors still re-queue without backoff
- _is_network_error() classification
Fixes#4767
Use the existing in-flight context governor to replace tool output that cannot fit the next model request with a bounded, actionable instruction. The model can retry with narrower arguments, use another tool, or explain the context limit without a second recovery state machine.
Map NANOBOT_WEB_TOKEN to channels.websocket.tokenIssueSecret instead of
the static token, and remove the static token. The gateway now issues
short-lived WebSocket/API tokens rather than accepting a long-lived
credential directly at the handshake, matching the public-WebUI login
flow and documentation. Users still enter the same NANOBOT_WEB_TOKEN,
and websocketRequiresToken remains true.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The WebSocket channel also serves the WebUI over plain HTTP, so on a
public endpoint (e.g. a Render *.onrender.com service) the underlying
websockets library logs a full-traceback ERROR for every request that
isn't a valid GET handshake: HEAD probes ("unsupported HTTP method;
expected GET"), port scanners, uptime monitors, and TLS-to-plain-port
attempts. These are internet background noise, not server faults.
WebSocketHandshakeNoiseFilter already suppressed "opening handshake
failed" records caused by mid-handshake disconnects; widen it to also
suppress records whose exception chain contains websockets'
InvalidMessage, which covers both non-GET methods and malformed/empty
requests. Genuine server-side handshake errors still log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Updated entrypoint.sh to initialize the on-disk config only if it does not already exist, preserving user edits across restarts.
- Enhanced privilege dropping logic to ensure the container does not run as root if the privilege drop fails.
- Clarified comments in Dockerfile and entrypoint.sh for better understanding of the privilege management process.
- Updated README.md to include a note about persistent disks requiring a paid service on Render.
- Adjusted render.yaml to clarify the Docker command behavior and added a note regarding auto-deploy settings.
Adds a Render Blueprint (render.yaml) and supporting pieces so nanobot can
be deployed to Render in one click, with persistent memory across deploys.
- render.yaml: web service + 1GB persistent disk mounted at
/home/nanobot/.nanobot. Prompts for ANTHROPIC_API_KEY and
NANOBOT_WEB_TOKEN at deploy time (sync: false).
- render-config.json: committed gateway config that wires secrets via
${VAR} placeholders (resolved at runtime). Nothing secret is committed.
- entrypoint.sh: adds a branch gated on RENDER=true that copies the config
onto the mounted disk, chowns the root-owned mount, and drops to the
non-root nanobot user via setpriv. Local (non-Render) path is unchanged.
- Dockerfile: COPY render-config.json; USER nanobot -> USER root so the
entrypoint can chown the freshly-mounted disk before dropping privileges;
add PYTHONUNBUFFERED/PYTHONFAULTHANDLER for diagnosable crash output.
- README.md: Deploy to Render button + section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jobs.json hand-edits and asdict-style snake_case for schedule intervals and
runHistory crashed or silently disabled cron. Deserialize via Cron* from_store_dict
and shared get_camel_snake (also used by local triggers).
Kris LuandXubin Ren
santhrealandXubin Ren
chengyongruandchengyongru
Ben LenartsandXubin Ren
amplifierplusandchengyongru
michaelxerandXubin Ren
KDBandXubin Ren
axelray-devandXubin Ren
chengyongruandchengyongru
sunpengcheng05andchengyongru
golaandchengyongru
Pei Futongandchengyongru
