The recent-history section injected into the system prompt was capped by character count (_MAX_HISTORY_CHARS = 32_000). Characters are a poor proxy for tokens: ~32k chars of English is ~8k tokens, but the same char count of CJK text or code can be far more, so the cap could let the section blow well past its intended size on non English/code-heavy histories.
Add a reusable truncate_text_to_tokens() helper (reusing the tiktoken
cl100k_base encoder already used elsewhere, with a char-based fallback) and
switch the digest cap to a token budget (_MAX_HISTORY_TOKENS = 8_000),
matching the previous English-text size while holding regardless of
content.
w.antarandXubin Ren
973a5ee507