Maintainer edit: make token truncation include the suffix within the budget and route the consolidator through the shared helper so recent-history and archive truncation keep the same semantics.
The recent-history section injected into the system prompt was capped by character count (_MAX_HISTORY_CHARS = 32_000). Characters are a poor proxy for tokens: ~32k chars of English is ~8k tokens, but the same char count of CJK text or code can be far more, so the cap could let the section blow well past its intended size on non English/code-heavy histories.
Add a reusable truncate_text_to_tokens() helper (reusing the tiktoken
cl100k_base encoder already used elsewhere, with a char-based fallback) and
switch the digest cap to a token budget (_MAX_HISTORY_TOKENS = 8_000),
matching the previous English-text size while holding regardless of
content.
Move the fenced-code-block-aware splitting logic out of the shared
split_message helper (used by Signal, Slack, Discord, Weixin, etc.)
and into a Telegram-specific _split_telegram_markdown function.
The shared split_message remains a plain-text chunker. The Telegram
channel now uses _split_telegram_markdown for its raw Markdown paths
that feed _markdown_to_telegram_html, preventing broken HTML rendering
when splits fall inside fenced code blocks.
Also fixes a regression where content beginning with whitespace before
a fence could emit a whitespace-only chunk.
Addresses review feedback on #4257.
When split_message splits a long message, it now checks whether the
split point falls inside a fenced code block. If so, it either moves
the split to before the opening fence or closes/reopens the fence
across chunks, preventing broken HTML rendering.
Addresses #4250
yorkhellenandXubin Ren
chengyongruandXubin Ren
w.antarandXubin Ren
axelray-devandXubin Ren