feat(transcription): add AssemblyAI as transcription provider
Add AssemblyAI as a third transcription provider option alongside OpenAI and Groq. AssemblyAI offers better accuracy for certain audio types (distant voices, noisy environments) and serves as a reliable fallback when other providers struggle. Changes: - Add AssemblyAITranscriptionProvider class in providers/transcription.py - Add 'assemblyai' option in base channel's transcribe_audio() - Per-channel configuration via transcriptionProvider in config Usage: Set transcriptionProvider: 'assemblyai' and provide an AssemblyAI API key via transcriptionApiKey in the channel config.
This commit is contained in:
comadreja
Xubin Ren
+1
-1
@@ -28,9 +28,9 @@ Use these when you want deeper customization, integration, or extension details.
|
||||
|
||||
| Topic | Repo docs | What it covers |
|
||||
|---|---|---|
|
||||
| Development | [`development.md`](./development.md) | Contributor notes for adding providers and transcription adapters |
|
||||
| Memory | [`memory.md`](./memory.md) | How nanobot stores, consolidates, and restores memory |
|
||||
| Python SDK | [`python-sdk.md`](./python-sdk.md) | Use nanobot programmatically from Python |
|
||||
| Channel plugin guide | [`channel-plugin-guide.md`](./channel-plugin-guide.md) | Build and test custom chat channel plugins |
|
||||
| WebSocket channel | [`websocket.md`](./websocket.md) | Real-time WebSocket access and protocol details |
|
||||
| Custom tools | [`my-tool.md`](./my-tool.md) | Inspect and tune runtime state with the `my` tool |
|
||||
|
||||
|
||||
+9
-45
@@ -119,7 +119,7 @@ ANTHROPIC_API_KEY="$(bw get password api/anthropic)" nanobot agent
|
||||
## Providers
|
||||
|
||||
> [!TIP]
|
||||
> - **Voice transcription**: Voice messages and WebUI/desktop microphone input use the shared top-level `transcription` settings. By default Groq Whisper is used; set `transcription.provider` to `"openai"` for OpenAI Whisper, `"openrouter"` for OpenRouter speech-to-text models, or `"xiaomi_mimo"` for Xiaomi MiMo ASR. API keys still live in the matching `providers.<provider>` config.
|
||||
> - **Voice transcription**: Voice messages and WebUI/desktop microphone input use the shared top-level `transcription` settings. By default Groq Whisper is used; set `transcription.provider` to `"openai"` for OpenAI Whisper, `"openrouter"` for OpenRouter speech-to-text models, `"xiaomi_mimo"` for Xiaomi MiMo ASR, or `"assemblyai"` for AssemblyAI. API keys still live in the matching `providers.<provider>` config.
|
||||
> - **MiniMax Coding Plan**: Exclusive discount links for the nanobot community: [Overseas](https://platform.minimax.io/subscribe/coding-plan?code=9txpdXw04g&source=link) · [Mainland China](https://platform.minimaxi.com/subscribe/token-plan?code=GILTJpMTqZ&source=link)
|
||||
> - **MiniMax (Mainland China)**: If your API key is from MiniMax's mainland China platform (minimaxi.com), set `"apiBase": "https://api.minimaxi.com/v1"` in your minimax provider config.
|
||||
> - **MiniMax thinking mode**: Use `providers.minimaxAnthropic` when you want `reasoningEffort` / thinking mode. MiniMax exposes that capability through its Anthropic-compatible endpoint, so nanobot keeps it as a separate provider instead of guessing MiniMax-specific thinking parameters on the generic OpenAI-compatible `minimax` endpoint. It uses the same `MINIMAX_API_KEY`. Default Anthropic-compatible base URL: `https://api.minimax.io/anthropic`; for mainland China use `https://api.minimaxi.com/anthropic`.
|
||||
@@ -143,6 +143,7 @@ ANTHROPIC_API_KEY="$(bw get password api/anthropic)" nanobot agent
|
||||
| `azure_openai` | LLM (Azure OpenAI) | [portal.azure.com](https://portal.azure.com) |
|
||||
| `bedrock` | LLM (AWS Bedrock Converse, Claude/Nova/Llama/etc.) | [aws.amazon.com/bedrock](https://aws.amazon.com/bedrock/) |
|
||||
| `openai` | LLM + Voice transcription (Whisper) | [platform.openai.com](https://platform.openai.com) |
|
||||
| `assemblyai` | Voice transcription only | [assemblyai.com](https://www.assemblyai.com/) |
|
||||
| `deepseek` | LLM (DeepSeek direct) | [platform.deepseek.com](https://platform.deepseek.com) |
|
||||
| `groq` | LLM + Voice transcription (Whisper, default) | [console.groq.com](https://console.groq.com) |
|
||||
| `minimax` | LLM (MiniMax direct) | [platform.minimaxi.com](https://platform.minimaxi.com) |
|
||||
@@ -957,48 +958,8 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Adding a New Provider (Developer Guide)</b></summary>
|
||||
|
||||
nanobot uses a **Provider Registry** (`nanobot/providers/registry.py`) as the single source of truth.
|
||||
Adding a new provider only takes **2 steps** — no if-elif chains to touch.
|
||||
|
||||
**Step 1.** Add a `ProviderSpec` entry to `PROVIDERS` in `nanobot/providers/registry.py`:
|
||||
|
||||
```python
|
||||
ProviderSpec(
|
||||
name="myprovider", # config field name
|
||||
keywords=("myprovider", "mymodel"), # model-name keywords for auto-matching
|
||||
env_key="MYPROVIDER_API_KEY", # env var name
|
||||
display_name="My Provider", # shown in `nanobot status`
|
||||
default_api_base="https://api.myprovider.com/v1", # OpenAI-compatible endpoint
|
||||
)
|
||||
```
|
||||
|
||||
**Step 2.** Add a field to `ProvidersConfig` in `nanobot/config/schema.py`:
|
||||
|
||||
```python
|
||||
class ProvidersConfig(BaseModel):
|
||||
...
|
||||
myprovider: ProviderConfig = ProviderConfig()
|
||||
```
|
||||
|
||||
That's it! Environment variables, model routing, config matching, and `nanobot status` display will all work automatically.
|
||||
|
||||
**Common `ProviderSpec` options:**
|
||||
|
||||
| Field | Description | Example |
|
||||
|-------|-------------|---------|
|
||||
| `default_api_base` | OpenAI-compatible base URL | `"https://api.deepseek.com"` |
|
||||
| `env_extras` | Additional env vars to set | `(("ZHIPUAI_API_KEY", "{api_key}"),)` |
|
||||
| `model_overrides` | Per-model parameter overrides | `(("kimi-k2.5", {"temperature": 1.0}), ("kimi-k2.6", {"temperature": 1.0}),)` |
|
||||
| `is_gateway` | Can route any model (like OpenRouter) | `True` |
|
||||
| `detect_by_key_prefix` | Detect gateway by API key prefix | `"sk-or-"` |
|
||||
| `detect_by_base_keyword` | Detect gateway by API base URL | `"openrouter"` |
|
||||
| `strip_model_prefix` | Strip provider prefix before sending to gateway | `True` (for AiHubMix) |
|
||||
| `supports_max_completion_tokens` | Use `max_completion_tokens` instead of `max_tokens`; required for providers that reject both being set simultaneously (e.g. VolcEngine) | `True` |
|
||||
|
||||
</details>
|
||||
Contributor notes for adding new providers live in
|
||||
[`development.md`](./development.md#adding-an-llm-provider).
|
||||
|
||||
## Model Presets
|
||||
|
||||
@@ -1122,8 +1083,8 @@ Configure transcription under the top-level `transcription` section:
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `enabled` | `true` | Enables audio transcription for both chat-channel voice messages and WebUI/desktop microphone input. |
|
||||
| `provider` | `"groq"` | Transcription backend: `"groq"`, `"openai"`, `"openrouter"`, or `"xiaomi_mimo"`. |
|
||||
| `model` | provider default | Optional transcription model override. Defaults to `whisper-large-v3` for Groq, `whisper-1` for OpenAI, `openai/whisper-1` for OpenRouter, and `mimo-v2.5-asr` for Xiaomi MiMo ASR. OpenRouter accepts only speech-to-text models on its transcription endpoint, such as `nvidia/parakeet-tdt-0.6b-v3`, `openai/whisper-1`, or `openai/gpt-4o-transcribe`; chat LLMs are rejected there. |
|
||||
| `provider` | `"groq"` | Transcription backend: `"groq"`, `"openai"`, `"openrouter"`, `"xiaomi_mimo"`, or `"assemblyai"`. |
|
||||
| `model` | provider default | Optional transcription model override. Defaults to `whisper-large-v3` for Groq, `whisper-1` for OpenAI, `openai/whisper-1` for OpenRouter, `mimo-v2.5-asr` for Xiaomi MiMo ASR, and `universal-3-pro,universal-2` for AssemblyAI. OpenRouter accepts only speech-to-text models on its transcription endpoint, such as `nvidia/parakeet-tdt-0.6b-v3`, `openai/whisper-1`, or `openai/gpt-4o-transcribe`; chat LLMs are rejected there. AssemblyAI accepts a comma-separated model fallback list. |
|
||||
| `language` | `null` | Optional ISO-639 language hint, e.g. `"en"`, `"zh"`, `"ko"`, or `"ja"`. |
|
||||
| `maxDurationSec` | `120` | Maximum WebUI/desktop recording duration. |
|
||||
| `maxUploadMb` | `25` | Maximum WebUI/desktop audio upload size. |
|
||||
@@ -1155,6 +1116,9 @@ Transcription credentials are intentionally not stored in `transcription`. Put t
|
||||
|
||||
Selecting a transcription provider does not configure credentials by itself. For example, the effective provider may default to Groq for compatibility, but transcription is only usable when `providers.groq.apiKey` or the matching environment-backed config is available. The Settings UI writes only the top-level `transcription` fields.
|
||||
|
||||
If you are adding a new transcription provider, see
|
||||
[`development.md`](./development.md#adding-a-transcription-provider).
|
||||
|
||||
## Channel Settings
|
||||
|
||||
Global settings that apply to all channels. Configure under the `channels` section in `~/.nanobot/config.json`:
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
# Development
|
||||
|
||||
This page collects contributor-facing notes for extending nanobot. User-facing setup
|
||||
and runtime options live in [`configuration.md`](./configuration.md).
|
||||
|
||||
## Adding an LLM Provider
|
||||
|
||||
nanobot uses the provider registry in `nanobot/providers/registry.py` as the
|
||||
source of truth for LLM provider metadata. Most OpenAI-compatible providers need
|
||||
only two changes.
|
||||
|
||||
1. Add a `ProviderSpec` entry to `PROVIDERS`:
|
||||
|
||||
```python
|
||||
ProviderSpec(
|
||||
name="myprovider",
|
||||
keywords=("myprovider", "mymodel"),
|
||||
env_key="MYPROVIDER_API_KEY",
|
||||
display_name="My Provider",
|
||||
default_api_base="https://api.myprovider.com/v1",
|
||||
)
|
||||
```
|
||||
|
||||
2. Add a field to `ProvidersConfig` in `nanobot/config/schema.py`:
|
||||
|
||||
```python
|
||||
class ProvidersConfig(BaseModel):
|
||||
...
|
||||
myprovider: ProviderConfig = Field(default_factory=ProviderConfig)
|
||||
```
|
||||
|
||||
Environment variables, config matching, provider status, and WebUI credential
|
||||
display derive from those two entries.
|
||||
|
||||
Useful `ProviderSpec` options:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `default_api_base` | Default OpenAI-compatible base URL. |
|
||||
| `env_extras` | Additional environment variables derived from the provider config. |
|
||||
| `model_overrides` | Per-model request parameter overrides. |
|
||||
| `is_gateway` | Provider can route many model families, like OpenRouter. |
|
||||
| `detect_by_key_prefix` | Match configured gateways by API-key prefix. |
|
||||
| `detect_by_base_keyword` | Match configured gateways by API base URL. |
|
||||
| `strip_model_prefix` | Strip `provider/` before sending the model to the upstream API. |
|
||||
| `supports_max_completion_tokens` | Use `max_completion_tokens` instead of `max_tokens`. |
|
||||
| `is_transcription_only` | Provider has credentials but cannot serve chat completions. |
|
||||
|
||||
## Adding a Transcription Provider
|
||||
|
||||
Transcription is intentionally split into two layers:
|
||||
|
||||
- `nanobot/audio/transcription_registry.py` owns provider names, aliases, default
|
||||
models, and adapter loading.
|
||||
- `nanobot/providers/transcription.py` owns provider-specific HTTP behavior.
|
||||
|
||||
Credentials still live under `providers.<provider>` so chat channels, WebUI, and
|
||||
desktop resolve API keys and API bases the same way.
|
||||
|
||||
1. Add provider credentials to `ProvidersConfig`.
|
||||
|
||||
```python
|
||||
class ProvidersConfig(BaseModel):
|
||||
...
|
||||
my_stt: ProviderConfig = Field(default_factory=ProviderConfig)
|
||||
```
|
||||
|
||||
2. Add a `ProviderSpec` in `nanobot/providers/registry.py`.
|
||||
|
||||
For transcription-only providers, set `is_transcription_only=True` so they show up
|
||||
in credential/settings surfaces but stay out of chat model selection.
|
||||
|
||||
```python
|
||||
ProviderSpec(
|
||||
name="my_stt",
|
||||
keywords=("my_stt",),
|
||||
env_key="MY_STT_API_KEY",
|
||||
display_name="My STT",
|
||||
default_api_base="https://api.example.com/v1",
|
||||
is_transcription_only=True,
|
||||
)
|
||||
```
|
||||
|
||||
3. Add an adapter class in `nanobot/providers/transcription.py`.
|
||||
|
||||
Adapters receive resolved credentials and settings. They return an empty string
|
||||
for provider errors so channel voice messages fail quietly instead of crashing the
|
||||
agent loop.
|
||||
|
||||
```python
|
||||
class MySTTTranscriptionProvider:
|
||||
def __init__(
|
||||
self,
|
||||
api_key: str | None = None,
|
||||
api_base: str | None = None,
|
||||
language: str | None = None,
|
||||
model: str | None = None,
|
||||
):
|
||||
self.api_key = api_key or os.environ.get("MY_STT_API_KEY")
|
||||
self.api_base = api_base or "https://api.example.com/v1"
|
||||
self.language = language or None
|
||||
self.model = model or "my-default-stt-model"
|
||||
|
||||
async def transcribe(self, file_path: str | Path) -> str:
|
||||
...
|
||||
```
|
||||
|
||||
4. Register the adapter in `nanobot/audio/transcription_registry.py`.
|
||||
|
||||
```python
|
||||
TranscriptionProviderSpec(
|
||||
name="my_stt",
|
||||
default_model="my-default-stt-model",
|
||||
adapter="nanobot.providers.transcription:MySTTTranscriptionProvider",
|
||||
aliases=("mystt",),
|
||||
)
|
||||
```
|
||||
|
||||
5. Add tests.
|
||||
|
||||
At minimum, cover:
|
||||
|
||||
- config resolution in `tests/providers/test_transcription.py`
|
||||
- adapter request/response behavior and retry/error handling
|
||||
- WebUI settings payload/update behavior in `tests/webui/test_settings_api.py`
|
||||
- provider brand mapping if the provider appears in Settings
|
||||
|
||||
6. Update user-facing docs.
|
||||
|
||||
Add the provider to [`configuration.md`](./configuration.md) where users choose
|
||||
`transcription.provider`, but keep implementation details in this development
|
||||
guide.
|
||||
Reference in New Issue
Block a user