> ## Documentation Index
> Fetch the complete documentation index at: https://daily-docs-pr-5366.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Sarvam AI Speech-to-Text

> SarvamSTTService and SarvamRealtimeSTTService stream real-time STT for Indian languages with VAD-based or server-side endpointing.

## Overview

Sarvam AI offers two real-time speech recognition services for Indian languages:

* **`SarvamSTTService`**: Uses Sarvam's transcription WebSocket API with VAD-based segmentation and multiple audio formats (`saaras:v3` model)
* **`SarvamRealtimeSTTService`**: Uses Sarvam's realtime WebSocket endpoint (`saaras:v3-realtime` model) with server-side endpointing, in-band configuration updates, and lower latency

<CardGroup cols={2}>
  <Card title="Sarvam STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.sarvam.stt.html">
    Pipecat's API methods for Sarvam STT integration
  </Card>

  <Card title="SarvamSTTService Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-sarvam.py">
    Complete example with VAD-based turn detection
  </Card>

  <Card title="SarvamRealtimeSTTService Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-sarvam-realtime.py">
    Realtime STT with manual endpointing
  </Card>

  <Card title="Sarvam Documentation" icon="book" href="https://docs.sarvam.ai/api-reference-docs/api-guides-tutorials/speech-to-text/overview">
    Official Sarvam AI STT documentation and features
  </Card>

  <Card title="Sarvam AI Platform" icon="microphone" href="https://dashboard.sarvam.ai/">
    Access API keys and speech models
  </Card>
</CardGroup>

## Installation

To use Sarvam services, install the required dependency:

```bash theme={null}
uv add "pipecat-ai[sarvam]"
```

## Prerequisites

### Sarvam AI Account Setup

Before using Sarvam STT services, you need:

1. **Sarvam AI Account**: Sign up at [Sarvam AI](https://dashboard.sarvam.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**:
   * `SarvamSTTService`: Access to Saarika (STT) or Saaras (STT-Translate) models, including the `saaras:v3` model with support for multiple modes (transcribe, translate, verbatim, translit, codemix)
   * `SarvamRealtimeSTTService`: Access to the `saaras:v3-realtime` model

### Required Environment Variables

* `SARVAM_API_KEY`: Your Sarvam AI API key for authentication

## Configuration

### SarvamSTTService

<ParamField path="api_key" type="str" required>
  Sarvam API key for authentication.
</ParamField>

<ParamField path="model" type="str" default="saaras:v3" deprecated>
  Sarvam model to use. Allowed values: `"saarika:v2.5"` (standard STT),
  `"saaras:v2.5"` (STT-Translate, auto-detects language), `"saaras:v3"`
  (advanced, supports mode and fine-grained VAD). *Deprecated in v0.0.105. Use
  `settings=SarvamSTTService.Settings(...)` instead.*
</ParamField>

<ParamField path="sample_rate" type="int" default="None">
  Audio sample rate in Hz. Defaults to 16000 if not specified.
</ParamField>

<ParamField path="mode" type="Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix']" default="None">
  Mode of operation. Only applicable to models that support it (e.g.,
  `saaras:v3`). Defaults to the model's default mode.
</ParamField>

<ParamField path="input_audio_codec" type="str" default="wav">
  Audio codec/format of the input file.
</ParamField>

<ParamField path="params" type="SarvamSTTService.InputParams" default="None" deprecated>
  Configuration parameters for Sarvam STT service. *Deprecated in v0.0.105. Use
  `settings=SarvamSTTService.Settings(...)` instead.*
</ParamField>

<ParamField path="settings" type="SarvamSTTService.Settings" default="None">
  Runtime-configurable settings for the STT service. See [Settings](#settings)
  below.
</ParamField>

<ParamField path="keepalive_timeout" type="float" default="None">
  Seconds of no audio before sending silence to keep the connection alive.
  `None` disables keepalive.
</ParamField>

<ParamField path="ttfs_p99_latency" type="float" default="SARVAM_TTFS_P99">
  P99 latency from speech end to final transcript in seconds. Override for your
  deployment. See [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark).
</ParamField>

<ParamField path="keepalive_interval" type="float" default="5.0">
  Seconds between idle checks when keepalive is enabled.
</ParamField>

### Settings

Runtime-configurable settings passed via the `settings` constructor argument using `SarvamSTTService.Settings(...)`. These can be updated mid-conversation with `STTUpdateSettingsFrame`. See [Service Settings](/pipecat/fundamentals/service-settings) for details.

| Parameter                       | Type              | Default | Description                                                                                                                                                                                                                        |
| ------------------------------- | ----------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`                         | `str`             | `None`  | STT model identifier. *(Inherited from base STT settings.)*                                                                                                                                                                        |
| `language`                      | `Language \| str` | `None`  | Target language for transcription. *(Inherited from base STT settings.)* Behavior varies by model: `saarika:v2.5` defaults to "unknown" (auto-detect), `saaras:v2.5` ignores this (auto-detects), `saaras:v3` defaults to "en-IN". |
| `prompt`                        | `str`             | `None`  | Optional prompt to guide transcription/translation style. Only applicable to `saaras:v2.5`.                                                                                                                                        |
| `vad_signals`                   | `bool`            | `None`  | Enable VAD signals in responses. When enabled, the service broadcasts `UserStartedSpeakingFrame` and `UserStoppedSpeakingFrame` from the server.                                                                                   |
| `high_vad_sensitivity`          | `bool`            | `None`  | Enable high VAD sensitivity for more responsive speech detection.                                                                                                                                                                  |
| `positive_speech_threshold`     | `float`           | `None`  | VAD probability threshold (0.0-1.0) above which a frame is considered speech. Only for `saaras:v3`.                                                                                                                                |
| `negative_speech_threshold`     | `float`           | `None`  | VAD probability threshold (0.0-1.0) below which a frame is considered silence. Only for `saaras:v3`.                                                                                                                               |
| `min_speech_frames`             | `int`             | `None`  | Minimum consecutive speech frames to start a speech segment. Only for `saaras:v3`.                                                                                                                                                 |
| `first_turn_min_speech_frames`  | `int`             | `None`  | Minimum speech frames for the first user turn. Only for `saaras:v3`.                                                                                                                                                               |
| `negative_frames_count`         | `int`             | `None`  | Number of silence frames within the window to end a speech segment. Only for `saaras:v3`.                                                                                                                                          |
| `negative_frames_window`        | `int`             | `None`  | Sliding window size (in frames) for counting negative frames. Only for `saaras:v3`.                                                                                                                                                |
| `start_speech_volume_threshold` | `float`           | `None`  | Volume level (dB) below which audio is too quiet to be speech. Only for `saaras:v3`.                                                                                                                                               |
| `interrupt_min_speech_frames`   | `int`             | `None`  | Minimum speech frames to register a barge-in/interruption. Only for `saaras:v3`.                                                                                                                                                   |
| `pre_speech_pad_frames`         | `int`             | `None`  | Number of audio frames to prepend before detected speech onset. Only for `saaras:v3`.                                                                                                                                              |
| `num_initial_ignored_frames`    | `int`             | `None`  | Number of leading audio frames to skip at connection start. Only for `saaras:v3`.                                                                                                                                                  |

### SarvamRealtimeSTTService

<ParamField path="api_key" type="str" required>
  Sarvam API key for authentication.
</ParamField>

<ParamField path="base_url" type="str" default="wss://api.sarvam.ai/speech-to-text-realtime/ws">
  Realtime STT websocket endpoint.
</ParamField>

<ParamField path="endpointing" type="Literal['vad', 'manual']" default="vad">
  Which side detects turn boundaries: `vad` for Sarvam's own detection, or
  `manual` for the pipeline's. Decides the turn strategies this service asks the
  user aggregator to run. Defaults to `vad`.
</ParamField>

<ParamField path="sample_rate" type="int" default="None">
  Declared input audio sample rate, 8000 or 16000. `None` adopts the pipeline's
  input rate.
</ParamField>

<ParamField path="return_timestamps" type="bool" default="False">
  Whether final transcripts should include segment offsets.
</ParamField>

<ParamField path="prefix_padding_ms" type="int" default="None">
  Optional VAD prefix padding, used only under `endpointing="vad"`.
</ParamField>

<ParamField path="settings" type="SarvamRealtimeSTTService.Settings" default="None">
  Runtime-updatable realtime settings. See [Realtime
  Settings](#realtime-settings) below.
</ParamField>

<ParamField path="should_interrupt" type="bool" default="True">
  Whether the bot should be interrupted when Sarvam detects user speech. Passed
  to the user turn strategies this service recommends.
</ParamField>

<ParamField path="ttfs_p99_latency" type="float" default="SARVAM_REALTIME_TTFS_P99">
  P99 latency from speech end to final transcript in seconds. Override for your
  deployment.
</ParamField>

### Realtime Settings

Runtime-configurable settings passed via the `settings` constructor argument using `SarvamRealtimeSTTService.Settings(...)`. These can be updated mid-conversation via `config.update()` or `STTUpdateSettingsFrame`.

| Parameter                | Type                                                                    | Default              | Description                                                                                                |
| ------------------------ | ----------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------- |
| `model`                  | `str`                                                                   | `saaras:v3-realtime` | STT model identifier. *(Fixed for realtime service.)*                                                      |
| `language`               | `Language \| str`                                                       | `None`               | Target language for transcription. *(Inherited from base STT settings.)*                                   |
| `language_code`          | `str`                                                                   | `en-IN`              | Sarvam realtime language code or `auto` for automatic detection.                                           |
| `stream_type`            | `Literal['fast', 'balanced', 'simulated']`                              | `balanced`           | Streaming cadence: `fast` for lowest latency, `balanced` for moderate latency, or `simulated` for testing. |
| `mode`                   | `Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix']` | `transcribe`         | Realtime STT task mode.                                                                                    |
| `prompt`                 | `str \| None`                                                           | `None`               | Optional decoding prompt to guide transcription style.                                                     |
| `threshold`              | `float \| None`                                                         | `None`               | Optional VAD sensitivity threshold (only applies under `endpointing="vad"`).                               |
| `silence_duration_ms`    | `int \| None`                                                           | `None`               | Optional silence duration for end-of-speech detection (only applies under `endpointing="vad"`).            |
| `min_speech_duration_ms` | `int \| None`                                                           | `None`               | Optional minimum speech duration (only applies under `endpointing="vad"`).                                 |

## Usage

### Basic Setup (SarvamSTTService)

```python theme={null}
from pipecat.services.sarvam.stt import SarvamSTTService

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
)
```

### With Language and Model Configuration

```python theme={null}
from pipecat.services.sarvam.stt import SarvamSTTService
from pipecat.transcriptions.language import Language

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    mode="transcribe",
    settings=SarvamSTTService.Settings(
        model="saaras:v3",
        language=Language.HI_IN,
        prompt="Transcribe Hindi conversation about technology.",
    ),
)
```

### With Server-Side VAD

```python theme={null}
from pipecat.services.sarvam.stt import SarvamSTTService

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamSTTService.Settings(
        vad_signals=True,
        high_vad_sensitivity=True,
    ),
)
```

### Basic Realtime Setup

```python theme={null}
from pipecat.services.sarvam.stt import SarvamRealtimeSTTService

stt = SarvamRealtimeSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
)
```

### Realtime with Language and Stream Type

```python theme={null}
from pipecat.services.sarvam.stt import SarvamRealtimeSTTService
from pipecat.transcriptions.language import Language

stt = SarvamRealtimeSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamRealtimeSTTService.Settings(
        language_code="hi-IN",
        stream_type="fast",
        mode="transcribe",
    ),
)
```

### Realtime with Manual Endpointing

```python theme={null}
from pipecat.services.sarvam.stt import SarvamRealtimeSTTService
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

# With manual endpointing, the pipeline drives turn boundaries
stt = SarvamRealtimeSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    endpointing="manual",
    settings=SarvamRealtimeSTTService.Settings(
        language_code="en-IN",
        stream_type="balanced",
    ),
)

# A VAD analyzer is required to mark turn boundaries
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)
```

### Updating Realtime Configuration

```python theme={null}
# Update settings mid-conversation without reconnecting
await stt.update_config(
    language_code="ta-IN",
    stream_type="fast",
    threshold=0.5,
)
```

## Notes

### SarvamSTTService

* **Default model changed**: As of this update, the default model is `saaras:v3` (previously `saarika:v2.5`). Applications that relied on the previous default should set `settings=SarvamSTTService.Settings(model="saarika:v2.5")` explicitly.
* **Supported languages**: Bengali (bn-IN), Gujarati (gu-IN), Hindi (hi-IN), Kannada (kn-IN), Malayalam (ml-IN), Marathi (mr-IN), Tamil (ta-IN), Telugu (te-IN), Punjabi (pa-IN), Odia (od-IN), English (en-IN), and Assamese (as-IN).
* **Model-specific parameter validation**: The service validates that parameters are compatible with the selected model. For example, `prompt` is only supported with `saaras:v2.5`, `language` is not supported with `saaras:v2.5` (which auto-detects language), and the fine-grained VAD parameters are only supported with `saaras:v3`.
* **Fine-grained VAD tuning (saaras:v3 only)**: The `saaras:v3` model supports server-side VAD with 10 tuning parameters for speech detection thresholds, frame-count controls, pre-speech padding, interruption sensitivity, and initial-frame skipping. These parameters are only available with the `saaras:v3` model.
* **VAD modes**: When `vad_signals=False` (default), the service relies on Pipecat's local VAD and flushes the server buffer on `VADUserStoppedSpeakingFrame`. When `vad_signals=True`, the service uses Sarvam's server-side VAD and broadcasts speaking frames from the server.

### SarvamRealtimeSTTService

* **Supported sample rates**: Only 8000 Hz and 16000 Hz are supported. The service validates the sample rate at initialization.
* **Supported languages**: `auto`, Bengali (bn-IN), Gujarati (gu-IN), Hindi (hi-IN), Kannada (kn-IN), Malayalam (ml-IN), Marathi (mr-IN), Tamil (ta-IN), Telugu (te-IN), Punjabi (pa-IN), Odia (or-IN), English (en-IN), Assamese (as-IN), Urdu (ur-IN), Nepali (ne-IN), Konkani (kok-IN), Kashmiri (ks-IN), Sindhi (sd-IN), Sanskrit (sa-IN), Santali (sat-IN), Manipuri (mni-IN), Bodo (brx-IN), Maithili (mai-IN), and Dogri (doi-IN).
* **Endpointing modes**:
  * `endpointing="vad"` (default): Sarvam's server-side VAD decides turn boundaries. The service proposes turn frames via `ProposedUserStartedSpeakingFrame` and `ProposedUserStoppedSpeakingFrame`.
  * `endpointing="manual"`: The pipeline drives turn boundaries via `VADUserStartedSpeakingFrame` and `VADUserStoppedSpeakingFrame`. Requires a VAD analyzer on the user aggregator.
* **VAD analyzer required**: A VAD analyzer is required in either endpointing mode. Under `vad` it times transcription latency; under `manual` it also marks the turn for Sarvam.
* **In-band configuration updates**: Settings can be updated mid-conversation via `update_config()` without reconnecting. Connection-only values (`sample_rate`, `return_timestamps`, `prefix_padding_ms`, `endpointing`) cannot be updated at runtime.
* **Stream types**: `fast` provides lowest latency, `balanced` provides moderate latency with better quality, and `simulated` is for testing. The stream type controls server-side flush cadence; the client always sends audio in 50ms chunks.

<Tip>
  The `InputParams` / `params=` pattern is deprecated as of v0.0.105. Use
  `Settings` / `settings=` instead. See the [Service Settings
  guide](/pipecat/fundamentals/service-settings) for migration details.
</Tip>

## Event Handlers

In addition to the standard [service connection events](/api-reference/server/events/service-events) (`on_connected`, `on_disconnected`, `on_connection_error`), Sarvam STT provides:

| Event               | Description                         |
| ------------------- | ----------------------------------- |
| `on_speech_started` | Speech detected in the audio stream |
| `on_speech_stopped` | Speech stopped                      |
| `on_utterance_end`  | End of utterance detected           |

```python theme={null}
@stt.event_handler("on_speech_started")
async def on_speech_started(service):
    print("User started speaking")

@stt.event_handler("on_utterance_end")
async def on_utterance_end(service):
    print("Utterance ended")
```