# Audio Handling: Two Approaches Compared

## Overview

Our system supports two distinct approaches for handling audio content:

1. **Groq Transcription First** (auto_transcribe=True): Transcribe audio to text using Groq Whisper API, then send text to any model
2. **Direct Audio** (auto_transcribe=False): Send audio directly to models that support native audio processing (like Google Gemini)

## Approach 1: Groq Transcription First

### How It Works

```
Audio File
    ↓
Groq Whisper API (Transcription)
    ↓
Text Transcription
    ↓
Any Text-Based Model (OpenAI, Anthropic, Google, etc.)
    ↓
Response
```

### Configuration

```python
{
    "role": "user",
    "content": [
        {
            "type": "audio",
            "url": "https://example.com/audio.m4a",
            "auto_transcribe": True,  # ← Enable Groq transcription
            "transcription_model": "whisper-large-v3-turbo",
            "transcription_language": "en",
        },
        {
            "type": "text",
            "text": "Please summarize what was said.",
        },
    ],
}
```

### What Happens

1. Audio is detected with `auto_transcribe=True`
2. Before sending to the model, audio is transcribed using Groq Whisper API
3. Audio content is replaced with text: `[Audio Transcription]: {transcribed_text}`
4. Text is sent to the model (any model that accepts text)
5. Model responds based on the transcription

### Advantages

✅ **Universal**: Works with ALL text-based models (OpenAI, Anthropic, Groq, xAI, Cerebras, Together, Google)
✅ **Cost-Effective**: Groq transcription is very cheap compared to native audio processing
✅ **Fast**: Near-instant transcription with Groq's infrastructure
✅ **Cached**: Transcriptions are cached to avoid redundant API calls
✅ **Trackable**: Full usage metrics and quality indicators
✅ **Flexible**: Can use different Whisper models based on needs

### Disadvantages

❌ **Two-Step Process**: Requires transcription before model processing
❌ **Potential Quality Loss**: Transcription may miss audio nuances (tone, emotion, etc.)
❌ **Text-Only**: Model only sees text, not original audio
❌ **Extra API Call**: Adds a Groq API call before the main model call

### Best For

- **Cost Optimization**: When you want to minimize costs
- **Universal Compatibility**: When using models without audio support
- **Batch Processing**: When processing many audio files
- **Long Audio**: When audio is very long (cheaper to transcribe than process natively)
- **Multilingual**: When you need translation to English

### Cost Example

For a 60-second audio file:
- Groq Transcription: ~$0.0001 (extremely cheap)
- Google Text Processing: Standard text pricing
- **Total**: Very cost-effective

## Approach 2: Direct Audio to Google

### How It Works

```
Audio File
    ↓
Google Gemini API (Native Audio Processing)
    ↓
Response (based on audio analysis)
```

### Configuration

```python
{
    "role": "user",
    "content": [
        {
            "type": "audio",
            "url": "https://example.com/audio.m4a",
            "auto_transcribe": False,  # ← Send directly to model
        },
        {
            "type": "text",
            "text": "Please summarize what was said.",
        },
    ],
}
```

### What Happens

1. Audio is detected with `auto_transcribe=False`
2. Audio is sent directly to Google Gemini API
3. Google processes the audio internally
4. Google responds based on native audio analysis

### Advantages

✅ **Maximum Quality**: Model analyzes original audio (tone, emotion, nuances)
✅ **Single API Call**: No intermediate transcription step
✅ **Native Processing**: Leverages Google's advanced audio understanding
✅ **Simpler Flow**: Direct audio-to-response pipeline
✅ **Audio Features**: Can analyze non-verbal cues, music, sound effects

### Disadvantages

❌ **Google Only**: Only works with Google Gemini models
❌ **Higher Cost**: Native audio processing is more expensive
❌ **No Transcription Cache**: Can't reuse transcription for other models
❌ **Limited Control**: Can't choose transcription model or settings

### Best For

- **Maximum Accuracy**: When you need the highest quality audio analysis
- **Audio Nuances**: When tone, emotion, or non-verbal cues matter
- **Google Ecosystem**: When already using Google Gemini
- **Short Audio**: When audio is brief (cost difference is minimal)
- **Real-Time**: When you need immediate audio-to-response

### Cost Example

For a 60-second audio file:
- Google Native Audio Processing: Higher cost (varies by model)
- **Total**: More expensive but higher quality

## Side-by-Side Comparison

| Feature | Groq Transcription First | Direct Audio to Google |
|---------|-------------------------|------------------------|
| **Supported Models** | All text-based models | Google Gemini only |
| **Cost** | Very low | Higher |
| **Speed** | Fast (2 API calls) | Fast (1 API call) |
| **Quality** | Text-based (transcription) | Native audio analysis |
| **Audio Nuances** | Lost in transcription | Preserved |
| **Caching** | Yes (transcription cached) | No |
| **Multilingual** | Excellent (Whisper) | Good (Google) |
| **Translation** | Built-in (to English) | Limited |
| **Usage Tracking** | Detailed metrics | Standard |
| **Best For** | Cost, flexibility | Quality, simplicity |

## Test Cases

### Test 1: Groq Transcription → Google

```python
audio_transcription_test_settings = {
    "ai_model_id": "gemini-3-flash-preview",
    "config": {
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "audio",
                        "url": "https://example.com/audio.m4a",
                        "auto_transcribe": True,
                        "transcription_model": "whisper-large-v3-turbo",
                        "transcription_language": "en",
                    },
                    {
                        "type": "text",
                        "text": "Please summarize what was said in 2-3 sentences.",
                    },
                ],
            },
        ],
    },
}
```

**Expected Flow**:
1. Audio downloaded from URL
2. Groq Whisper transcribes audio
3. Transcription: `[Audio Transcription]: {text}`
4. Google Gemini receives text
5. Google responds with summary

### Test 2: Direct Audio → Google

```python
audio_direct_google_settings = {
    "ai_model_id": "gemini-3-flash-preview",
    "config": {
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "audio",
                        "url": "https://example.com/audio.m4a",
                        "auto_transcribe": False,
                    },
                    {
                        "type": "text",
                        "text": "Please summarize what was said in 2-3 sentences.",
                    },
                ],
            },
        ],
    },
}
```

**Expected Flow**:
1. Audio downloaded from URL
2. Audio sent directly to Google Gemini
3. Google processes audio natively
4. Google responds with summary

## When to Use Which Approach

### Use Groq Transcription First When:

1. **Using non-Google models** (OpenAI, Anthropic, etc.)
2. **Cost is a primary concern**
3. **Processing many audio files** (batch processing)
4. **Audio is long** (>5 minutes)
5. **Need transcription for other purposes** (documentation, search, etc.)
6. **Multilingual content** that needs translation
7. **Want detailed quality metrics**

### Use Direct Audio to Google When:

1. **Already using Google Gemini**
2. **Quality is paramount** (need audio nuances)
3. **Audio is short** (<2 minutes)
4. **Analyzing tone, emotion, or non-verbal cues**
5. **Need simplest possible flow**
6. **Processing music or sound effects** (not just speech)
7. **Real-time audio analysis**

## Hybrid Approach

You can even use both approaches in the same application:

```python
def process_audio(audio_url, model_id, audio_length_seconds):
    """Smart audio processing based on model and audio length"""
    
    # For Google models with short audio, use direct
    if model_id.startswith("gemini") and audio_length_seconds < 120:
        return {
            "type": "audio",
            "url": audio_url,
            "auto_transcribe": False,  # Direct to Google
        }
    
    # For everything else, use Groq transcription
    else:
        return {
            "type": "audio",
            "url": audio_url,
            "auto_transcribe": True,  # Groq first
            "transcription_model": "whisper-large-v3-turbo",
        }
```

## Implementation Details

### Preprocessing Pipeline

When `auto_transcribe=True`, the preprocessing happens in `UnifiedAIClient.execute()`:

```python
# In unified_client.py
if should_preprocess_audio(config.messages):
    config.messages = preprocess_audio_in_messages(config.messages, debug=debug)
```

This:
1. Detects audio with `auto_transcribe=True`
2. Calls `AudioContent.get_transcription()`
3. Replaces audio with text content
4. Preserves metadata about the transcription

### Metadata Storage

Transcription metadata is automatically stored:

```python
{
    "transcription": {
        "usage": {
            "duration_seconds": 60.5,
            "billed_duration": 60.5,
            "model": "whisper-large-v3-turbo",
            "language": "en",
            "operation": "transcription",
            "file_size_mb": 2.3,
        },
        "quality_metrics": {
            "avg_logprob": -0.15,
            "avg_no_speech_prob": 0.02,
            "avg_compression_ratio": 1.85,
        },
        "language": "en",
        "duration": 60.5,
    }
}
```

## Running the Tests

```bash
# Run the execution test
python ai/tests/execution_test.py
```

In the test file, uncomment the desired test:

```python
# Test Groq transcription approach
settings_to_use = audio_transcription_test_settings

# OR test direct audio approach
settings_to_use = audio_direct_google_settings
```

## Conclusion

Both approaches have their place:

- **Groq Transcription First**: Universal, cost-effective, flexible
- **Direct Audio to Google**: High quality, simple, native processing

Choose based on your specific needs for cost, quality, and model compatibility.
