# Audio Automatic Fallback to Transcription

## Overview

The system now **automatically transcribes audio** when sending to APIs that don't support native audio input. This ensures audio content is never silently dropped and always reaches the model in some form.

## The Problem (Before)

```python
# Sending audio to OpenAI (doesn't support audio)
{
    "type": "audio",
    "url": "audio.m4a",
    # auto_transcribe not set
}

# Result: Audio was SILENTLY IGNORED ❌
# - No transcription happened
# - No warning was shown
# - Audio content was lost
# - Model never saw the audio
```

## The Solution (Now)

```python
# Sending audio to OpenAI (doesn't support audio)
{
    "type": "audio",
    "url": "audio.m4a",
    # auto_transcribe not set
}

# Result: AUTOMATIC FALLBACK ✓
# - System detects OpenAI doesn't support audio
# - Automatically transcribes using Groq
# - Sends transcription as text
# - Model receives the content
# - Warning logged in debug mode
```

## How It Works

### Decision Logic

```
Does audio have auto_transcribe=True?
├─ YES → Transcribe (user explicitly requested)
└─ NO → Does API support native audio?
    ├─ YES → Send audio directly (e.g., Google Gemini)
    └─ NO → Transcribe automatically (fallback)
```

### API Support Matrix

| API Provider | Native Audio Support | Behavior |
|--------------|---------------------|----------|
| **Google Gemini** | ✅ Yes | Audio sent directly (unless auto_transcribe=True) |
| **OpenAI** | ❌ No | Auto-transcribed as fallback |
| **Anthropic** | ❌ No | Auto-transcribed as fallback |
| **Groq** | ❌ No | Auto-transcribed as fallback |
| **xAI** | ❌ No | Auto-transcribed as fallback |
| **Cerebras** | ❌ No | Auto-transcribed as fallback |
| **Together** | ❌ No | Auto-transcribed as fallback |

## Examples

### Example 1: OpenAI (No Audio Support)

```python
# Configuration
{
    "ai_model_id": "gpt-4.1-mini",
    "config": {
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "url": "https://example.com/audio.m4a",
                    # No auto_transcribe flag
                },
                {
                    "type": "text",
                    "text": "Summarize this audio."
                }
            ]
        }]
    }
}

# What happens:
# 1. System detects OpenAI doesn't support audio
# 2. Automatically transcribes audio using Groq
# 3. Replaces audio with: "[Audio Transcription]: {text}"
# 4. Sends to OpenAI as text
# 5. Logs warning: "API doesn't support audio - transcribing as fallback"
```

### Example 2: Google (Has Audio Support)

```python
# Configuration
{
    "ai_model_id": "gemini-3-flash-preview",
    "config": {
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "url": "https://example.com/audio.m4a",
                    # No auto_transcribe flag
                },
                {
                    "type": "text",
                    "text": "Summarize this audio."
                }
            ]
        }]
    }
}

# What happens:
# 1. System detects Google supports audio
# 2. Sends audio directly to Google (no transcription)
# 3. Google processes audio natively
# 4. No fallback needed ✓
```

### Example 3: Explicit Transcription (Any API)

```python
# Configuration
{
    "ai_model_id": "gemini-3-flash-preview",  # Even with Google
    "config": {
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "url": "https://example.com/audio.m4a",
                    "auto_transcribe": True,  # Explicit request
                },
                {
                    "type": "text",
                    "text": "Summarize this audio."
                }
            ]
        }]
    }
}

# What happens:
# 1. User explicitly requested transcription
# 2. Transcribes using Groq (even though Google supports audio)
# 3. Sends transcription as text
# 4. Honors user's explicit choice ✓
```

## Debug Output

### Automatic Fallback (API Doesn't Support Audio)

```
[Unified Client] API 'openai_standard' doesn't support audio - auto-transcribing as fallback
[Audio Preprocessing] ⚠️ API doesn't support audio - transcribing as fallback: https://example.com/audio.m4a
[Audio Transcription] Using cached transcription for audio (171 characters)
[Audio Preprocessing] ✓ Audio transcribed successfully (171 characters, 17.3s)
[Usage Tracking] Tracked Groq transcription: 17.3s (17.3s billed)
```

### Explicit Transcription (User Requested)

```
[Unified Client] Audio auto-transcription explicitly enabled - preprocessing messages
[Audio Preprocessing] Auto-transcribing audio from: https://example.com/audio.m4a
[Audio Transcription] Using cached transcription for audio (171 characters)
[Audio Preprocessing] ✓ Audio transcribed successfully (171 characters, 17.3s)
```

### Direct Audio (API Supports It)

```
# No preprocessing messages - audio sent directly
```

## Benefits

### 1. No Silent Failures

**Before:**
```python
# Audio sent to OpenAI
# → Silently dropped
# → No warning
# → User thinks it worked but audio was lost
```

**After:**
```python
# Audio sent to OpenAI
# → Automatically transcribed
# → Warning logged
# → Audio content preserved as text
```

### 2. Universal Audio Support

```python
# Same code works with ANY model
audio_message = {
    "type": "audio",
    "url": "audio.m4a",
}

# Works with:
# - Google (native audio)
# - OpenAI (auto-transcribed)
# - Anthropic (auto-transcribed)
# - Any provider (automatic fallback)
```

### 3. Cost Optimization

```python
# Google model - no transcription cost
{
    "ai_model_id": "gemini-3-flash-preview",
    "content": [{"type": "audio", "url": "audio.m4a"}]
}
# → Sent directly, no Groq API call

# OpenAI model - minimal transcription cost
{
    "ai_model_id": "gpt-4.1-mini",
    "content": [{"type": "audio", "url": "audio.m4a"}]
}
# → Auto-transcribed, small Groq cost (~$0.0002 for 17s)
```

### 4. Explicit Control When Needed

```python
# Force transcription even with Google
{
    "ai_model_id": "gemini-3-flash-preview",
    "content": [{
        "type": "audio",
        "url": "audio.m4a",
        "auto_transcribe": True,  # Explicit override
    }]
}
# → Transcribed despite Google supporting audio
```

## Implementation Details

### Audio Support Detection

```python
# ai/audio/audio_support.py

AUDIO_SUPPORTED_API_CLASSES = {
    "google_standard",
    "google_thinking",
    "google_thinking_3",
    "google_image_generation",
}

def api_class_supports_audio(api_class: str) -> bool:
    """Check if API class supports native audio"""
    return api_class.lower() in AUDIO_SUPPORTED_API_CLASSES
```

### Preprocessing Logic

```python
# ai/audio/audio_preprocessing.py

def preprocess_audio_in_messages(messages, api_class, debug):
    api_supports_audio = api_class_supports_audio(api_class)
    
    for content in message.content:
        if isinstance(content, AudioContent):
            should_transcribe = False
            
            if content.auto_transcribe:
                # User explicitly requested
                should_transcribe = True
                reason = "explicit"
            elif not api_supports_audio:
                # Automatic fallback
                should_transcribe = True
                reason = "fallback"
                content.auto_transcribe = True  # Enable for get_transcription()
            
            if should_transcribe:
                transcription = content.get_transcription()
                # Replace with TextContent...
```

### Usage Tracking

Automatic fallback transcriptions are tracked just like explicit transcriptions:

```python
{
    "usage_history": [
        {
            "api": "groq_transcription",  # Fallback transcription
            "model": "whisper-large-v3-turbo",
            "input_tokens": 1729,  # 17.29s * 100
            "metadata": {
                "duration_seconds": 17.29,
                "audio_source": "https://..."
            }
        },
        {
            "api": "openai",  # Main model
            "model": "gpt-4.1-mini",
            "input_tokens": 89,
            "output_tokens": 35
        }
    ]
}
```

## Migration Guide

### Old Code (Risky)

```python
# Audio might be silently dropped
{
    "type": "audio",
    "url": "audio.m4a",
}
```

### New Code (Safe)

```python
# Same code, but now safe - automatic fallback
{
    "type": "audio",
    "url": "audio.m4a",
}

# Or be explicit if you prefer
{
    "type": "audio",
    "url": "audio.m4a",
    "auto_transcribe": True,  # Explicit (works everywhere)
}

# Or specify you want native audio (Google only)
{
    "type": "audio",
    "url": "audio.m4a",
    "auto_transcribe": False,  # Only works if API supports audio
}
```

## Testing

### Test Fallback Behavior

```python
# Test with OpenAI (should auto-transcribe)
settings = {
    "ai_model_id": "gpt-4.1-mini",
    "debug": True,
    "config": {
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "url": "https://example.com/audio.m4a",
                    # No auto_transcribe flag
                },
                {
                    "type": "text",
                    "text": "Summarize this audio."
                }
            ]
        }]
    }
}

# Expected debug output:
# "API 'openai_standard' doesn't support audio - auto-transcribing as fallback"
# "⚠️ API doesn't support audio - transcribing as fallback: ..."
```

### Test Native Audio

```python
# Test with Google (should send directly)
settings = {
    "ai_model_id": "gemini-3-flash-preview",
    "debug": True,
    "config": {
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "url": "https://example.com/audio.m4a",
                    # No auto_transcribe flag
                },
                {
                    "type": "text",
                    "text": "Summarize this audio."
                }
            ]
        }]
    }
}

# Expected: No preprocessing messages (audio sent directly)
```

## Adding New Audio-Capable APIs

When a new API adds audio support, update the configuration:

```python
# ai/audio/audio_support.py

AUDIO_SUPPORTED_API_CLASSES.add("new_api_class_name")
```

That's it! The system will automatically stop transcribing for that API.

## Summary

✅ **Automatic Fallback** - Audio never silently dropped
✅ **Smart Detection** - Knows which APIs support audio
✅ **Warning Logs** - Clear visibility when fallback happens
✅ **Usage Tracking** - Fallback transcriptions tracked
✅ **Explicit Control** - Can override with auto_transcribe flag
✅ **Cost Efficient** - Only transcribes when necessary
✅ **Universal Code** - Same audio code works with any API

Your audio content is now guaranteed to reach the model! 🎉
