Metadata-Version: 2.4
Name: filter_connector_gcs
Version: 2.0.1
License-Expression: Apache-2.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: google-cloud-storage==2.17.0
Requires-Dist: openfilter[all]<0.2.0,>=0.1.0
Provides-Extra: dev
Requires-Dist: build==1.2.1; extra == "dev"
Requires-Dist: setuptools==72.2.0; extra == "dev"
Requires-Dist: twine<7,>=6.1.0; extra == "dev"
Requires-Dist: wheel==0.44.0; extra == "dev"
Requires-Dist: pytest==8.3.4; extra == "dev"
Requires-Dist: pytest-cov==6.0.0; extra == "dev"
Dynamic: license-file

# GCS Upload

A specialized OpenFilter component that uploads video segments and images to Google Cloud Storage buckets. Supports segmented video outputs, image uploads from upstream filters, optional manifest generation, and comprehensive configuration validation.

## Features

- **Google Cloud Storage Output**: Uploads video segments to `gs://` destinations with wildcards and segment intervals
- **Image Upload Support**: Monitors directories for images from upstream filters and uploads them to GCS
- **Manifest Generation**: Creates JSON manifests listing all uploaded files with nested field support
- **Configuration Validation**: Prevents typos with helpful error messages and suggestions
- **Concurrent Uploading**: Background threads handle uploads with file locking for images
- **Flexible Templates**: Manifest templates from `file://`, `gs://`, or cached sources

## Quick Start

### Prerequisites

**IMPORTANT!** You need access to GCP and the `gcloud` CLI installed and authenticated:

```bash
# Authenticate with Google Cloud
gcloud auth login
gcloud auth application-default login

# Set your project (replace with your project ID)
gcloud config set project your-project-id
gcloud auth configure-docker us-west1-docker.pkg.dev
```

### Installation

```bash
# Create virtual environment
virtualenv venv
source venv/bin/activate

# Install the filter
make install
```

### Basic Usage

```python
from openfilter import Filter

# Simple video recording pipeline
filters = [
    Filter("VideoIn", {
        "sources": "file://sample_video.mp4",
        "outputs": "tcp://127.0.0.1:5550"
    }),
    Filter("FilterConnectorGCS", {
        "sources": "tcp://127.0.0.1:5550",
        "outputs": "gs://my-bucket/videos/video_%Y-%m-%d_%H-%M-%S.mp4!segtime=0.5",
        "workdir": "./temp_videos",
        "timeout": 60.0
    }),
    Filter("Webvis", {
        "sources": "tcp://127.0.0.1:5550",
        "outputs": "tcp://127.0.0.1:8080"
    })
]

Filter.run_multi(filters, exit_time=30.0)
```

## Documentation

For comprehensive documentation including:
- Complete configuration reference
- Sample pipelines and use cases
- Troubleshooting guides
- API documentation

**Refer to [docs/overview.md](docs/overview.md)**

## Development

### Running Locally

```bash
# Run the filter locally
make run

# Navigate to http://localhost:8000 to see the video
```

### Running in Docker

```bash
# Build the filter docker image
make build-image

# Generate docker-compose.yaml (if needed)
make compose

# Run the containerized filter
make run-image
```

**Note**: If your filter uses GPU, ensure the `deploy:` section in `docker-compose.yaml` includes GPU configuration:

```yaml
deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]
```

### Testing

```bash
# Run unit tests
make test

# Run specific test files
pytest tests/test_vid2gs.py -v
pytest tests/test_smoke_simple.py -v
pytest tests/test_integration_config_normalization.py -v
```

## Configuration Examples

### Basic Video Upload
```json
{
    "id": "gcs_uploader",
    "sources": "tcp://127.0.0.1:5550",
    "outputs": "gs://my-bucket/videos/video_%Y-%m-%d_%H-%M-%S.mp4!segtime=0.5",
    "workdir": "./temp_videos",
    "timeout": 60.0
}
```

### With Image Upload and Manifest
```json
{
    "id": "gcs_uploader",
    "sources": "tcp://127.0.0.1:5550",
    "outputs": "gs://my-bucket/videos/stream_%Y-%m-%d_%H-%M-%S.mp4!segtime=1.0",
    "image_directory": "./unique_frames",
    "manifest": "file://manifest_template.json",
    "manifest_field": "stream_data.files",
    "workdir": "./temp_processing",
    "timeout": 120.0
}
```

### Multiple Outputs
```json
{
    "id": "multi_gcs_uploader",
    "sources": "tcp://127.0.0.1:5550",
    "outputs": [
        "gs://primary-bucket/videos/feed_%Y-%m-%d_%H-%M-%S.mp4!segtime=0.2",
        "gs://backup-bucket/archive/feed_%Y-%m-%d_%H-%M-%S.mp4!segtime=2.0"
    ],
    "manifest": "gs://primary-bucket/templates/manifest.json",
    "manifest_field": "recordings.files",
    "workdir": "./temp",
    "timeout": 180.0
}
```

## Environment Variables

| Variable | Required | Description |
|----------|----------|-------------|
| `GOOGLE_APPLICATION_CREDENTIALS` | Yes | Path to GCP service account key JSON file |

## Use Cases

- **Security Camera Monitoring**: 24/7 recording with automatic cloud backup
- **Content Creation**: Multi-destination uploads with thumbnail extraction
- **IoT Data Collection**: Device-specific data collection with structured metadata
- **Live Streaming Archive**: Real-time streaming with dual outputs

See [docs/overview.md](docs/overview.md) for detailed use case examples and sample pipelines.

## Publishing

To publish a new version:

1. **Update Version**: Ensure `VERSION` file has a production semver tag (e.g., `v2.0.0`)
2. **Update Release Notes**: Add new entry to `RELEASE.md` with changes
3. **Merge to Main**: CI will automatically:
   - Build and publish Docker image to GAR OCI registry
   - Build and publish Python wheel to GAR Python registry
   - Push docs to production and development documentation sites

**Important**: Releases are documentation-driven. Not updating `RELEASE.md` will not trigger a release.
