Metadata-Version: 2.4
Name: telescope-cortex
Version: 0.0.1a1
Summary: Headless BI for AI Native applications
License: MIT
License-File: LICENSE
Author: Nilotpal Roychowdhury
Author-email: nilotpal@jointelescope.com
Requires-Python: >=3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Provides-Extra: api
Requires-Dist: alembic (>=1.16.5,<2.0.0)
Requires-Dist: duckdb (>=1.4.0,<2.0.0)
Requires-Dist: duckdb-engine (>=0.17.0,<0.18.0)
Requires-Dist: psycopg[binary,pool] (==3.2.9)
Requires-Dist: pydantic (>=2.10.6,<3.0.0)
Requires-Dist: pymysql[rsa] (>=1.1.2,<2.0.0)
Requires-Dist: pytz (>=2025.1,<2026.0)
Requires-Dist: sqlalchemy (>=2.0.38,<3.0.0)
Requires-Dist: sqlalchemy-bigquery (>=1.12.1,<2.0.0)
Description-Content-Type: text/markdown

# Cortex Headless BI

## Lightweight Modular Analytics Platform

A modular, lightweight analytics engine built in Python to power customer-facing analytics applications. The platform provides a unified semantic layer for defining business data models, a dynamic query engine that integrates with heterogeneous data sources, and a robust user management and authorization system—all accessible via a FastAPI-powered REST API. The semantic layer is designed to support advanced AI agent integration for intelligent analytics, natural language querying, and automated insights generation.

## Overview

This platform is designed to abstract complex data sources into a business-friendly semantic layer. It enables developers to define data models in JSON (with YAML support planned), dynamically generate queries across multiple data sources, and securely expose analytics functionality to both admin/developer users and end users.

### Key Features

- **Semantic Layer**
  - Define and manage data models in JSON with measures, dimensions, filters, and aggregations
  - Dynamic context aware schema generation
  - Advanced output formatting with IN_QUERY and POST_QUERY(Coming Soon) transformation modes
  - Versioning and audit trails for metrics and data models
  - Parameter system for dynamic query generation
  - Metric extension and inheritance support (Coming Soon)

- **Query Engine**
  - Translates semantic definitions into optimized queries
  - Real-time output formatting during query execution and post-processing
  - Multi-layer caching with Redis and in-memory backends
  - Planned support for multi-source queries (PostgreSQL, MySQL, BigQuery, SQLite)
  - Pre-aggregations and rollup tables for performance optimization
  - Query bindings for automatic rollup table utilization

- **Data Source Integration**
  - PostgreSQL, MySQL, BigQuery with persistent connectors
  - SQLite for in-memory analytics
  - Extensible factory pattern for adding new data sources
  - Schema introspection and humanized schema generation

- **Dashboard & Visualization Engine**
  - Multi-view dashboard system with executive, operational, and tactical types
  - Support for 10+ visualization types: single value, gauge, bar/line/area/pie/donut charts, scatter plots, heatmaps, and tables
  - Advanced chart features with ECharts integration
  - Field mapping and data transformation for visualizations
  - Widget-level metric execution with override support

- **Multi-Tenancy**
  - Hierarchical organization: Workspaces → Environments → Consumers
  - Consumer groups for organizing users with shared properties
  - Environment-level isolation for dev/staging/production
  - Context-aware query execution based on consumer properties

- **API-First Approach**
  - All functionality exposed via FastAPI-based REST endpoints
  - Auto-generated OpenAPI documentation with Scalar FastAPI
  - Comprehensive request/response validation with Pydantic

- **Query History & Monitoring**
  - Automatic logging of all query executions
  - Query performance analytics and slow query identification
  - Cache hit rate tracking and statistics
  - Execution history with filtering and search capabilities

- **Performance Optimization**
  - Pre-aggregation specs with rollup [IN PROGRESS]
  - Configurable TTL-based caching strategies

## Architecture

The project follows a layered architecture within a monorepo, ensuring modularity, ease of maintenance, and independent evolution of key components.

### Semantic Layer

This semantic layer is designed with AI agent integration in mind, providing:

- **Structured Semantic Models**: JSON-based metric definitions with measures, dimensions, joins, and aggregations
- **Advanced Output Formatting**: Support for data transformations at both query time (IN_QUERY) and post-execution (POST_QUERY)
- **Context-Aware Execution**: Consumer properties and environment isolation for personalized data access
- **Query Abstraction**: Database-agnostic query generation from semantic definitions
- **Execution Logging**: Comprehensive query execution logs for AI training and optimization
- **Parameter System**: Dynamic parameter substitution for flexible query generation
- **Validation Pipeline**: Automated validation and compilation of semantic models

This foundation will enable AI agents to:
- Translate natural language queries into semantic metric definitions
- Recommend relevant metrics and dimensions based on user context
- Optimize query performance through pattern analysis
- Generate automated insights and anomaly detection
- Learn from user behavior and query patterns for continuous improvement

### AI Agent Integration Points

1. **Natural Language Interface**: Convert user questions into `SemanticMetric` instances
2. **Intelligent Discovery**: Semantic search and recommendation across available metrics
3. **Automated Modeling**: AI-powered generation of data models from schema analysis
4. **Context Personalization**: Leverage consumer properties for role-based suggestions
5. **Performance Optimization**: Query pattern analysis and optimization recommendations
6. **Quality Monitoring**: Automated data quality assessment and anomaly detection

## Getting Started

### Prerequisites
- Python 3.12+
- PostgreSQL (or other supported database)

### Installation

#### Production Installation (Recommended)
```bash
# Install core package
pip install telescope-cortex

# Install with API extras
pip install telescope-cortex[api]

# Set up environment variables
export CORTEX_AUTO_APPLY_DB_MIGRATIONS=true
# Configure your database settings (see Environment Configuration below)

# Start the API server
python -m cortex.api
```

#### Local Development Installation
```bash
# Clone the repository
git clone https://github.com/TelescopeAI/cortex
cd cortex

# Install core dependencies only
poetry install --only main

# Install with all dependencies including FastAPI
poetry install --with api

# Set up environment variables
cp local.env .env
# Edit .env with your configuration

# Enable auto-migration for development
export CORTEX_AUTO_APPLY_DB_MIGRATIONS=true

# Start the API server
poetry run uvicorn cortex.api.main:app --reload --host 0.0.0.0 --port 9002
```

#### Database Migrations

The platform supports automatic database migrations on startup. Set the following environment variable to enable:

```bash
export CORTEX_AUTO_APPLY_DB_MIGRATIONS=true
```

For more information on running migrations manually or troubleshooting migration issues, see the [Migration Guide](migrations/MIGRATION_GUIDE.md).

#### Environment Configuration

Key environment variables:
```bash
# Database configuration
CORTEX_DB_TYPE=postgresql  # postgresql, sqlite
CORTEX_DB_HOST=localhost
CORTEX_DB_PORT=5432
CORTEX_DB_NAME=cortex
CORTEX_DB_USERNAME=root
CORTEX_DB_PASSWORD=password

# Cache configuration
CORTEX_CACHE_ENABLED=true
CORTEX_CACHE_BACKEND=redis  # redis or memory
CORTEX_CACHE_REDIS_URL=redis://localhost:6379

# Pre-aggregations
CORTEX_PREAGGREGATIONS_ENABLED=true

# API configuration
API_BASE_URL=http://localhost:9002
ALLOWED_ORIGINS=http://localhost:3000,http://localhost:3001
```

### Quick Start - Creating Your First Semantic Model

1. **Define a Data Source**:
```json
{
  "name": "sales_db",
  "source_type": "postgresql",
  "config": {
    "host": "localhost",
    "database": "sales",
    "username": "user",
    "password": "password"
  }
}
```

2. **Create a Semantic Metric with Output Formatting**:
```json
{
  "name": "monthly_revenue",
  "description": "Total revenue aggregated by month",
  "table_name": "sales",
  "measures": [
    {
      "name": "revenue",
      "type": "sum",
      "query": "amount",
      "formatting": [
        {
          "name": "currency_format",
          "type": "format",
          "mode": "post_query",
          "format_string": "${:,.2f}"
        }
      ]
    }
  ],
  "dimensions": [
    {
      "name": "month",
      "query": "sale_date",
      "formatting": [
        {
          "name": "date_format",
          "type": "cast",
          "mode": "in_query",
          "target_type": "date"
        }
      ]
    }
  ]
}
```

3. **Execute Queries**:
```python
from cortex.core.query.executor import QueryExecutor
from cortex.core.semantics.metrics.metric import SemanticMetric

executor = QueryExecutor()
result = executor.execute_metric(
    metric=your_metric,
    data_model=your_model,
    parameters={"start_date": "2024-01-01"}
)
```

## Studio

The platform includes a modern Vue.js frontend built with Nuxt 4 and TypeScript for creating and managing dashboards, metrics, and data visualizations.

### Features
- **Workspace & Environment Management**: Multi-tenant workspace management with environment isolation
- **Data Source Configuration**: Visual interface for connecting and configuring data sources
- **Data Model Builder**: Create and manage data models with schema introspection
- **Metric Builder**: Visual interface for creating semantic metrics with measures, dimensions, filters, and aggregations
- **Dashboard Builder**: Create multi-view dashboards with drag-and-drop widget placement
- **Visualization Editor**: Configure 10+ chart types with advanced field mapping
- **Consumer & Group Management**: Manage end users and consumer groups
- **Query History**: View and analyze query execution history and performance
- **Pre-aggregation Management**: Configure and monitor rollup tables
- **Real-time Preview**: Instant visualization of metric results during development

### Stack
- **Framework**: Nuxt 4
- **Language**: TypeScript with full type safety
- **Charts**: ECharts for advanced visualizations
- **UI**: Tailwind CSS with custom components
- **State Management**: Composables with reactive state

### Setup
```bash
cd frontend/cortex
yarn install

# Development mode
yarn run dev

# Production build
yarn run build
```

## API Reference

The platform provides a comprehensive REST API for all operations. Access the interactive API documentation at:
- **API Docs**: `http://localhost:9002/docs`
- **Classic UI**: `http://localhost:9002/docs/classic`
- **ReDoc UI**: `http://localhost:9002/redoc`

### Core API Endpoints

| Resource | Endpoint | Description |
|----------|----------|-------------|
| **Workspaces** | `/api/v1/workspaces` | Top-level organizational units |
| **Environments** | `/api/v1/environments` | Development stages within workspaces |
| **Data Sources** | `/api/v1/data/sources` | Database connection management |
| **Data Models** | `/api/v1/data/models` | Business data model definitions |
| **Metrics** | `/api/v1/metrics` | Semantic metric creation and execution |
| **Dashboards** | `/api/v1/dashboards` | Dashboard and widget management |
| **Consumers** | `/api/v1/consumers` | End user management |
| **Consumer Groups** | `/api/v1/consumers/groups` | User group management |
| **Query History** | `/api/v1/query/history` | Query execution logs and analytics |
| **Pre-aggregations** | `/api/v1/pre-aggregations` | Rollup table management |

### Example API Usage

```python
import httpx

# Create a workspace
response = httpx.post("http://localhost:9002/api/v1/workspaces", json={
    "name": "My Workspace",
    "description": "Production workspace"
})
workspace_id = response.json()["id"]

# Create an environment
response = httpx.post("http://localhost:9002/api/v1/environments", json={
    "workspace_id": workspace_id,
    "name": "Production",
    "description": "Production environment"
})
environment_id = response.json()["id"]

# Create a data source
response = httpx.post("http://localhost:9002/api/v1/data/sources", json={
    "environment_id": environment_id,
    "name": "Sales Database",
    "alias": "sales_db",
    "source_catalog": "database",
    "source_type": "postgresql",
    "config": {
        "host": "localhost",
        "port": 5432,
        "database": "sales",
        "username": "user",
        "password": "password"
    }
})

# Execute a metric
response = httpx.post(f"http://localhost:9002/api/v1/metrics/{metric_id}/execute", json={
    "parameters": {"start_date": "2024-01-01"},
    "cache": {"enabled": true, "ttl": 3600}
})
```

## Development

### Project Structure
```
cortex/
├── cortex/                   # Core Python package
│   ├── api/                  # FastAPI REST API (optional)
│   │   ├── routers/          # API endpoint routers
│   │   ├── schemas/          # Request/response schemas
│   │   └── main.py           # API application
│   ├── core/                 # Core semantic layer and query engine
│   │   ├── cache/            # Caching implementations
│   │   ├── connectors/       # Database connectors
│   │   ├── consumers/        # User management
│   │   ├── dashboards/       # Dashboard engine
│   │   ├── data/             # Data models and sources
│   │   ├── preaggregations/  # Pre-aggregation system
│   │   ├── query/            # Query engine
│   │   ├── semantics/        # Semantic layer
│   │   ├── storage/          # Database storage
│   │   └── workspaces/       # Multi-tenancy
├── frontend/cortex/          # Nuxt admin interface
│   ├── app/                  # Application code
│   │   ├── components/       # Vue components
│   │   ├── composables/      # Composable functions
│   │   ├── pages/            # Page components
│   │   └── types/            # TypeScript types
├── migrations/               # Alembic database migrations
└── pyproject.toml            # Poetry configuration
```

### Key Components

#### Backend
- **Semantic Layer**: Core data modeling with measures, dimensions, filters, and aggregations
- **Query Engine**: Database-agnostic SQL generation and execution
- **Cache Manager**: Multi-backend caching with Redis and in-memory support
- **Pre-aggregation Service**: Rollup table management and query optimization
- **Dashboard Engine**: Widget execution and visualization data processing
- **Consumer Management**: Multi-tenant user and group management
- **Query History**: Execution logging and performance analytics

#### Frontend
- **Workspace Management**: Multi-tenant workspace and environment configuration
- **Data Source Configurator**: Visual database connection setup
- **Metric Builder**: Drag-and-drop semantic metric creation
- **Dashboard Designer**: Multi-view dashboard builder with widget library
- **Query Explorer**: Query history and performance monitoring
- **User Management**: Consumer and group administration

### Database Migrations

The platform uses Alembic for database schema management:

```bash
# Create a new migration
alembic revision --autogenerate -m "description"

# Apply migrations
alembic upgrade head

# Rollback one migration
alembic downgrade -1

# View migration history
alembic history
```

Auto-apply migrations on startup (for development):
```bash
export CORTEX_AUTO_APPLY_DB_MIGRATIONS=true
```

## Technology Stack

### Backend
- **Language**: Python 3.12+
- **Web Framework**: FastAPI with Pydantic v2
- **Database ORM**: SQLAlchemy 2.0
- **Migrations**: Alembic
- **Databases**: PostgreSQL, MySQL, BigQuery, SQLite
- **Caching**: Redis, In-Memory
- **Package Manager**: Poetry

### Frontend
- **Framework**: Nuxt 4
- **Language**: Vue and TypeScript
- **UI Library**: Tailwind CSS
- **Charts**: Apache ECharts
- **Package Manager**: Yarn

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Contributions

Contributions are welcome! Please feel free to submit a [Pull Request](https://github.com/TelescopeAI/cortex/compare).

## Support

For questions and support:
- Open an issue on GitHub
- Email: support@jointelescope.com
- Documentation: [docs.jointelescope.com](https://docs.jointelescope.com)

## Roadmap

Upcoming features:
- [ ] File-based data sources (CSV, Excel, Google Sheets)
- [ ] Advanced AI agent integration
- [ ] Natural language query interface
- [ ] Automated metric recommendations
- [ ] Multi-database joins
- [ ] Real-time streaming analytics
- [ ] User authentication and authorization system
- [ ] Advanced access control and row-level security
- [ ] Embedded analytics SDK
- [ ] Mobile-responsive dashboard views
- [ ] MongoDB and DuckDB support


