Metadata-Version: 2.4
Name: graphnight
Version: 0.1.0
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Dist: pydantic>=2.6.0
Requires-Dist: httpx>=0.26.0
Requires-Dist: pandas>=2.2.0
Requires-Dist: pytest>=8.0.0 ; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0 ; extra == 'dev'
Requires-Dist: ruff>=0.3.0 ; extra == 'dev'
Requires-Dist: mypy>=1.8.0 ; extra == 'dev'
Provides-Extra: dev
Summary: High-performance semantic layer for AI agents and humans
Author: GraphNight Team
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# GraphNight Architecture Blueprint

## Overview

GraphNight is a high-performance, embeddable semantic layer for AI agents and humans, inspired by SLayer but built with Rust for performance and GraphQL for flexible querying. It enables governed, shared access to data and metrics across databases.

## Core Goals (from SLayer)

1. **Expressive semantic queries** - Define metrics once, query with flexible expressions (`revenue:sum`, `revenue:sum / *:count`, `time_shift(revenue:sum, -1, 'year')`)
2. **Agent-first search → inspect → query flow** - Search tool for discovery, memory store for business context
3. **Multi-interface** - GraphQL, REST, Python, MCP, Flight SQL, Postgres wire protocol
4. **Governance** - Row-level security, forced filters, audit trails
5. **Embeddable** - Standalone tool or Python/Rust library
6. **Data stack agnostic** - Importers for dbt, Cube, Ossie

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                        GraphNight Architecture                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │
│  │   GraphQL    │  │    REST      │  │   Python     │  ← Interfaces │
│  │   (async-    │  │   (axum/     │  │   (PyO3      │              │
│  │   graphql)   │  │   tide)      │  │   bindings)  │              │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘              │
│         │                 │                 │                        │
│         └─────────────────┼─────────────────┘                        │
│                           ▼                                           │
│              ┌────────────────────────┐                              │
│              │   Query Planner        │                              │
│              │   (Rust core)          │                              │
│              └───────────┬────────────┘                              │
│                          │                                            │
│         ┌───────────────┼───────────────┐                            │
│         ▼               ▼               ▼                            │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐                       │
│  │  Formula   │ │   Join     │ │  Security  │                       │
│  │  Engine    │ │  Walker    │ │  Layer     │                       │
│  └────────────┘ └────────────┘ └────────────┘                       │
│         │               │               │                            │
│         └───────────────┼───────────────┘                            │
│                         ▼                                             │
│              ┌────────────────────────┐                              │
│              │   SQL Generator        │  (DataFusion / custom)       │
│              └───────────┬────────────┘                              │
│                          │                                            │
│         ┌───────────────┼───────────────┐                            │
│         ▼               ▼               ▼                            │
│  ┌────────────┐ ┌────────────┐ ┌────────────┐                       │
│  │ PostgreSQL │ │   MySQL    │ │  SQLite/   │                       │
│  │  (sqlx)    │ │  (sqlx)    │ │  DuckDB    │                       │
│  └────────────┘ └────────────┘ └────────────┘                       │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    Storage Layer (YAML/JSON/SQLite)           │   │
│  │  Models │ Datasources │ Memories │ Search Index │ Config      │   │
│  └──────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘
```

## Rust Workspace Structure

```
rust-engine/
├── Cargo.toml                    # Workspace root
├── graphnight-core/              # Core domain models & logic
│   ├── src/
│   │   ├── models.rs             # Query, Measure, Dimension, Model, DataSource
│   │   ├── formula.rs            # Formula parser, registry, expressions
│   │   ├── query/
│   │   │   ├── planner.rs        # Logical query plan
│   │   │   ├── optimizer.rs      # Plan optimization
│   │   │   └── validator.rs      # Semantic validation
│   │   ├── join/
│   │   │   ├── walker.rs         # Join path finding
│   │   │   └── graph.rs          # Join graph representation
│   │   ├── security/
│   │   │   ├── policy.rs         # Session policies, forced filters
│   │   │   └── rls.rs            # Row-level security
│   │   └── errors.rs             # Error types
│   └── Cargo.toml
│
├── graphnight-sql/               # SQL generation & execution
│   ├── src/
│   │   ├── generator.rs          # SQL AST → dialect-specific SQL
│   │   ├── dialects/
│   │   │   ├── postgres.rs
│   │   │   ├── mysql.rs
│   │   │   ├── sqlite.rs
│   │   │   └── duckdb.rs
│   │   ├── executor.rs           # Async query execution (sqlx)
│   │   ├── datafusion.rs         # DataFusion integration for complex queries
│   │   └── cache.rs              # Query plan cache
│   └── Cargo.toml
│
├── graphnight-storage/           # Persistence layer
│   ├── src/
│   │   ├── backend.rs            # StorageBackend trait
│   │   ├── yaml.rs               # YAML file storage
│   │   ├── sqlite.rs             # SQLite storage (embedded)
│   │   ├── postgres.rs           # Postgres metadata storage
│   │   ├── search.rs             # Embedding-based search (tantivy)
│   │   └── migrations.rs         # Schema migrations
│   └── Cargo.toml
│
├── graphnight-graphql/           # GraphQL API
│   ├── src/
│   │   ├── schema.rs             # async-graphql schema definition
│   │   ├── resolvers/
│   │   │   ├── query.rs          # Query resolvers
│   │   │   ├── mutation.rs       # Mutation resolvers
│   │   │   └── subscription.rs   # Live query subscriptions
│   │   ├── directives/
│   │   │   ├── auth.rs           # @auth, @rbac
│   │   │   ├── cache.rs          # @cache
│   │   │   └── rate_limit.rs     # @rateLimit
│   │   └── context.rs            # Request context
│   └── Cargo.toml
│
├── graphnight-python/            # Python bindings (PyO3)
│   ├── src/
│   │   ├── lib.rs                # Module definition
│   │   ├── client.rs             # Python client class
│   │   ├── query.rs              # Query model bindings
│   │   ├── model.rs              # Model bindings
│   │   ├── datasource.rs         # Datasource bindings
│   │   ├── memory.rs             # Memory bindings
│   │   └── search.rs             # Search bindings
│   ├── pyproject.toml            # Python package config
│   └── Cargo.toml
│
├── graphnight-server/            # Standalone server binary
│   ├── src/
│   │   ├── main.rs               # Entry point
│   │   ├── config.rs             # Configuration
│   │   ├── graphql_server.rs     # Tide + async-graphql
│   │   ├── rest_server.rs        # Axum REST API
│   │   ├── mcp_server.rs         # MCP protocol server
│   │   └── flight_sql_server.rs  # Arrow Flight SQL
│   └── Cargo.toml
│
└── graphnight-cli/               # CLI tool
    ├── src/
    │   ├── main.rs
    │   ├── commands/
    │   │   ├── query.rs
    │   │   ├── model.rs
    │   │   ├── datasource.rs
    │   │   ├── memory.rs
    │   │   └── search.rs
    │   └── output.rs             # Table, JSON, CSV formatters
    └── Cargo.toml
```

## Data Models

### Query Model (Rust)

```rust
pub struct Query {
    pub name: Option<String>,                    // Run by name
    pub source_model: Option<SourceSpec>,        // For multi-stage
    pub measures: Vec<Measure>,                  // revenue:sum, time_shift(...)
    pub dimensions: Vec<Dimension>,              // Group by fields
    pub time_dimensions: Vec<TimeDimension>,     // DATE_TRUNC granularity
    pub filters: Vec<Filter>,                    // WHERE clauses
    pub order: Vec<OrderBy>,                     // ORDER BY
    pub limit: Option<usize>,
    pub offset: Option<usize>,
    pub whole_periods_only: Option<bool>,        // Complete time periods
    pub distinct_dimension_values: Option<bool>, // Distinct on dimensions
    pub stage_ref: Option<String>,               // Multi-stage reference
}
```

### Formula Expressions

| Expression | Meaning |
|------------|---------|
| `revenue:sum` | Sum of revenue |
| `revenue:avg` | Average revenue |
| `revenue:sum / *:count` | Revenue per row |
| `time_shift(revenue:sum, -1, 'month')` | Prior month revenue |
| `ratio(revenue:sum, cost:sum)` | Revenue/cost ratio |
| `pct_change(revenue:sum)` | Month-over-month % change |
| `running_total(revenue:sum)` | Cumulative sum |
| `cohort(users, 'month', revenue:sum)` | Cohort analysis |

### Model Definition (YAML)

```yaml
name: orders
datasource: postgres_primary
description: "Order facts table"
measures:
  - name: revenue
    expression: "amount_usd:sum"
    label: "Revenue (USD)"
    format: "currency_usd"
  - name: order_count
    expression: "*:count"
    label: "Orders"
  - name: aov
    expression: "ratio(revenue:sum, order_count:count)"
    label: "Average Order Value"
dimensions:
  - name: status
    label: "Order Status"
  - name: customer_id
    label: "Customer"
time_dimensions:
  - name: created_at
    label: "Order Date"
joins:
  - name: customers
    model: customers
    type: left
    on: [customer_id, id]
    alias: cust
```

## GraphQL Schema

```graphql
type Query {
  # Execute semantic query
  query(input: QueryInput!, dryRun: Boolean, explain: Boolean): QueryResponse!
  
  # Multi-stage DAG query
  multiStageQuery(inputs: [QueryInput!]!, dryRun: Boolean): MultiStageResponse!
  
  # Introspection
  models(datasource: String): [ModelInfo!]!
  model(name: String!, datasource: String): ModelInfo
  datasources: [DatasourceInfo!]!
  
  # Search & discovery
  search(q: String!, limit: Int): [SearchResult!]!
  inspect(model: String!, datasource: String): InspectResult!
  
  # Memories
  memories(filter: MemoryFilter): [Memory!]!
  memory(id: ID!): Memory
}

type Mutation {
  # Model management
  createModel(input: CreateModelInput!): ModelInfo!
  updateModel(name: String!, input: UpdateModelInput!): ModelInfo!
  deleteModel(name: String!, datasource: String): Boolean!
  
  # Datasource management
  createDatasource(input: CreateDatasourceInput!): DatasourceInfo!
  updateDatasource(name: String!, input: UpdateDatasourceInput!): DatasourceInfo!
  
  # Memories
  saveMemory(input: SaveMemoryInput!): Memory!
  forgetMemory(id: ID!): ForgetMemoryResponse!
  
  # Ingestion
  ingestModels(datasource: String!): IngestionReport!
}

type Subscription {
  # Live query results
  liveQuery(input: QueryInput!, intervalMs: Int!): QueryResponse!
  
  # Model changes
  modelChanges(datasource: String): ModelChangeEvent!
}

# Input types for flexible query construction
input QueryInput {
  name: String
  sourceModel: SourceSpecInput
  measures: [MeasureInput!]
  dimensions: [DimensionInput!]
  timeDimensions: [TimeDimensionInput!]
  filters: [FilterInput!]
  order: [OrderByInput!]
  limit: Int
  offset: Int
  wholePeriodsOnly: Boolean
  distinctDimensionValues: Boolean
}

input MeasureInput {
  formula: String!           # "revenue:sum" or "time_shift(revenue:sum, -1, 'month')"
  label: String
  format: String
  aggregation: AggregationType
}

# ... other input types

enum AggregationType {
  SUM
  AVG
  COUNT
  MIN
  MAX
  COUNT_DISTINCT
}

type QueryResponse {
  data: [JSON!]!
  columns: [String!]!
  sql: String
  attributes: ResponseAttributes
  population: Int
  populationInferred: Boolean
  executionTimeMs: Float!
}
```

## Query Execution Pipeline

```
Query Input (GraphQL/REST/Python)
         │
         ▼
┌────────────────────────┐
│  Semantic Validation   │  ← Check model exists, fields valid, types match
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│   Formula Expansion    │  ← Resolve shorthand, time_shift, ratio, etc.
│   (FormulaRegistry)    │
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│   Join Resolution      │  ← Find join paths, detect cycles, pick join types
│   (JoinWalker)         │
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│  Security Policy Check │  ← Apply forced filters, RLS, column masking
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│   Logical Plan         │  ← Project, Filter, Aggregate, Join, Sort, Limit
│   (DataFusion/Custom)  │
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│   Plan Optimization    │  ← Predicate pushdown, join reorder, partition pruning
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│   SQL Generation       │  ← Dialect-specific SQL (Postgres, MySQL, etc.)
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│   Async Execution      │  ← sqlx connection pool, streaming results
└───────────┬────────────┘
            │
            ▼
┌────────────────────────┐
│   Response Formatting  │  ← Arrow → JSON, metadata, population estimate
└────────────────────────┘
```

## Performance Optimizations

| Optimization | Implementation |
|--------------|----------------|
| **Query plan cache** | LRU cache keyed by normalized query hash |
| **Connection pooling** | sqlx pool per datasource, configurable size |
| **Streaming results** | Async streams, avoid materializing large results |
| **Predicate pushdown** | Push filters into SQL WHERE clauses |
| **Partition pruning** | Detect time filters, prune partitions |
| **Join optimization** | Reorder joins by cardinality estimates |
| **Vectorized execution** | DataFusion for complex analytics |
| **Prepared statements** | Cache parameterized queries |
| **Result caching** | Redis/In-memory with TTL, invalidation on model change |

## Security Model

```rust
pub struct SessionPolicy {
    pub forced_filters: Vec<Filter>,      // Always applied
    pub allowed_models: Option<Vec<String>>,  // Whitelist
    pub denied_models: Option<Vec<String>>,   // Blacklist
    pub column_masks: HashMap<String, MaskFn>, // PII masking
    pub row_filter: Option<Filter>,       // RLS predicate
    pub max_rows: Option<usize>,          // Result limit
    pub allowed_datasources: Option<Vec<String>>,
}

// Applied automatically in query planner
impl QueryPlanner {
    fn apply_policy(&self, query: Query, policy: &SessionPolicy) -> Result<Query> {
        // 1. Merge forced filters
        // 2. Validate model access
        // 3. Apply column masks to SELECT
        // 4. Add RLS to WHERE
        // 5. Enforce row limit
    }
}
```

## Multi-Stage Queries (DAG)

```yaml
# Example: Cohort analysis
queries:
  - name: cohort_sizes
    measures: ["users:count"]
    dimensions: ["cohort_month"]
    
  - name: cohort_revenue
    source_model: 
      model: cohort_sizes  # Reference previous stage
      alias: cohorts
    measures: ["revenue:sum"]
    dimensions: ["cohort_month", "months_since_first_order"]
```

Execution: Topological sort → Execute stages in parallel where possible → Stream results

## Python API (PyO3)

```python
# High-level client
from graphnight import GraphNightClient

client = GraphNightClient("http://localhost:8080")

# Simple query
result = client.query({
    "measures": ["revenue:sum", "orders:count"],
    "dimensions": ["store", "month"],
    "time_dimensions": [{"dimension": "created_at", "granularity": "month"}],
    "filters": [{"field": "status", "operator": "eq", "value": "completed"}],
    "order": [{"field": "revenue:sum", "descending": True}],
    "limit": 100
})

# DataFrame support
df = client.query_df(query)  # Returns pandas.DataFrame

# Model management
client.create_model({
    "name": "orders",
    "datasource": "postgres",
    "measures": [{"formula": "amount:sum", "label": "Revenue"}],
    "dimensions": [{"name": "status"}],
})

# Memories
client.save_memory(
    learning="Revenue spikes on Black Friday",
    linked_entities=["revenue:sum", "orders:count"]
)

# Search
results = client.search("monthly revenue by store")
```

## Configuration

```toml
# graphnight.toml
[server]
host = "0.0.0.0"
port = 8080
workers = 4

[storage]
type = "sqlite"  # yaml, sqlite, postgres
path = "./graphnight.db"

[datasources]
# Defined via API or CLI

[security]
enable_rls = true
default_max_rows = 10000
audit_log = true

[cache]
enabled = true
type = "redis"  # memory, redis
ttl_seconds = 300
max_size_mb = 512

[observability]
metrics_port = 9090
tracing_level = "info"
```

## Deployment Options

| Mode | Use Case |
|------|----------|
| **Embedded (Python)** | `import graphnight; engine = GraphNightEngine(storage=YAMLStorage())` |
| **Standalone Server** | `graphnight-server --config graphnight.toml` |
| **Docker** | `docker run -p 8080:8080 graphnight/server` |
| **Kubernetes** | Helm chart with horizontal scaling |
| **Serverless** | AWS Lambda / Cloudflare Workers (Rust → WASM) |

## Testing Strategy

```
tests/
├── unit/
│   ├── core/           # Formula parsing, validation
│   ├── sql/            # SQL generation per dialect
│   └── storage/        # Backend implementations
├── integration/
│   ├── postgres/       # Testcontainers
│   ├── mysql/
│   └── sqlite/
├── graphql/            # Schema validation, resolver tests
├── python/             # PyO3 binding tests
└── benchmarks/         # Criterion.rs benchmarks
```

## Roadmap

### Phase 1: Core Engine (Week 1-2)
- [ ] Rust workspace setup
- [ ] Core models (Query, Measure, Dimension, Model)
- [ ] Formula parser & registry
- [ ] SQL generator (PostgreSQL dialect)
- [ ] YAML storage backend
- [ ] Basic query execution

### Phase 2: GraphQL API (Week 2-3)
- [ ] async-graphql schema
- [ ] Query/Mutation resolvers
- [ ] Tide server integration
- [ ] Subscription support (WebSocket)

### Phase 3: Python Bindings (Week 3-4)
- [ ] PyO3 module
- [ ] Client class with async/sync API
- [ ] pandas DataFrame support
- [ ] Type stubs (.pyi)

### Phase 4: Advanced Features (Week 4-6)
- [ ] Join walker & graph
- [ ] Security policies & RLS
- [ ] Multi-stage DAG queries
- [ ] Search (tantivy embeddings)
- [ ] Memories
- [ ] MCP server

### Phase 5: Production Hardening (Week 6-8)
- [ ] Connection pooling
- [ ] Query plan cache
- [ ] Observability (metrics, tracing)
- [ ] Benchmarks & optimization
- [ ] Documentation
- [ ] CI/CD pipeline
