# DV Flow Manager (dfm)

DV Flow Manager is a YAML-based build system and execution engine designed for silicon design and verification projects. It orchestrates tasks through declarative workflows with dataflow-based dependency management.

## Core Concepts

**Tasks**: Fundamental units of behavior that accept data from dependencies and produce outputs. Tasks can be:
- Built-in (std library: FileSet, Message, CreateFile, Exec, SetEnv, Prompt)
- Plugin-based (e.g., hdlsim for HDL simulators, pss for Portable Stimulus)
- Custom Python implementations
- Compound tasks containing subtasks

**Packages**: Parameterized namespaces that organize tasks, types, and configurations. Packages support:
- Inheritance and extension
- Fragment-based organization for large projects
- Plugin discovery via Python entry points
- Multiple configurations (debug, release, etc.)

**Dataflow**: Tasks communicate via typed data items rather than global variables. Each task:
- Receives data from dependencies specified via `needs`
- Produces outputs consumed by dependent tasks
- Can filter inputs with `consumes` patterns
- Controls output forwarding with `passthrough` patterns

**Types**: Custom data structures for task parameters, supporting inheritance and reuse.

**Expressions**: Dynamic parameter evaluation using `${{ }}` syntax for referencing parameters, calculations, and conditional logic.

## Flow Definition Format

Flows are defined in `flow.yaml` or `flow.dv` files:

```yaml
package:
  name: my_project
  
  with:
    debug:
      type: bool
      value: false
  
  tasks:
    - name: rtl_files
      uses: std.FileSet
      with:
        type: systemVerilogSource
        base: src/rtl
        include: "*.sv"
    
    - name: sim_image
      uses: hdlsim.vlt.SimImage
      needs: [rtl_files]
      with:
        top: [my_top]
    
    - name: sim_run
      uses: hdlsim.vlt.SimRun
      needs: [sim_image]
```

## Key Features

- **Incremental execution**: Tracks task dependencies and mementos to avoid redundant work
- **Parallel execution**: Concurrent task execution with configurable parallelism (-j flag)
- **Fileset tracking**: Maintains dependency trees for proper file ordering in HDL compilation
- **Conditional execution**: Tasks can run conditionally via `iff` expressions
- **Task override**: Replace task implementations for different configurations
- **Run directory management**: Isolated or shared workspaces via `rundir` modes
- **Multiple UI modes**: log, progress bar, or full TUI for monitoring execution
- **Execution tracing**: Google Event Trace format output for performance analysis

## Command Line Interface

```bash
# Run tasks (auto-selects from available tasks if none specified)
dfm run [tasks...] [-j N] [--clean] [-f] [-c config] [-D param=value]

# Show task information
dfm show [task] [-a] [-v]

# Generate dependency graph
dfm graph [task] [-o file] [-f dot]
```

## Common Use Cases

1. **HDL Compilation & Simulation**: Collect source files, compile with simulators (Verilator, Xcelium, VCS), run tests
2. **Verification Flows**: Build testbenches, run regression suites, collect coverage
3. **Synthesis Flows**: RTL to netlist transformation with tool-specific configurations
4. **Code Generation**: AI-assisted workflows using std.Prompt task with LLM integration
5. **Multi-tool Workflows**: Coordinate multiple EDA tools with proper dataflow

## Package Ecosystem

- **std**: Built-in standard library (always available)
- **hdlsim**: HDL simulator integration (dv-flow-libhdlsim package)
- **pss**: Portable Stimulus support (dv-flow-libpss package)
- Custom packages via Python plugins with entry points

## Advanced Features

- **Task strategies**: Matrix-based parallel task generation
- **Data item filtering**: Pattern-based consume/passthrough controls
- **Environment management**: std.SetEnv with glob expansion
- **Configuration inheritance**: Configs can extend base configurations
- **Package fragments**: Split large packages across multiple files
- **Task extensions**: Modify imported tasks without replacement

## Integration

DV Flow Manager integrates with:
- Version control systems (works alongside git/svn)
- CI/CD pipelines (log mode for non-interactive execution)
- EDA tools via plugin packages
- AI assistants (GitHub Copilot CLI, planned: OpenAI, Claude)

## File Structure

```
project/
├── flow.yaml           # Main package definition
├── rundir/             # Task execution workspace
│   ├── cache/          # Task mementos and artifacts
│   ├── log/            # Execution traces and logs
│   └── <task_dirs>/    # Individual task output directories
└── packages/           # Optional sub-packages
```

## Dependencies

Core dependencies: jq, pydantic, pyyaml, rich, tomli, toposort, svdep

Installation: `pip install dv-flow-mgr`

## Documentation Structure

- **docs/quickstart.rst**: Installation and first flow tutorial
- **docs/userguide/fundamentals.rst**: Core concepts (tasks, packages, dataflow)
- **docs/userguide/tasks_using.rst**: Task usage patterns and examples
- **docs/userguide/packages.rst**: Package organization and plugin system
- **docs/userguide/stdlib.rst**: Standard library task reference
- **docs/cmdref.rst**: Command line interface reference
- **docs/incremental.rst**: Incremental execution and caching
- **src/dv_flow/mgr/**: Python implementation
- **tests/**: Unit, system, and performance tests

## Key Design Principles

1. **Declarative over imperative**: Specify what to build, not how to build it
2. **Dataflow over shared state**: Tasks coordinate via typed data items
3. **Composition over monoliths**: Build complex tasks from simple ones
4. **Incremental by default**: Only rebuild what changed
5. **Tool-agnostic**: Works with any EDA tool via plugins
