Metadata-Version: 2.4
Name: ascend-forge
Version: 0.1.2
Summary: NPU operator build & test framework
License: 
                                         Apache License
                                   Version 2.0, January 2004
                                http://www.apache.org/licenses/
        
           TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
        
           1. Definitions.
        
              "License" shall mean the terms and conditions for use, reproduction,
              and distribution as defined by Sections 1 through 9 of this document.
        
              "Licensor" shall mean the copyright owner or entity authorized by
              the copyright owner that is granting the License.
        
              "Legal Entity" shall mean the union of the acting entity and all
              other entities that control, are controlled by, or are under common
              control with that entity. For the purposes of this definition,
              "control" means (i) the power, direct or indirect, to cause the
              direction or management of such entity, whether by contract or
              otherwise, or (ii) ownership of fifty percent (50%) or more of the
              outstanding shares, or (iii) beneficial ownership of such entity.
        
              "You" (or "Your") shall mean an individual or Legal Entity
              exercising permissions granted by this License.
        
              "Source" form shall mean the preferred form for making modifications,
              including but not limited to software source code, documentation
              source, and configuration files.
        
              "Object" form shall mean any form resulting from mechanical
              transformation or translation of a Source form, including but
              not limited to compiled object code, generated documentation,
              and conversions to other media types.
        
              "Work" shall mean the work of authorship, whether in Source or
              Object form, made available under the License, as indicated by a
              copyright notice that is included in or attached to the work
              (an example is provided in the Appendix below).
        
              "Derivative Works" shall mean any work, whether in Source or Object
              form, that is based on (or derived from) the Work and for which the
              editorial revisions, annotations, elaborations, or other modifications
              represent, as a whole, an original work of authorship. For the purposes
              of this License, Derivative Works shall not include works that remain
              separable from, or merely link (or bind by name) to the interfaces of,
              the Work and Derivative Works thereof.
        
              "Contribution" shall mean any work of authorship, including
              the original version of the Work and any modifications or additions
              to that Work or Derivative Works thereof, that is intentionally
              submitted to the Licensor for inclusion in the Work by the copyright owner
              or by an individual or Legal Entity authorized to submit on behalf of
              the copyright owner. For the purposes of this definition, "submitted"
              means any form of electronic, verbal, or written communication sent
              to the Licensor or its representatives, including but not limited to
              communication on electronic mailing lists, source code control systems,
              and issue tracking systems that are managed by, or on behalf of, the
              Licensor for the purpose of discussing and improving the Work, but
              excluding communication that is conspicuously marked or otherwise
              designated in writing by the copyright owner as "Not a Contribution."
        
              "Contributor" shall mean Licensor and any individual or Legal Entity
              on behalf of whom a Contribution has been received by the Licensor and
              subsequently incorporated within the Work.
        
           2. Grant of Copyright License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              copyright license to reproduce, prepare Derivative Works of,
              publicly display, publicly perform, sublicense, and distribute the
              Work and such Derivative Works in Source or Object form.
        
           3. Grant of Patent License. Subject to the terms and conditions of
              this License, each Contributor hereby grants to You a perpetual,
              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
              (except as stated in this section) patent license to make, have made,
              use, offer to sell, sell, import, and otherwise transfer the Work,
              where such license applies only to those patent claims licensable
              by such Contributor that are necessarily infringed by their
              Contribution(s) alone or by combination of their Contribution(s)
              with the Work to which such Contribution(s) was submitted. If You
              institute patent litigation against any entity (including a
              cross-claim or counterclaim in a lawsuit) alleging that the Work
              or a Contribution incorporated within the Work constitutes direct
              or contributory patent infringement, then any patent licenses
              granted to You under this License for that Work shall terminate
              as of the date such litigation is filed.
        
           4. Redistribution. You may reproduce and distribute copies of the
              Work or Derivative Works thereof in any medium, with or without
              modifications, and in Source or Object form, provided that You
              meet the following conditions:
        
              (a) You must give any other recipients of the Work or
                  Derivative Works a copy of this License; and
        
              (b) You must cause any modified files to carry prominent notices
                  stating that You changed the files; and
        
              (c) You must retain, in the Source form of any Derivative Works
                  that You distribute, all copyright, patent, trademark, and
                  attribution notices from the Source form of the Work,
                  excluding those notices that do not pertain to any part of
                  the Derivative Works; and
        
              (d) If the Work includes a "NOTICE" text file as part of its
                  distribution, then any Derivative Works that You distribute must
                  include a readable copy of the attribution notices contained
                  within such NOTICE file, excluding any notices that do not
                  pertain to any part of the Derivative Works, in at least one
                  of the following places: within a NOTICE text file distributed
                  as part of the Derivative Works; within the Source form or
                  documentation, if provided along with the Derivative Works; or,
                  within a display generated by the Derivative Works, if and
                  wherever such third-party notices normally appear. The contents
                  of the NOTICE file are for informational purposes only and
                  do not modify the License. You may add Your own attribution
                  notices within Derivative Works that You distribute, alongside
                  or as an addendum to the NOTICE text from the Work, provided
                  that such additional attribution notices cannot be construed
                  as modifying the License.
        
              You may add Your own copyright statement to Your modifications and
              may provide additional or different license terms and conditions
              for use, reproduction, or distribution of Your modifications, or
              for any such Derivative Works as a whole, provided Your use,
              reproduction, and distribution of the Work otherwise complies with
              the conditions stated in this License.
        
           5. Submission of Contributions. Unless You explicitly state otherwise,
              any Contribution intentionally submitted for inclusion in the Work
              by You to the Licensor shall be under the terms and conditions of
              this License, without any additional terms or conditions.
              Notwithstanding the above, nothing herein shall supersede or modify
              the terms of any separate license agreement you may have executed
              with Licensor regarding such Contributions.
        
           6. Trademarks. This License does not grant permission to use the trade
              names, trademarks, service marks, or product names of the Licensor,
              except as required for reasonable and customary use in describing the
              origin of the Work and reproducing the content of the NOTICE file.
        
           7. Disclaimer of Warranty. Unless required by applicable law or
              agreed to in writing, Licensor provides the Work (and each
              Contributor provides its Contributions) on an "AS IS" BASIS,
              WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
              implied, including, without limitation, any warranties or conditions
              of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
              PARTICULAR PURPOSE. You are solely responsible for determining the
              appropriateness of using or redistributing the Work and assume any
              risks associated with Your exercise of permissions under this License.
        
           8. Limitation of Liability. In no event and under no legal theory,
              whether in tort (including negligence), contract, or otherwise,
              unless required by applicable law (such as deliberate and grossly
              negligent acts) or agreed to in writing, shall any Contributor be
              liable to You for damages, including any direct, indirect, special,
              incidental, or consequential damages of any character arising as a
              result of this License or out of the use or inability to use the
              Work (including but not limited to damages for loss of goodwill,
              work stoppage, computer failure or malfunction, or any and all
              other commercial damages or losses), even if such Contributor
              has been advised of the possibility of such damages.
        
           9. Accepting Warranty or Additional Liability. While redistributing
              the Work or Derivative Works thereof, You may choose to offer,
              and charge a fee for, acceptance of support, warranty, indemnity,
              or other liability obligations and/or rights consistent with this
              License. However, in accepting such obligations, You may act only
              on Your own behalf and on Your sole responsibility, not on behalf
              of any other Contributor, and only if You agree to indemnify,
              defend, and hold each Contributor harmless for any liability
              incurred by, or claims asserted against, such Contributor by reason
              of your accepting any such warranty or additional liability.
        
           END OF TERMS AND CONDITIONS
        
           Copyright 2026 xiaoleixin
        
           Licensed under the Apache License, Version 2.0 (the "License");
           you may not use this file except in compliance with the License.
           You may obtain a copy of the License at
        
               http://www.apache.org/licenses/LICENSE-2.0
        
           Unless required by applicable law or agreed to in writing, software
           distributed under the License is distributed on an "AS IS" BASIS,
           WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
           See the License for the specific language governing permissions and
           limitations under the License.
        
Project-URL: Homepage, https://gitcode.com/xiaoleixin/forge
Project-URL: Repository, https://gitcode.com/xiaoleixin/forge
Project-URL: Issues, https://gitcode.com/xiaoleixin/forge/issues
Keywords: ascend,npu,operator,benchmark,testing,bisect
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: httpx>=0.25; extra == "dev"
Requires-Dist: httpx-ws>=0.5; extra == "dev"
Provides-Extra: web
Requires-Dist: fastapi>=0.110; extra == "web"
Requires-Dist: uvicorn[standard]>=0.27; extra == "web"
Requires-Dist: websockets>=12.0; extra == "web"
Requires-Dist: pydantic>=2.5; extra == "web"
Dynamic: license-file

# forge

NPU 算子编译测试框架。通过 JSON 配置文件驱动，统一管理不同代码仓的算子编译和测试流程，支持精度测试、性能测试和 git bisect 问题定位。

新增算子仓只需添加一个 JSON 配置文件，无需修改任何代码。

**零第三方运行依赖**，仅使用 Python 标准库。

## 安装

### 系统依赖

- Python >= 3.9
- Git（bisect 功能需要）

### Python 依赖

运行时无需任何第三方库，仅使用 Python 标准库（argparse、json、csv、subprocess 等）。

开发依赖：[pytest](https://docs.pytest.org/) >= 7.0（仅运行测试需要）

### 安装步骤

**方式一：venv 虚拟环境**

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

**方式二：conda 环境**

```bash
# 在已有 conda 环境中安装
conda activate your_env
pip install -e ".[dev]"

# 或者新建专用环境
conda create -n forge python=3.9 -y
conda activate forge
pip install -e ".[dev]"
```

**仅安装（不含 pytest）：**

```bash
pip install -e .
```

## Web UI

使用浏览器界面（配置编辑、任务运行、bisect 可视化）：

```bash
pip install 'ascend-forge[web]'   # 方括号引号不要省，装上 web 额外依赖
forge web                          # http://127.0.0.1:8765
```

只用 CLI（不需要浏览器界面）：

```bash
pip install ascend-forge           # 不含 web 依赖
```

详见 `forge_web/README.md`。

## 架构

```
用户 CLI 命令
    │
    ▼
cli.py ─── 解析参数，分派命令（argparse）
    │
    ▼
config.py ─── 加载 JSON 配置，校验字段，替换占位符
    │
    ├──▶ task.py ─── 通用任务执行引擎（TaskRunner 基类）
    │        ├─ 单步/多步执行、skip 检查、capture 变量传递
    │        ├─ 日志记录、超时控制、ignore_failure
    │        └─ 子类通过 _post_step/_make_result 扩展
    │
    ├──▶ builder.py ─── 编译任务（BuildRunner → TaskRunner）
    │        ├─ _post_step: 产物检查（存在性、非空、时效性）
    │        └─ 支持 checkout 到指定 commit 编译（自动 stash/恢复）
    │
    ├──▶ tester.py ─── 测试任务（TestRunner → TaskRunner）
    │        ├─ _post_step: 结果提取（stdout 正则 / CSV 文件）
    │        └─ source_env 环境注入、repeat 重复聚合
    │
    ├──▶ result.py ─── 结果持久化（~/.forge/results/）与对比
    │
    ├──▶ workflow.py ─── 工作流引擎（WorkflowRunner）
    │        └─ 按顺序执行命名任务序列，共享变量传递
    │
    └──▶ bisector.py ─── 二分查找问题 commit
             └─ 循环：checkout → workflow → 判断 good/bad
```

### 项目结构

```
forge/
├── forge/
│   ├── cli.py          # argparse CLI 入口
│   ├── config.py       # 配置加载、校验、占位符替换
│   ├── executor.py     # 通用子进程执行器（超时、输出捕获、实时流式输出）
│   ├── task.py         # 通用任务执行引擎（TaskRunner 基类 + 共享工具函数）
│   ├── builder.py      # 编译任务（BuildRunner 子类 + git checkout/restore）
│   ├── tester.py       # 测试任务（TestRunner 子类 + 结果提取 + repeat 聚合）
│   ├── result.py       # 结果持久化与对比
│   ├── workflow.py     # 工作流引擎（WorkflowRunner：命名任务序列 + 变量传递）
│   ├── bisector.py     # Git bisect 编排（使用 WorkflowRunner 执行每步）
│   ├── log.py          # 日志系统（文件持久化 + 屏幕输出）
│   └── formatter.py    # ASCII 表格格式化（bisect 结果输出）
├── configs/            # 算子仓配置文件（JSON）
├── tests/              # 测试用例（259 个，含端到端测试）
└── pyproject.toml
```

### 核心模块职责

| 模块 | 职责 |
|------|------|
| `config.py` | 加载 JSON、校验必填字段、`{placeholder}` 替换、路径解析 |
| `executor.py` | subprocess 封装：超时控制、环境变量注入、输出捕获、实时流式输出 |
| `task.py` | 通用任务执行引擎（`TaskRunner` 基类），封装单步/多步执行、skip、capture、日志等公共流程 |
| `builder.py` | `BuildRunner` 子类：产物检查（`_post_step`）+ git checkout/restore |
| `tester.py` | `TestRunner` 子类：结果提取（`_post_step`）+ source_env + repeat 聚合 |
| `result.py` | 测试结果 JSON 持久化到 `~/.forge/results/`，支持按 commit 加载和结果对比 |
| `workflow.py` | 工作流引擎（`WorkflowRunner`），按顺序执行命名任务序列，步骤间通过 `capture` 共享变量 |
| `bisector.py` | 在 good/bad commit 之间二分，使用 `WorkflowRunner` 执行每步的 workflow，支持 skip 和分支恢复 |
| `log.py` | 日志文件持久化到 `~/.forge/logs/`，控制屏幕输出级别 |

## 配置文件

每个算子仓对应一个 JSON 文件，放在 `configs/` 目录下（或通过 `--config-dir` / `FORGE_CONFIG_DIR` 指定其他目录）。

真实仓配置推荐使用命名规范 `repo-function-scenario.json`，让文件名直接表达仓库归属、用途和默认场景，例如：

- `omni-accuracy-fuzz.json`
- `server-deploy-mixed.json`
- `transformer-bisect-default.json`

示例或演示配置可以保留通用前缀，但仍建议通过后缀表达主要用途和场景，例如 `example-build-base.json`、`demo-env-source.json`。

配置支持两种格式：

### 新格式（推荐）

使用 `tasks` + `workflows` + `bisect.workflow` 定义所有任务和执行流程：

```json
{
  "name": "my_repo",
  "description": "Ascend 910 算子库",
  "working_dir": "/home/user/repos/my_repo",

  "env": {
    "http_proxy": "http://proxy.example.com:8080"
  },

  "tasks": {
    "build": {
      "steps": [
        {
          "name": "compile",
          "command": "bash build.sh",
          "args": ["--ops={op_name}", "--target={target}"],
          "working_dir": ".",
          "env": { "ASCEND_HOME": "/usr/local/Ascend" },
          "timeout": 600,
          "success_pattern": "Build succeeded",
          "artifacts": ["output/libkernel.so", "output/op_*.bin"]
        },
        {
          "name": "install",
          "command": "bash install.sh",
          "working_dir": ".",
          "timeout": 120,
          "success_pattern": "Install succeeded"
        }
      ]
    },
    "test_accuracy": {
      "source_env": "{install_path}/set_env.sh",
      "steps": [
        {
          "name": "generate input",
          "command": "python gen_input.py",
          "args": ["--op", "{op_name}"],
          "working_dir": "tests/"
        },
        {
          "name": "run operator",
          "command": "bash run_op.sh",
          "args": ["{op_name}"],
          "working_dir": "."
        },
        {
          "name": "compare result",
          "command": "python compare.py",
          "args": ["--op", "{op_name}"],
          "working_dir": "tests/",
          "success_pattern": "PASS",
          "result": {
            "type": "stdout",
            "pattern": "accuracy: ([0-9.]+)"
          }
        }
      ],
      "timeout": 300
    },
    "test_performance": {
      "command": "bash run_perf.sh",
      "args": ["--op", "{op_name}", "--iterations", "100"],
      "working_dir": "tests/",
      "timeout": 0,
      "result": {
        "type": "file",
        "path": "output/perf_result.csv",
        "format": "csv",
        "value_column": 3,
        "op_column": 0,
        "header": true,
        "delimiter": ",",
        "aggregate": "avg"
      }
    }
  },

  "workflows": {
    "accuracy_pipeline": {
      "steps": ["build", "test_accuracy"]
    },
    "perf_pipeline": {
      "steps": ["build", "test_performance"]
    }
  },

  "bisect": {
    "test_type": "accuracy",
    "workflows": ["accuracy_pipeline"],
    "regression": 15.0,
    "threshold": null
  }
}
```

### 旧格式（向后兼容）

使用 `build` + `tests` + `bisect.build_before_test` 的旧格式仍然可用，框架会自动转换并显示迁移提示：

```json
{
  "name": "my_repo",
  "description": "Ascend 910 算子库",
  "working_dir": "/home/user/repos/my_repo",

  "env": {
    "http_proxy": "http://proxy.example.com:8080"
  },

  "build": {
    "command": "bash build.sh",
    "args": ["--target=ascend910", "--build-type=Release"],
    "working_dir": ".",
    "env": {
      "ASCEND_HOME": "/usr/local/Ascend"
    },
    "timeout": 600,
    "success_pattern": "Build succeeded",
    "artifacts": ["output/libkernel.so", "output/op_*.bin"]
  },

  "tests": {
    "accuracy": {
      "command": "python run_accuracy.py",
      "args": ["--op", "{op_name}"],
      "working_dir": "tests/",
      "timeout": 300,
      "success_pattern": "PASS",
      "result": {
        "type": "stdout",
        "pattern": "accuracy: ([0-9.]+)"
      }
    },
    "performance": {
      "command": "bash run_perf.sh",
      "args": ["--op", "{op_name}"],
      "working_dir": "tests/",
      "timeout": 0,
      "result": {
        "type": "file",
        "path": "output/perf_result.csv",
        "format": "csv",
        "value_column": 3,
        "op_column": 0,
        "header": true,
        "delimiter": ","
      }
    }
  },

  "bisect": {
    "build_before_test": true,
    "regression": 15.0,
    "threshold": null
  }
}
```

**自动转换规则：**
- `build` 配置 → `tasks.build`
- `tests.accuracy` → `tasks.test_accuracy`，`tests.performance` → `tasks.test_performance`
- 列表格式 workflow `["a", "b"]` → 对象格式 `{"steps": ["a", "b"]}`
- `bisect.build_before_test: true` → 自动生成 `_bisect_auto` workflow `["build", "test_{type}"]`

建议逐步迁移到新格式。

### 字段说明

**顶层字段：**

| 字段 | 必填 | 说明 |
|------|------|------|
| `name` | 是 | 仓库唯一标识 |
| `description` | 否 | 简短概述，适合 `forge list` 这类列表视图展示 |
| `usage` | 否 | 详细使用说明，适合 `forge show` / `forge usage` 展示 |
| `working_dir` | 是 | 工作目录路径（绝对路径，或相对于配置文件所在目录） |
| `vars` | 否 | 自定义变量字典，可在所有占位符中使用。CLI `--arg` 可覆盖 |
| `env` | 否 | 全局环境变量字典，所有任务继承。task/step 级别的 `env` 可覆盖 |
| `source_env` | 否 | 环境变量脚本路径（绝对路径，支持 `{placeholder}`）。任务执行前 source 该脚本注入环境变量 |

`description` 只负责提供一句话的短说明，供 `forge list` 快速浏览；`usage` 则用于承载更完整的使用指导、参数约定和示例命令。建议在 `usage` 中采用轻量级小节标题：`使用场景:`、`使用方法:`、`示例:`，方便 `forge show` 和 `forge usage` 直接输出。

**tasks 字段：**

| 字段 | 必填 | 说明 |
|------|------|------|
| `tasks.<name>.command` | 是* | 任务命令（单步模式） |
| `tasks.<name>.args` | 否 | 命令参数列表 |
| `tasks.<name>.working_dir` | 否 | 工作目录，相对于 working_dir，默认 `"."` |
| `tasks.<name>.env` | 否 | 额外环境变量（覆盖顶层 `env`） |
| `tasks.<name>.timeout` | 否 | 超时秒数，默认 600。设为 `0` 或 `null` 不限时 |
| `tasks.<name>.source_env` | 否 | 该任务专用的 source_env，覆盖上层配置 |
| `tasks.<name>.success_pattern` | 否 | 正则匹配 stdout 判断成功，支持字符串或数组（AND）。不配置则仅依赖退出码 |
| `tasks.<name>.artifacts` | 否 | 产物路径列表，支持 glob 通配符。检查文件存在、非空、且修改时间在执行之后 |
| `tasks.<name>.result.type` | 否 | `"stdout"` 或 `"file"` |
| `tasks.<name>.result.pattern` | stdout 时必填 | 正则表达式，第一个捕获组为数值 |
| `tasks.<name>.result.path` | file 时必填 | CSV 文件路径，相对于 working_dir |
| `tasks.<name>.result.value_column` | file 时必填 | 数值所在列（从 0 开始） |
| `tasks.<name>.result.op_column` | 否 | 算子名所在列 |
| `tasks.<name>.result.header` | 否 | 是否有表头行，默认 true |
| `tasks.<name>.result.delimiter` | 否 | CSV 分隔符，默认 `","` |
| `tasks.<name>.result.aggregate` | 否 | 多行聚合方式：`avg`、`last`、`min`、`max`，不配置则取最后一行 |
| `tasks.<name>.result.op_filter` | 否 | CSV 算子名匹配模板，支持 `{placeholder}` 和通配符 |
| `tasks.<name>.repeat` | 否 | 重复次数（CLI `--repeat` 优先） |
| `tasks.<name>.repeat_agg` | 否 | 重复聚合方式：`avg`（默认）、`avg_without_outliers` 或 `max`（CLI `--repeat-agg` 优先） |
| `tasks.<name>.outlier_detection.method` | 否 | 异常值检测方法，当前支持 `iqr` |
| `tasks.<name>.outlier_detection.max_outlier_ratio` | 否 | 异常值比例上限；超过时任务失败 |
| `tasks.<name>.outlier_detection.iqr_multiplier` | 否 | IQR 判定倍数，结合四分位距计算异常值边界 |
| `tasks.<name>.outlier_detection.min_samples` | 否 | 触发异常值分析所需的最小样本数 |
| `tasks.<name>.steps` | 否 | 多步骤配置（见"多步骤任务流程"） |

行为说明：当异常值比例大于阈值时，任务失败并输出样本、边界、异常值和比例；当比例不超过阈值时，先移除异常值再求平均；当样本数小于 `min_samples` 时，跳过异常值分析并直接使用普通平均值。

*使用 `steps` 时不需要顶层 `command`

**workflows 字段：**

| 字段 | 必填 | 说明 |
|------|------|------|
| `workflows.<name>.tasks` | 是 | 任务名称列表，按顺序执行 |

**bisect 字段：**

| 字段 | 必填 | 说明 |
|------|------|------|
| `bisect.workflows` | 是 | bisect 使用的工作流数组。支持字符串简写或完整对象格式 |
| `bisect.test_type` | 否 | 测试类型（`accuracy` 或 `performance`），默认 `accuracy`。CLI `--type` 已废弃 |
| `bisect.post_workflows` | 否 | bisect 结束后执行的工作流数组，默认不执行 |
| `bisect.git_repo` | 否 | git 操作目录，默认等于 working_dir |
| `bisect.regression` | 否 | 性能退化阈值百分比，默认 10。CLI `--regression` 优先 |
| `bisect.threshold` | 否 | 绝对性能阈值，不配置则使用百分比模式。CLI `--threshold` 优先 |
| `bisect.repeat` | 否 | 每步重复次数，默认 1。CLI `--repeat` 优先 |
| `bisect.repeat_agg` | 否 | 重复聚合方式：`avg`（默认）、`avg_without_outliers` 或 `max` |

**bisect.workflows 数组格式：**

每个工作流条目可以是：
- **字符串简写**：仅工作流名称，默认 `repeatable: true`
- **完整对象**：包含 `name` 和 `repeatable` 字段

```json
{
  "bisect": {
    "workflows": [
      "build_pipeline",           // 简写格式，repeatable 默认 true
      {"name": "test_pipeline", "repeatable": false}  // 完整格式
    ]
  }
}
```

| 字段 | 必填 | 说明 |
|------|------|------|
| `name` | 是 | 工作流名称，必须在 `workflows` 中定义 |
| `repeatable` | 否 | 是否应用 `--repeat` 参数，默认 `true`。设为 `false` 时强制单次执行 |

**bisect.post_workflows 数组格式：**

bisect 完成并恢复原始分支后执行的工作流。支持字符串或对象：

```json
{
  "bisect": {
    "workflows": ["build_test"],
    "post_workflows": ["cleanup", "report"]
  }
}
```

执行顺序：所有 bisect 工作流 → 恢复原始分支 → 所有 post_workflows。

**向后兼容：**

旧格式自动转换：
- `bisect.workflow: "name"` → `bisect.workflows: ["name"]`（显示废弃警告）
- `bisect.build_before_test: true` → 自动生成 `_bisect_auto` 工作流（显示废弃警告）
- `bisect.post_workflow: true` → 执行所有 bisect 工作流
- `bisect.post_workflow: "name"` → `bisect.post_workflows: ["name"]` |

### 全局环境变量（env）

顶层 `env` 字段定义全局环境变量，所有任务自动继承：

```json
{
  "env": {
    "http_proxy": "http://proxy.example.com:8080",
    "https_proxy": "http://proxy.example.com:8080"
  },
  "tasks": {
    "build": {"command": "bash build.sh"},
    "deploy": {
      "command": "bash deploy.sh",
      "env": {"DEBUG": "1"}
    }
  }
}
```

**优先级（高到低）：** step `env` > task `env` > 顶层 `env` > `source_env`

### 环境变量注入（source_env）

安装算子后通常需要 source 一个脚本设置环境变量，供任务使用。可在顶层或任务级别配置：

```json
{
  "name": "my_repo",
  "working_dir": "/path/to/repo",
  "tasks": {
    "accuracy": {
      "source_env": "{install_path}/{commit}/set_env.bash",
      "steps": [...]
    }
  }
}
```

**流程：** 执行任务前，框架先 `source <script>`，捕获新增/修改的环境变量，注入到命令环境中。

- **必须是绝对路径**（替换占位符后），相对路径会报错
- 支持 `{placeholder}` 替换（如 `{install_path}`、`{commit}`）
- 任务级别 `source_env` 优先于顶层 `source_env`
- 环境变量对所有步骤生效
- 步骤级别的 `env` 配置优先于 `source_env`

### 占位符

配置中的 `command`、`args`、`result.path`、`artifacts` 支持占位符，运行时自动替换。

**内置占位符：**

| 占位符 | 来源 | 说明 |
|--------|------|------|
| `{working_dir}` | 配置文件 | 工作目录绝对路径 |
| `{commit}` | `--commit` 或自动检测 | 当前 git commit hash（未指定 `--commit` 时自动获取 HEAD） |
| `{timestamp}` | 自动生成 | 当前时间戳，格式 `20260327_103000`，用于区分同名文件 |

**自定义占位符：**

通过配置文件 `vars` 字段预定义常用变量，避免每次 CLI 传入：

```json
{
  "name": "my_repo",
  "working_dir": "/path/to/repo",
  "vars": {
    "install_path": "/usr/local/Ascend",
    "target": "ascend910"
  },
  "tasks": {
    "build": {
      "command": "bash build.sh",
      "args": ["--target={target}"]
    }
  }
}
```

也可通过 `--arg KEY=VALUE` 或 `--arg @file.json` 在 CLI 传入或覆盖配置中的变量：

```bash
# 传入自定义变量
forge build my_repo --arg op_name=matmul --arg target=ascend910

# 从 JSON 文件加载变量
forge build my_repo --arg @vars.json

# 混合 JSON 文件和命令行覆盖
forge build my_repo --arg @common.json --arg target=ascend310

# 快捷方式
# --op  → --arg op_name=VALUE
# --cs  → --arg case=VALUE
# --ip  → --arg install_path=VALUE
forge build my_repo --op matmul
forge test my_repo -t acc --cs case1
forge build my_repo --ip /usr/local/Ascend

# CLI 覆盖配置中的 vars（target 从 ascend910 变为 ascend310）
forge build my_repo --arg target=ascend310
```

`@file.json` 只支持扁平 JSON 对象，所有值都必须是字符串；多个 `--arg` 会按命令行顺序合并，后面的值覆盖前面的值。

最小 JSON 对象示例：

```json
{
  "op_name": "matmul",
  "target": "ascend910"
}
```

**优先级（高到低）：** CLI `--arg` > `capture`（运行时捕获） > 配置 `vars` > 内置变量

**变量预检查：** 执行前自动扫描配置中所有占位符，缺失则报错。通过检查后打印变量表：

```
Variables:
  op_name = matmul
  commit = abc1234567890
```

配置文件中使用：
```json
"tasks": {
  "build": {
    "command": "bash build.sh",
    "args": ["--op", "{op_name}", "--target", "{target}"]
  }
}
```

所有任务命令均支持 `--arg` 和 `--op`。若配置中使用了占位符但未通过 CLI 传入，会报错提示。

### 多步骤任务流程

任务可以由多个步骤组成（如编译 → 安装），使用 `steps` 列表：

```json
"tasks": {
  "build": {
    "steps": [
      {
        "name": "compile",
        "command": "bash build.sh",
        "args": ["--ops={op_name}"],
        "working_dir": ".",
        "timeout": 600,
        "success_pattern": "Build succeeded",
        "artifacts": ["output/libkernel.so"]
      },
      {
        "name": "install",
        "command": "bash install.sh",
        "working_dir": ".",
        "timeout": 120,
        "success_pattern": "Install succeeded"
      }
    ]
  }
}
```

**规则：**
- 步骤按顺序执行，任一步骤失败则整体失败，后续步骤不执行
- 每个步骤可独立设置 `command`、`args`、`working_dir`、`timeout`、`success_pattern`、`artifacts`、`capture`
- `skip_if_exists`：指定产物路径（字符串或数组），支持 glob 和 `{placeholder}`，所有文件存在且非空则跳过
- `skip_if_artifacts_exist: true`：复用该步骤的 `artifacts` 列表作为跳过检查条件
- `artifacts` 检查在对应步骤执行后进行
- **向后兼容**：不使用 `steps` 时，原有单 `command` 写法不受影响

### 步骤跳过示例

```json
{
  "name": "compile",
  "command": "bash build.sh",
  "artifacts": ["output/libkernel.so", "output/op_*.bin"],
  "skip_if_artifacts_exist": true
}
```

等价于：

```json
{
  "name": "compile",
  "command": "bash build.sh",
  "skip_if_exists": ["output/libkernel.so", "output/op_*.bin"]
}
```

产物已存在且非空时输出：`Skipped (artifacts exist)`

### 步骤间变量传递（capture）

步骤可以从 stdout 中提取值，存为变量供后续步骤使用：

```json
"steps": [
  {
    "name": "get version",
    "command": "cat version.txt",
    "capture": {
      "version": "v([0-9.]+)"
    }
  },
  {
    "name": "install",
    "command": "bash install.sh",
    "args": ["--version", "{version}", "--path", "{install_path}"]
  }
]
```

**规则：**
- `capture` 是一个 dict，key 为变量名，value 为正则表达式（第一个捕获组）
- 从步骤的 stdout 中匹配，成功后注入到 variables，后续步骤可用 `{变量名}` 引用
- 匹配失败会打印 warning，不中断执行
- 变量预检查时会识别 `capture` 定义的变量，不会误报缺失

输出示例：
```
  Step 1/2: get version
    Captured: version = 2.5.1
  Step 2/2: install
    Command: bash install.sh --version 2.5.1 --path /usr/local
```

### 成功判断逻辑

1. 检查进程退出码，非 0 即失败
2. 若配置了 `success_pattern`，退出码为 0 时额外匹配 stdout，不匹配也判定失败
   - 字符串：单个正则匹配
   - 数组：所有正则都需匹配（AND 逻辑），如 `["Build succeeded", "Install succeeded"]`
3. 若配置了 `artifacts`，检查每个产物文件存在、非空、且修改时间在本次编译之后
4. 以上均为可选，不配置则仅依赖退出码

### 结果提取

精度测试和性能测试统一使用 `result` 配置块，支持两种方式：

**从标准输出提取：**
```json
{
  "result": {
    "type": "stdout",
    "pattern": "accuracy: ([0-9.]+)"
  }
}
```

**从 CSV 文件提取：**
```json
{
  "result": {
    "type": "file",
    "path": "output/result.csv",
    "format": "csv",
    "value_column": 3,
    "op_column": 0,
    "header": true,
    "delimiter": ",",
    "aggregate": "avg"
  }
}
```

**聚合方式（`aggregate` 字段）：** 同一算子有多行数据时的聚合策略：

| 值 | 说明 |
|----|------|
| `avg` | 平均值 |
| `last` | 最后一行 |
| `min` | 最小值 |
| `max` | 最大值 |
| 不配置 | 取最后一行（向后兼容） |

**算子名通配符匹配：** `--op` 支持 `fnmatch` 通配符（`*`、`?`、`[seq]`）：

```bash
# 匹配所有 matmul 变体
forge test my_repo --type performance --op "matmul*"

# 匹配 add_v1, add_v2（不匹配 add_v10）
forge test my_repo --type performance --op "add_v?"
```

### 结果提取示例

多步骤任务中，`result` 配置放在需要提取结果的步骤上（通常是最后一步）：

```json
"tasks": {
  "accuracy": {
    "steps": [
      {
        "name": "generate input",
        "command": "python gen_input.py",
        "args": ["--op", "{op_name}"],
        "working_dir": "tests/"
      },
      {
        "name": "run operator",
        "command": "bash run_op.sh",
        "args": ["{op_name}"],
        "working_dir": "."
      },
      {
        "name": "compare result",
        "command": "python compare.py",
        "args": ["--op", "{op_name}"],
        "working_dir": "tests/",
        "success_pattern": "PASS",
        "result": {
          "type": "stdout",
          "pattern": "accuracy: ([0-9.]+)"
        }
      }
    ],
    "timeout": 300
  }
}

## CLI 使用

全局选项必须放在子命令前面：

```bash
forge [--config-dir PATH] [-v/--verbose] <command> [options]
```

### 短选项速查

| 短选项 | 长选项 | 适用命令 |
|--------|--------|----------|
| `-v` | `--verbose` | 全局 |
| `-t` | `--type` | test, run（bisect 已废弃，使用 `bisect.test_type` 配置） |
| `-c` | `--commit` | build, test, run |
| `-s` | `--step` | build, run |
| `-b` | `--baseline` | test, run |
| `-w` | `--workflow` | bisect |
| `-g` | `--good` | bisect |
| `-b` | `--bad` | bisect |

`--type` 支持缩写：`acc` = `accuracy`，`perf` = `performance`。

### Daemon 和 Task 子命令

| 命令 | 说明 |
|------|------|
| `forge daemon start [--foreground]` | 启动 daemon（后台运行） |
| `forge daemon stop` | 停止 daemon |
| `forge daemon status` | 查看 daemon 状态和任务统计 |
| `forge daemon install` | 安装 systemd 服务（需 sudo） |
| `forge daemon uninstall` | 卸载 systemd 服务（需 sudo） |
| `forge task list [--all\|--running\|--completed]` | 列出任务 |
| `forge task show <id> [--tail N] [--follow]` | 查看任务输出 |
| `forge task cancel <id>` | 取消任务 |
| `forge task resume <id>` | 恢复被中断的任务 |

### 查看配置

```bash
# 列出所有已配置的算子仓
forge list

# 查看某个仓库的完整配置
forge show my_repo

# 校验配置文件是否合法
forge validate my_repo

# 使用自定义配置目录
forge --config-dir /path/to/configs list
```

### 迁移旧配置

如果配置文件使用旧格式（`repo_path`、`build`、`tests`），可一键迁移到新格式：

```bash
# 迁移配置目录下所有 JSON 文件
forge migrate

# 预览变更（不修改文件）
forge migrate --dry-run

# 使用自定义配置目录
forge migrate --config-dir /path/to/configs
```

**迁移内容：**

1. **路径字段迁移：**
   - `repo_path` → `working_dir`
   - `{repo_path}` 占位符 → `{working_dir}`
   - `test_path` → 移入 `vars.test_path`

2. **任务格式迁移：**
   - `build` → `tasks.build`
   - `tests.accuracy` → `tasks.accuracy`（无 `test_` 前缀）
   - `tests.performance` → `tasks.performance`
   - `tests.source_env` → 移除（需在具体任务中配置 `source_env`）

3. **Workflow 自动生成：**
   - `bisect.build_before_test: true` → 自动生成 `_bisect_default` workflow
   - 列表格式 workflow `["a", "b"]` → 对象格式 `{"steps": ["a", "b"]}`
   - `bisect.post_workflow: true` → 具体名称（如 `accuracy_pipeline`）

4. **Bisect 配置迁移（新）：**
   - `bisect.workflow: "name"` → `bisect.workflows: ["name"]`（数组格式）
   - `bisect.post_workflow: "name"` → `bisect.post_workflows: ["name"]`（数组格式）
   - 添加 `bisect.test_type` 字段（从 CLI `--type` 参数移入配置）

迁移示例输出：
```
example-build-base.json (migrated):
  - repo_path → working_dir
  - build → tasks.build
  - tests.accuracy → tasks.accuracy
  - tests.performance → tasks.performance
  - bisect.workflow → bisect.workflows (array format)
  - added bisect.test_type: accuracy

5 change(s) applied.
```

### 编译

```bash
# 编译当前 HEAD
forge build my_repo

# 编译指定 commit（自动 stash 未提交更改，编译后恢复）
forge build my_repo --commit abc123

# 编译指定算子
forge build my_repo --op matmul

# 传入多个自定义参数
forge build my_repo --arg op_name=matmul --arg target=ascend910

# 只执行指定步骤（多步骤配置时）
forge build my_repo --step install     # 只安装，跳过编译
forge build my_repo --step compile     # 只编译，跳过安装
```

### 测试

```bash
# 精度测试
forge test my_repo --type accuracy

# 性能测试
forge test my_repo --type performance

# 测试特定算子
forge test my_repo --type accuracy --op matmul

# 在指定 commit 上测试（自动编译）
forge test my_repo --type performance --commit abc123

# 与指定 commit 的历史结果对比
forge test my_repo --type performance --baseline abc123

# 与上次保存的结果对比
forge test my_repo --type performance --compare-last

# 性能测试跑 3 次取平均值
forge test my_repo -t perf --op matmul --repeat 3

# 性能测试跑 3 次取最大值
forge test my_repo -t perf --op matmul --repeat 3 --repeat-agg max
```

### 编译+测试（run）

一个命令完成 build → test，build 失败则不执行 test。

```bash
# 编译后跑精度测试
forge run my_repo --type accuracy --op matmul

# 编译后跑性能测试，并与上次结果对比
forge run my_repo --type performance --op matmul --compare-last

# 在指定 commit 上编译+测试
forge run my_repo --type accuracy --commit abc123

# 只执行指定编译步骤后测试
forge run my_repo --type accuracy --step install
```

### 自定义任务

配置文件中可通过 `tasks` 定义任意名称的任务：

```json
{
  "tasks": {
    "deploy": {
      "command": "bash deploy.sh",
      "artifacts": ["lib/libkernel.so"]
    },
    "lint": {
      "steps": [
        {"name": "check", "command": "flake8 .", "success_pattern": "0 errors"},
        {"name": "format", "command": "black --check ."}
      ]
    }
  }
}
```

```bash
# 运行自定义任务
forge task-run my_repo deploy
forge task-run my_repo lint

# 支持所有通用选项
forge task-run my_repo deploy --arg key=val --op matmul
```

自定义任务支持所有步骤特性：`command`、`args`、`working_dir`、`timeout`、`env`、`success_pattern`、`artifacts`、`result`、`capture`、`skip_if_exists`、`steps`。

**向后兼容：** 旧的 `build` 和 `tests` 配置自动映射为任务（`build` → `build`，`tests.accuracy` → `test_accuracy`）。`forge build`、`forge test`、`forge run`、`forge bisect` 命令不受影响。

### Workflow（任务编排）

通过 `workflows` 定义多个任务的串行执行序列，支持步骤间变量传递：

```json
{
  "tasks": {
    "build": {"command": "bash build.sh"},
    "setup_env": {
      "command": "bash setup_test_env.sh",
      "capture": { "test_env_path": "ENV_PATH=(.+)" }
    },
    "test_accuracy": {
      "command": "python run_test.py",
      "args": ["--env={test_env_path}"]
    }
  },
  "workflows": {
    "full_pipeline": {
      "steps": ["build", "setup_env", "test_accuracy"]
    },
    "test_only": {
      "steps": ["test_accuracy"]
    }
  }
}
```

```bash
# 运行 workflow
forge workflow my_repo full_pipeline
forge workflow my_repo test_only --arg op_name=matmul
```

按顺序执行，任一任务失败则停止。前序步骤通过 `capture` 提取的变量自动传递给后续步骤。

**向后兼容：** 旧的列表格式 `"ci": ["build", "test"]` 自动转换为 `"ci": {"steps": ["build", "test"]}`，并显示迁移提示。

### Bisect 问题定位

在 good 和 bad 两个 commit 之间自动二分查找引入问题的 commit。使用 `bisect.workflows` 数组定义每步的执行流程。

**配置示例：**

```json
{
  "bisect": {
    "test_type": "accuracy",
    "workflows": ["build_test"],
    "post_workflows": ["cleanup"],
    "regression": 15.0
  }
}
```

**CLI 使用：**

```bash
# 精度 bisect（test_type 从配置读取）
forge bisect my_repo --good abc123 --bad def456

# 性能 bisect（需在配置中设置 test_type: "performance")
forge bisect my_repo --good abc123 --bad def456 --op matmul

# 自定义退化阈值（默认 10%，配置或 CLI 指定）
forge bisect my_repo --good abc123 --bad def456 --regression 20

# 使用绝对阈值（耗时超过 15 即判定为 bad）
forge bisect my_repo --good abc123 --bad def456 --threshold 15

# 重复测试以减少波动（仅对 repeatable=true 的工作流生效）
forge bisect my_repo --good abc123 --bad def456 --repeat 3 --repeat-agg avg

# 指定工作流（覆盖配置）
forge bisect my_repo --good abc123 --bad def456 --workflow custom_pipeline
```

**废弃提示：** `--type/-t` 已废弃。测试类型应在配置文件的 `bisect.test_type` 字段设置。使用 CLI `--type` 时会显示警告。

**Bisect 流程：**

1. 获取 good..bad 之间的 commit 列表
2. 性能测试时，先在 good commit 上运行最后一个 workflow 获取基线值
3. 二分查找：checkout → 按顺序执行 workflows 数组 → 判断 good/bad
4. 精度 bisect：退出码 + success_pattern 判断
5. 性能 bisect：与基线对比，超过退化阈值判定为 bad
6. 非最后一个 workflow 失败 → 跳过该 commit
7. 跳过超过总步数 50% 时中止
8. 完成后自动恢复原始分支 → 执行 post_workflows

**多工作流执行：**

`bisect.workflows` 数组中的工作流按顺序执行：
- 前序工作流失败 → 跳过当前 commit（不计入 good/bad 判断）
- 最后一个工作流失败 → 根据失败类型决定（精度测试失败为 bad，其他失败为 skip）
- `repeatable=true` 的工作流会应用 `--repeat` 参数

**配置优先级：** CLI `--workflow` 覆盖配置 `bisect.workflows[0]`（单工作流模式）

### 环境变量

| 变量 | 说明 |
|------|------|
| `FORGE_CONFIG_DIR` | 配置文件目录路径，等价于 `--config-dir` |

## 后台任务（Daemon 模式）

Daemon 是一个常驻的后台进程，管理和执行提交的后台任务，支持任务持久化和自动恢复。

### 启动和停止

```bash
# 启动 daemon（后台运行，自动 fork）
forge daemon start

# 前台运行（调试用）
forge daemon start --foreground

# 停止 daemon
forge daemon stop

# 查看状态
forge daemon status
```

### 提交后台任务

任意命令加上 `--daemon` 即可后台运行：

```bash
# 后台编译
forge build my_repo --daemon

# 后台测试
forge test my_repo --type accuracy --daemon

# 后台 bisect
forge bisect my_repo --good abc --bad def --type accuracy --daemon
```

提交后立即返回 task_id：`Task submitted: task_abc123 (running in background)`

### 管理任务

```bash
# 查看运行中的任务（默认）
forge task list

# 查看所有任务
forge task list --all

# 查看已完成的任务
forge task list --completed

# 取消任务
forge task cancel task_abc123

# 恢复被中断的任务
forge task resume task_abc123
```

### 查看任务输出

```bash
# 查看累积输出（默认最近 100 行）
forge task show task_abc123

# 指定行数
forge task show task_abc123 --tail 50

# 实时跟踪（Ctrl+C 停止）
forge task show task_abc123 --follow
```

输出文件保存在 `~/.forge/logs/task_{task_id}.log`，任务结束后保留。

### 自动恢复

Daemon 重启后自动恢复被中断的任务（最多重试 3 次）。

### systemd 服务

```bash
# 安装为系统服务（需 root）
sudo forge daemon install

# 卸载服务
sudo forge daemon uninstall
```

安装后 daemon 随系统自动启动。

## 结果存储

测试结果自动保存到 `~/.forge/results/<repo_name>/`，JSON 格式：

```json
{
  "repo": "my_repo",
  "commit": "abc1234",
  "timestamp": "2026-03-23T10:00:00",
  "type": "performance",
  "results": {
    "matmul": 12.3,
    "add": 0.8
  }
}
```

可通过 `--compare-last` 或 `--baseline` 与历史结果对比。

## 日志系统

每次编译和测试的完整输出自动保存到日志文件，方便问题定位。

### 日志文件位置

```
~/.forge/logs/<repo_name>/
├── build_2026-03-24T10-30-00.log
├── test_accuracy_2026-03-24T10-35-00.log
└── test_performance_2026-03-24T10-40-00.log
```

### 日志内容

每个日志文件包含：
- 执行时间、完整命令、工作目录
- 退出码、是否超时、是否成功
- 编译/测试脚本的 **完整 stdout 和 stderr**

### 屏幕输出

```bash
# 默认模式：打印执行的命令和工作目录，失败时打印 stderr
forge build my_repo

# verbose 模式：实时逐行打印脚本的 stdout/stderr
forge --verbose build my_repo
forge --verbose test my_repo --type accuracy
```

### 排查编译/测试失败

1. 加 `--verbose` 查看实时输出：`forge --verbose build my_repo`
2. 查看日志文件：`cat ~/.forge/logs/my_repo/build_*.log`
3. 检查日志中的 stderr 和退出码
4. 对比手动执行时的环境变量（配置文件中 `tasks.build.env` 是否缺失关键变量）

## 扩展

- **新增算子仓**：在 `configs/` 下添加一个 JSON 文件即可，无需改代码
- **新增任务**：在配置文件的 `tasks` 下新增 key（如 `stress`），框架通用处理
- **新增结果文件格式**：在 `tester.py` 的 `extract_result_from_file` 中扩展解析器
- **新增任务类型**：继承 `TaskRunner`，覆盖 `_post_step()` 和 `_make_result()` 即可。执行引擎（单步/多步、skip、capture、日志等）自动继承

## 开发

```bash
# 运行测试
source .venv/bin/activate
python -m pytest tests/ -v
```
