Build an optimization workflow you can audit.

Usage

Optimize the serving distribution.

A single signature remains supported. For production selection, supply weighted cases for the shapes, batches, dtypes, and keyword forms that actually receive traffic.

from custom_dl_optimizer import (
    DeploymentConstraints, ExecutionTarget,
    InferenceOptimizer, OptimizationPolicy,
    WorkloadCase, WorkloadProfile,
)

optimizer = InferenceOptimizer(
    target=ExecutionTarget("cuda"),
    policy=OptimizationPolicy(
        objective="lifecycle_latency",
        enable_compile=True,
        plan_cache_dir=".custom-dl-cache",
        constraints=DeploymentConstraints(
            expected_calls=10_000,
            min_speedup=1.02,
        ),
    ),
)
profile = WorkloadProfile(
    name="serving",
    expected_calls=10_000,
    cases=(
        WorkloadCase("b1", args=(batch_1,), weight=80),
        WorkloadCase("b8", args=(batch_8,), weight=20),
    ),
)
decision = optimizer.select(model.eval(), profile)
decision.save_bundle("artifacts/optimization")

Read the decision

Each candidate records per-case first calls and parity, serial P50/P90/P95/P99 samples, bootstrap lifecycle bounds, incremental CUDA allocation, constraints, break-even calls, and failures. Overlapping evidence retains the native baseline.

Correctness boundary

Candidates must preserve output structure and satisfy torch.allclose using configured tolerances. Always run task-level accuracy evaluation before deployment.

Execution providers

Compare maintained runtimes under the same policy.

First-party providers cover Torch-TensorRT, ONNX Runtime, and TorchAO. Private backends can still implement the provider protocol.

from custom_dl_optimizer import (
    ExecutionTarget, InferenceOptimizer,
    ONNXRuntimeProvider,
    TorchAOQuantizationProvider,
    TorchTensorRTProvider,
)

decision = InferenceOptimizer(
    target=ExecutionTarget("cuda"),
    providers=(
        TorchTensorRTProvider(),
        ONNXRuntimeProvider(),
        TorchAOQuantizationProvider(),
    ),
).select_signature(model, sample)

Fair comparison

Keep transfers and precision consistent, separate engine-build cost from steady-state latency, warm caches, expose provider configuration, and never silently run another backend under the same provider name.

Agent toolkit

Register objects locally; expose names remotely.

Live modules and tensors stay in the host process. The toolkit exposes only runtime inspection, workload listing, optimization, and report retrieval.

from custom_dl_optimizer import ExecutionTarget, InferenceOptimizer, OptimizationAgentToolkit

toolkit = OptimizationAgentToolkit(
    InferenceOptimizer(target=ExecutionTarget("cuda"))
)
toolkit.register_workload(
    "encoder-b16",
    encoder,
    tokens,
    attention_mask=mask,
)

schemas = toolkit.tool_schemas()
report = toolkit.invoke(
    "custom_dl_optimize",
    {"workload": "encoder-b16"},
)

Security boundary

There is no tool argument for a file path, import, download, shell command, Python expression, model weight, or credential. The host remains responsible for deciding which workload names an agent may access.

Migration

Move from point estimates to a versioned decision.

Version 2 used a flat OptimizationConfig and provider modules. Version 3 separates target, policy, runner, plan, and decision contracts.

# Version 2
result = Optimizer(device="cuda", config=config).optimize(model, sample)

# Version 3
decision = InferenceOptimizer(
    target=ExecutionTarget("cuda"),
    policy=policy,
).select_signature(model, sample)
runner = decision.runner
report = decision.report

Report, bundle, paper-export, and cache schemas are version 3. Existing v2 cache records are ignored.