# t8 — associate BUDGET MEASUREMENT, Jetson AGX Orin 64GB (Ampere sm_87)
# =====================================================================
# 2026-08-25. Plan `lightning-on-orin`, task t8. Engine vllm/vllm-openai:v0.27.1,
# checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 with the DSpark
# drafter (5 speculative tokens), --max-model-len 128000, --kv-cache-dtype bfloat16.
#
# WHAT THIS MEASURES, AND WHAT IT DOES NOT
# ----------------------------------------
# The GO transcript (docs/evidence/2026-08-25-spike-lightning-vllm-orin.txt)
# measured util 0.7 SOLO -- every other lane on the board stopped. That number
# CANNOT be carried into a card profile: gpu_memory_utilization is a fraction of
# the WHOLE device, so every byte a co-resident engine holds is deducted from KV,
# which is the only elastic term. This run re-measures with the pooling gears
# resident, which is the shape the board actually deploys.
#
# CO-RESIDENT DURING THIS MEASUREMENT:
#   model-gear-vllm-embed    4.61 GiB   (instrument: docker stats)
#   model-gear-vllm-rerank   5.46 GiB   (instrument: docker stats)
#   prod-worker-1           ~0.02 GiB   (unrelated host workload, left running)
#
# NOT RESIDENT -- READ THIS BEFORE TRUSTING THE NUMBER:
#   `hand` (LiquidAI/LFM2.5-1.2B-Instruct, declared util 0.06 = ~3.68 GiB) was
#   NOT running: no vllm-hand container is scaffolded on this box at all. The
#   orin-associate shape DOES host it. The util declared below therefore reserves
#   hand's share as ARITHMETIC HEADROOM, not as measured co-residency. Do not
#   report this as "measured with hand". A boot with hand actually resident is
#   open follow-up work.
#
# INSTRUMENTS DISAGREE -- NAME THE ONE YOU MEAN (frame claims c32 / h13)
# ----------------------------------------------------------------------
# `nvidia-smi` reports memory.used [N/A] on this Tegra board; there is NO
# GPU-memory query here. `docker stats` accounts the two gears at ~10.1 GiB
# combined, while vLLM's own startup check saw 18.45 GiB of the 61.34 GiB device
# already gone. The difference is the host's own workload and allocator overhead.
# vLLM's number is the one that GOVERNS -- it is what the engine clamps against --
# so every figure below is the engine's own boot-log accounting unless stated.
#
# RESULT 1 -- util 0.7 (the vendor's value): REFUSED, by 0.05 GiB
# ---------------------------------------------------------------
#   ValueError: Free memory on device cuda:0 (42.89/61.34 GiB) on startup is less
#   than desired GPU memory utilization (0.7, 42.94 GiB). Decrease GPU memory
#   utilization or reduce GPU memory used by other processes.
#
# Recorded as a REFUSAL, not quietly rounded down (t8 acceptance criterion 2).
# It is a 50 MiB miss: the vendor's 0.7 fits this board SOLO and does not fit it
# with two ~0.6B pooling gears resident. That is the whole reason a card profile
# may not copy a vendor number forward.
#
# RESULT 2 -- util 0.63: BOOTS, SERVES, AND LEAVES ROOM FOR hand
# --------------------------------------------------------------
#   Model loading took          19.45 GiB   (identical to the solo run -- weights
#                                            are fixed; only KV is elastic)
#   Available KV cache memory   10.95 GiB
#   GPU KV cache size        1,249,280 tokens
#   Max concurrency @ 128,000     9.76x     (a KV-pool CEILING, not throughput)
#   Known-answer probe           PASS       ('Paris', finish_reason=stop)
#   Weight load time             51.3 s     (warm -- weights already in the HF cache)
#
# Post-boot host state (instrument: free -g): 53 GiB used, ~8 GiB available, which
# is why hand's ~3.68 GiB is believed to fit -- believed, not measured.
#
# COMPARISON TO THE SOLO RUN (same instrument, engine boot log):
#   util 0.70 solo:  KV 15.07 GiB / 1,720,320 tokens / 13.44x
#   util 0.63 + gears: KV 10.95 GiB / 1,249,280 tokens /  9.76x
#   Weights are 19.45 GiB in BOTH -- the entire difference is KV.
#
# EXCLUSIVE-ROLES ARITHMETIC (feeds t9's [[exclusive_roles]] group):
#   The board has 61.34 GiB of unified memory and ZERO swap. associate at util
#   0.63 holds ~37.7 GiB resident (instrument: docker stats). The llama.cpp GGUF
#   cortex holds ~27.6 GiB (docker stats) / ~33 GiB (the card profile's own
#   figure), and `senses` holds 0.45 x 61.3 = ~27.6 GiB. associate therefore
#   cannot co-reside with EITHER heavy lobe -- both sides of that arithmetic are
#   now measured rather than inferred.
#
# NETWORK: this container ran on the default bridge with NO published ports and
# NO --network host, so it was unreachable from the tailnet for the whole run
# (frame claim c30 / honesty condition h12). The GO spike, run verbatim per the
# vendor recipe, was NOT -- two tailnet peers reached it; see c46.
#
# ---------------------- raw `docker logs` output below -----------------------

WARNING 08-25 19:51:48 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` option. The `--model` option will be removed in a future version.
(APIServer pid=1) INFO 08-25 19:51:48 [api_utils.py:345] 
(APIServer pid=1) INFO 08-25 19:51:48 [api_utils.py:345]        █     █     █▄   ▄█
(APIServer pid=1) INFO 08-25 19:51:48 [api_utils.py:345]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.27.1
(APIServer pid=1) INFO 08-25 19:51:48 [api_utils.py:345]   █▄█▀ █     █     █     █  model   nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
(APIServer pid=1) INFO 08-25 19:51:48 [api_utils.py:345]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 08-25 19:51:48 [api_utils.py:345] 
(APIServer pid=1) INFO 08-25 19:51:48 [api_utils.py:273] non-default args: {'model_tag': 'nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4', 'trust_remote_code': True, 'max_model_len': 128000, 'served_model_name': ['associate'], 'mamba_backend': 'flashinfer', 'enable_mamba_cache_stochastic_rounding': True, 'mamba_cache_philox_rounds': 5, 'reasoning_parser': 'nemotron_v3', 'gpu_memory_utilization': 0.63, 'kv_cache_dtype': 'bfloat16', 'enable_prefix_caching': True, 'mamba_ssm_cache_dtype': 'float16', 'mamba_cache_mode': 'align', 'max_num_batched_tokens': 16384, 'speculative_config': {'method': 'dspark', 'model': 'nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark', 'num_speculative_tokens': 5, 'kv_cache_dtype': 'bfloat16'}}
(APIServer pid=1) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(APIServer pid=1) INFO 08-25 19:51:49 [model.py:645] Resolved architecture: NemotronHForCausalLM
(APIServer pid=1) INFO 08-25 19:51:49 [model.py:1883] Using max model len 128000
(APIServer pid=1) /usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU0 Orin which is of compute capability (CC) 8.7.
(APIServer pid=1) The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
(APIServer pid=1) - 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
(APIServer pid=1) - 9.0 which supports hardware CC >=9.0,<10.0
(APIServer pid=1) - 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
(APIServer pid=1) - 11.0 which supports hardware CC >=11.0,<12.0
(APIServer pid=1) - 12.0 which supports hardware CC >=12.0,<13.0
(APIServer pid=1) No published PyTorch CUDA builds for release 2.13.0+cu130 support this GPU. Visit https://pytorch.org/get-started/locally/ to find a compatible release.
(APIServer pid=1)   _warn_unsupported_code(d, device_cc, code_ccs)
(APIServer pid=1) INFO 08-25 19:51:59 [model.py:645] Resolved architecture: Qwen3DSparkModel
(APIServer pid=1) INFO 08-25 19:51:59 [model.py:1883] Using max model len 1048576
(APIServer pid=1) INFO 08-25 19:51:59 [speculative.py:1157] Overriding draft model max model len from 1048576 to 128000
(APIServer pid=1) INFO 08-25 19:51:59 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=16384.
(APIServer pid=1) INFO 08-25 19:51:59 [config.py:638] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=1) WARNING 08-25 19:51:59 [modelopt.py:380] Detected ModelOpt fp8 checkpoint (quant_algo=FP8). Please note that the format is experimental and could change.
(APIServer pid=1) WARNING 08-25 19:51:59 [modelopt.py:1021] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=1) WARNING 08-25 19:51:59 [modelopt.py:1021] Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=1) WARNING 08-25 19:51:59 [modelopt.py:1693] Detected ModelOpt MXFP8 checkpoint. Please note that the format is experimental and could change in future.
(APIServer pid=1) INFO 08-25 19:51:59 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) WARNING 08-25 19:51:59 [vllm.py:2267] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:422: UserWarning: Found GPU0 Orin which is of compute capability (CC) 8.7.
The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
- 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
- 9.0 which supports hardware CC >=9.0,<10.0
- 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
- 11.0 which supports hardware CC >=11.0,<12.0
- 12.0 which supports hardware CC >=12.0,<13.0
No published PyTorch CUDA builds for release 2.13.0+cu130 support this GPU. Visit https://pytorch.org/get-started/locally/ to find a compatible release.
  _warn_unsupported_code(d, device_cc, code_ccs)
(EngineCore pid=129) INFO 08-25 19:52:19 [core.py:121] Initializing a V1 LLM engine (v0.27.1) with config: model='nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4', speculative_config=SpeculativeConfig(method='dspark', model='nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark', num_spec_tokens=5), tokenizer='nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=128000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=modelopt_mixed, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=bfloat16, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='nemotron_v3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=associate, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=129) INFO 08-25 19:52:20 [parallel_state.py:1640] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.2:55875 backend=nccl
(EngineCore pid=129) INFO 08-25 19:52:20 [parallel_state.py:1977] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=129) INFO 08-25 19:52:20 [gpu_worker.py:385] Using V2 Model Runner
(EngineCore pid=129) INFO 08-25 19:52:21 [model_runner.py:308] Loading model from scratch...
(EngineCore pid=129) INFO 08-25 19:52:21 [__init__.py:665] Selected MarlinFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
(EngineCore pid=129) INFO 08-25 19:52:21 [__init__.py:1030] Using MarlinNvFp4LinearKernel for NVFP4 GEMM
(EngineCore pid=129) INFO 08-25 19:52:21 [nvfp4.py:285] Using 'MARLIN' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION'].
(EngineCore pid=129) INFO 08-25 19:52:22 [cuda.py:482] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=129) INFO 08-25 19:52:22 [flash_attn.py:789] Using FlashAttention version 2
(EngineCore pid=129) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(EngineCore pid=129) INFO 08-25 19:52:25 [weight_utils.py:867] Filesystem type for checkpoints: EXT4. Checkpoint size: 20.08 GiB. Available RAM: 25.08 GiB.
(EngineCore pid=129) INFO 08-25 19:52:25 [weight_utils.py:890] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:   0% Completed | 0/52 [00:00<?, ?it/s]
(EngineCore pid=129) 
Loading safetensors checkpoint shards:   2% Completed | 1/52 [00:00<00:18,  2.71it/s]
(EngineCore pid=129) WARNING 08-25 19:52:26 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:   4% Completed | 2/52 [00:01<00:29,  1.70it/s]
(EngineCore pid=129) WARNING 08-25 19:52:26 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:   8% Completed | 4/52 [00:01<00:21,  2.27it/s]
(EngineCore pid=129) WARNING 08-25 19:52:27 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  13% Completed | 7/52 [00:02<00:14,  3.13it/s]
(EngineCore pid=129) WARNING 08-25 19:52:28 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  17% Completed | 9/52 [00:02<00:12,  3.47it/s]
(EngineCore pid=129) WARNING 08-25 19:52:28 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  21% Completed | 11/52 [00:03<00:10,  3.78it/s]
(EngineCore pid=129) WARNING 08-25 19:52:29 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  27% Completed | 14/52 [00:03<00:08,  4.52it/s]
(EngineCore pid=129) WARNING 08-25 19:52:29 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  31% Completed | 16/52 [00:20<01:32,  2.58s/it]
(EngineCore pid=129) WARNING 08-25 19:52:46 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  35% Completed | 18/52 [00:21<01:05,  1.92s/it]
(EngineCore pid=129) WARNING 08-25 19:52:47 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  40% Completed | 21/52 [00:21<00:38,  1.25s/it]
(EngineCore pid=129) WARNING 08-25 19:52:47 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  44% Completed | 23/52 [00:22<00:28,  1.03it/s]
(EngineCore pid=129) WARNING 08-25 19:52:48 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  48% Completed | 25/52 [00:23<00:21,  1.27it/s]
(EngineCore pid=129) WARNING 08-25 19:52:48 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  54% Completed | 28/52 [00:23<00:14,  1.67it/s]
(EngineCore pid=129) WARNING 08-25 19:52:49 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  58% Completed | 30/52 [00:24<00:12,  1.81it/s]
(EngineCore pid=129) WARNING 08-25 19:52:50 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  62% Completed | 32/52 [00:25<00:10,  1.88it/s]
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  65% Completed | 34/52 [00:25<00:07,  2.52it/s]
(EngineCore pid=129) WARNING 08-25 19:52:51 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  67% Completed | 35/52 [00:26<00:08,  2.03it/s]
(EngineCore pid=129) WARNING 08-25 19:52:52 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  71% Completed | 37/52 [00:27<00:07,  2.06it/s]
(EngineCore pid=129) WARNING 08-25 19:52:53 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  75% Completed | 39/52 [00:28<00:06,  2.09it/s]
(EngineCore pid=129) WARNING 08-25 19:52:54 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  79% Completed | 41/52 [00:29<00:05,  2.07it/s]
(EngineCore pid=129) WARNING 08-25 19:52:55 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  85% Completed | 44/52 [00:30<00:03,  2.38it/s]
(EngineCore pid=129) WARNING 08-25 19:52:56 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  88% Completed | 46/52 [00:31<00:02,  2.36it/s]
(EngineCore pid=129) WARNING 08-25 19:52:57 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  92% Completed | 48/52 [00:32<00:01,  2.42it/s]
(EngineCore pid=129) WARNING 08-25 19:52:57 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:  96% Completed | 50/52 [00:32<00:00,  2.47it/s]
(EngineCore pid=129) WARNING 08-25 19:52:58 [routed_experts.py:1090] Unexpected gate/up projection names: up_proj, . Fused gate/up mapping will be skipped.
(EngineCore pid=129) 
Loading safetensors checkpoint shards: 100% Completed | 52/52 [00:33<00:00,  2.37it/s]
(EngineCore pid=129) 
Loading safetensors checkpoint shards: 100% Completed | 52/52 [00:33<00:00,  1.53it/s]
(EngineCore pid=129) 
(EngineCore pid=129) INFO 08-25 19:52:59 [default_loader.py:430] Loading weights took 33.93 seconds
(EngineCore pid=129) WARNING 08-25 19:52:59 [marlin_utils_fp8.py:112] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(EngineCore pid=129) WARNING 08-25 19:52:59 [marlin.py:34] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(EngineCore pid=129) WARNING 08-25 19:52:59 [marlin_utils_fp4.py:354] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
(EngineCore pid=129) INFO 08-25 19:53:00 [nvfp4.py:544] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=129) INFO 08-25 19:53:08 [eagle3_utils.py:28] Using Eagle3 auxiliary layers from config: (2, 6, 20, 30, 42, 52)
(EngineCore pid=129) INFO 08-25 19:53:08 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=129) WARNING 08-25 19:53:08 [vllm.py:2267] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
(EngineCore pid=129) WARNING 08-25 19:53:08 [modelopt.py:1021] Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is experimental and could change in future.
(EngineCore pid=129) WARNING 08-25 19:53:08 [modelopt.py:1021] Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4). Please note that the format is experimental and could change in future.
(EngineCore pid=129) INFO 08-25 19:53:08 [cuda.py:482] Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN'].
(EngineCore pid=129) INFO 08-25 19:53:10 [weight_utils.py:867] Filesystem type for checkpoints: EXT4. Checkpoint size: 1.26 GiB. Available RAM: 18.27 GiB.
(EngineCore pid=129) 
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore pid=129) 
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 39.27it/s]
(EngineCore pid=129) 
(EngineCore pid=129) INFO 08-25 19:53:11 [default_loader.py:430] Loading weights took 0.94 seconds
(EngineCore pid=129) WARNING 08-25 19:53:11 [modelopt.py:1354] In W4A16_NVFP4 linear, the global weight scale (weight_scale_2) differs across fused parallel layers (e.g. q/k/v_proj). This will likely reduce accuracy. Consider a checkpoint with a shared global scale.
(EngineCore pid=129) INFO 08-25 19:53:12 [model_runner.py:329] Model loading took 19.45 GiB and 51.329649 seconds
(EngineCore pid=129) INFO 08-25 19:53:12 [topk_topp_sampler.py:62] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=129) INFO 08-25 19:53:12 [interface.py:911] Setting attention block size to 1120 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=129) INFO 08-25 19:53:15 [caching.py:335] reconstructed serializable fn from standalone compile artifacts. num_artifacts=8 num_submods=30
(EngineCore pid=129) INFO 08-25 19:53:15 [decorators.py:311] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/8f1e1824a616042e46cebb3b8bce03878c93c682c6293d5a9db1314ec81c9ebd/rank_0_0/model
(EngineCore pid=129) INFO 08-25 19:53:15 [monitor.py:53] torch.compile took 1.54 s in total
(EngineCore pid=129) INFO 08-25 19:53:15 [mamba_mixer2.py:596] Warming up Mamba2 SSD Triton kernels...
(EngineCore pid=129) INFO 08-25 19:53:25 [monitor.py:81] Initial profiling/warmup run took 9.66 s
(EngineCore pid=129) INFO 08-25 19:53:26 [caching.py:335] reconstructed serializable fn from standalone compile artifacts. num_artifacts=3 num_submods=7
(EngineCore pid=129) INFO 08-25 19:53:26 [decorators.py:311] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/fe4ca03071d0cb035a5dba202d689fc7838ec5bdf8a78845ad8b56ec2289e2bb/rank_0_0/model
(EngineCore pid=129) INFO 08-25 19:53:26 [monitor.py:53] torch.compile took 0.13 s in total
(EngineCore pid=129) INFO 08-25 19:53:26 [monitor.py:81] Initial profiling/warmup run took 0.01 s
(EngineCore pid=129) INFO 08-25 19:53:27 [gpu_worker.py:563] Available KV cache memory: 10.95 GiB
(EngineCore pid=129) WARNING 08-25 19:53:27 [kv_cache_utils.py:1261] Add 1 padding layers, may waste at most 4.35% KV cache memory
(EngineCore pid=129) INFO 08-25 19:53:27 [kv_cache_utils.py:2235] GPU KV cache size: 1,249,280 tokens
(EngineCore pid=129) INFO 08-25 19:53:27 [kv_cache_utils.py:2236] Maximum concurrency for 128,000 tokens per request: 9.76x
(EngineCore pid=129) INFO 08-25 19:53:27 [ssu_dispatch.py:129] Using FlashInfer Mamba SSU algorithm: auto
(EngineCore pid=129) INFO 08-25 19:53:27 [ssu_dispatch.py:310] Using flashinfer Mamba SSU backend.
(EngineCore pid=129) INFO 08-25 19:53:27 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=129) 
Capturing CUDA graphs (PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (PIECEWISE):   2%|▏         | 1/51 [00:35<29:38, 35.57s/it]
Capturing CUDA graphs (PIECEWISE):   4%|▍         | 2/51 [00:35<12:08, 14.87s/it]
Capturing CUDA graphs (PIECEWISE):   6%|▌         | 3/51 [00:36<06:36,  8.26s/it]
Capturing CUDA graphs (PIECEWISE):   8%|▊         | 4/51 [00:36<04:01,  5.14s/it]
Capturing CUDA graphs (PIECEWISE):  10%|▉         | 5/51 [00:37<02:37,  3.42s/it]
Capturing CUDA graphs (PIECEWISE):  12%|█▏        | 6/51 [00:37<01:47,  2.38s/it]
Capturing CUDA graphs (PIECEWISE):  14%|█▎        | 7/51 [00:37<01:15,  1.72s/it]
Capturing CUDA graphs (PIECEWISE):  16%|█▌        | 8/51 [00:38<00:55,  1.28s/it]
Capturing CUDA graphs (PIECEWISE):  18%|█▊        | 9/51 [00:38<00:41,  1.02it/s]
Capturing CUDA graphs (PIECEWISE):  20%|█▉        | 10/51 [00:38<00:32,  1.28it/s]
Capturing CUDA graphs (PIECEWISE):  22%|██▏       | 11/51 [00:39<00:25,  1.56it/s]
Capturing CUDA graphs (PIECEWISE):  24%|██▎       | 12/51 [00:39<00:21,  1.84it/s]
Capturing CUDA graphs (PIECEWISE):  25%|██▌       | 13/51 [00:39<00:17,  2.13it/s]
Capturing CUDA graphs (PIECEWISE):  27%|██▋       | 14/51 [00:40<00:15,  2.35it/s]
Capturing CUDA graphs (PIECEWISE):  29%|██▉       | 15/51 [00:40<00:14,  2.55it/s]
Capturing CUDA graphs (PIECEWISE):  31%|███▏      | 16/51 [00:40<00:12,  2.73it/s]
Capturing CUDA graphs (PIECEWISE):  33%|███▎      | 17/51 [00:40<00:11,  2.89it/s]
Capturing CUDA graphs (PIECEWISE):  35%|███▌      | 18/51 [00:41<00:10,  3.04it/s]
Capturing CUDA graphs (PIECEWISE):  37%|███▋      | 19/51 [00:41<00:10,  3.20it/s]
Capturing CUDA graphs (PIECEWISE):  39%|███▉      | 20/51 [00:41<00:09,  3.31it/s]
Capturing CUDA graphs (PIECEWISE):  41%|████      | 21/51 [00:42<00:08,  3.41it/s]
Capturing CUDA graphs (PIECEWISE):  43%|████▎     | 22/51 [00:42<00:08,  3.50it/s]
Capturing CUDA graphs (PIECEWISE):  45%|████▌     | 23/51 [00:42<00:07,  3.63it/s]
Capturing CUDA graphs (PIECEWISE):  47%|████▋     | 24/51 [00:42<00:07,  3.73it/s]
Capturing CUDA graphs (PIECEWISE):  49%|████▉     | 25/51 [00:43<00:06,  3.85it/s]
Capturing CUDA graphs (PIECEWISE):  51%|█████     | 26/51 [00:43<00:06,  3.94it/s]
Capturing CUDA graphs (PIECEWISE):  53%|█████▎    | 27/51 [00:43<00:05,  4.03it/s]
Capturing CUDA graphs (PIECEWISE):  55%|█████▍    | 28/51 [00:43<00:05,  4.10it/s]
Capturing CUDA graphs (PIECEWISE):  57%|█████▋    | 29/51 [00:44<00:05,  4.15it/s]
Capturing CUDA graphs (PIECEWISE):  59%|█████▉    | 30/51 [00:44<00:05,  4.10it/s]
Capturing CUDA graphs (PIECEWISE):  61%|██████    | 31/51 [00:44<00:04,  4.10it/s]
Capturing CUDA graphs (PIECEWISE):  63%|██████▎   | 32/51 [00:44<00:04,  4.16it/s]
Capturing CUDA graphs (PIECEWISE):  65%|██████▍   | 33/51 [00:45<00:04,  4.24it/s]
Capturing CUDA graphs (PIECEWISE):  67%|██████▋   | 34/51 [00:45<00:03,  4.28it/s]
Capturing CUDA graphs (PIECEWISE):  69%|██████▊   | 35/51 [00:45<00:03,  4.31it/s]
Capturing CUDA graphs (PIECEWISE):  71%|███████   | 36/51 [00:45<00:03,  4.34it/s]
Capturing CUDA graphs (PIECEWISE):  73%|███████▎  | 37/51 [00:45<00:03,  4.37it/s]
Capturing CUDA graphs (PIECEWISE):  75%|███████▍  | 38/51 [00:46<00:02,  4.37it/s]
Capturing CUDA graphs (PIECEWISE):  76%|███████▋  | 39/51 [00:46<00:02,  4.38it/s]
Capturing CUDA graphs (PIECEWISE):  78%|███████▊  | 40/51 [00:46<00:02,  4.38it/s]
Capturing CUDA graphs (PIECEWISE):  80%|████████  | 41/51 [00:46<00:02,  4.40it/s]
Capturing CUDA graphs (PIECEWISE):  82%|████████▏ | 42/51 [00:47<00:02,  4.41it/s]
Capturing CUDA graphs (PIECEWISE):  84%|████████▍ | 43/51 [00:47<00:01,  4.43it/s]
Capturing CUDA graphs (PIECEWISE):  86%|████████▋ | 44/51 [00:47<00:01,  4.42it/s]
Capturing CUDA graphs (PIECEWISE):  88%|████████▊ | 45/51 [00:47<00:01,  4.44it/s]
Capturing CUDA graphs (PIECEWISE):  90%|█████████ | 46/51 [00:47<00:01,  4.30it/s]
Capturing CUDA graphs (PIECEWISE):  92%|█████████▏| 47/51 [00:48<00:00,  4.31it/s]
Capturing CUDA graphs (PIECEWISE):  94%|█████████▍| 48/51 [00:48<00:00,  4.34it/s]
Capturing CUDA graphs (PIECEWISE):  96%|█████████▌| 49/51 [00:48<00:00,  4.33it/s]
Capturing CUDA graphs (PIECEWISE):  98%|█████████▊| 50/51 [00:48<00:00,  4.35it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:49<00:00,  2.48it/s]
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 51/51 [00:49<00:00,  1.03it/s]
(EngineCore pid=129) 
Capturing CUDA graphs (FULL):   0%|          | 0/48 [00:00<?, ?it/s]
Capturing CUDA graphs (FULL):   2%|▏         | 1/48 [00:00<00:16,  2.90it/s]
Capturing CUDA graphs (FULL):   4%|▍         | 2/48 [00:00<00:15,  3.00it/s]
Capturing CUDA graphs (FULL):   6%|▋         | 3/48 [00:01<00:14,  3.01it/s]
Capturing CUDA graphs (FULL):   8%|▊         | 4/48 [00:01<00:14,  3.08it/s]
Capturing CUDA graphs (FULL):  10%|█         | 5/48 [00:01<00:13,  3.16it/s]
Capturing CUDA graphs (FULL):  12%|█▎        | 6/48 [00:01<00:13,  3.22it/s]
Capturing CUDA graphs (FULL):  15%|█▍        | 7/48 [00:02<00:12,  3.27it/s]
Capturing CUDA graphs (FULL):  17%|█▋        | 8/48 [00:02<00:11,  3.39it/s]
Capturing CUDA graphs (FULL):  19%|█▉        | 9/48 [00:02<00:11,  3.46it/s]
Capturing CUDA graphs (FULL):  21%|██        | 10/48 [00:03<00:10,  3.53it/s]
Capturing CUDA graphs (FULL):  23%|██▎       | 11/48 [00:03<00:10,  3.61it/s]
Capturing CUDA graphs (FULL):  25%|██▌       | 12/48 [00:03<00:09,  3.68it/s]
Capturing CUDA graphs (FULL):  27%|██▋       | 13/48 [00:03<00:09,  3.69it/s]
Capturing CUDA graphs (FULL):  29%|██▉       | 14/48 [00:04<00:09,  3.74it/s]
Capturing CUDA graphs (FULL):  31%|███▏      | 15/48 [00:04<00:08,  3.79it/s]
Capturing CUDA graphs (FULL):  33%|███▎      | 16/48 [00:04<00:08,  3.86it/s]
Capturing CUDA graphs (FULL):  35%|███▌      | 17/48 [00:04<00:07,  3.92it/s]
Capturing CUDA graphs (FULL):  38%|███▊      | 18/48 [00:05<00:07,  4.00it/s]
Capturing CUDA graphs (FULL):  40%|███▉      | 19/48 [00:05<00:07,  4.06it/s]
Capturing CUDA graphs (FULL):  42%|████▏     | 20/48 [00:05<00:06,  4.09it/s]
Capturing CUDA graphs (FULL):  44%|████▍     | 21/48 [00:05<00:06,  4.17it/s]
Capturing CUDA graphs (FULL):  46%|████▌     | 22/48 [00:06<00:06,  4.23it/s]
Capturing CUDA graphs (FULL):  48%|████▊     | 23/48 [00:06<00:05,  4.29it/s]
Capturing CUDA graphs (FULL):  50%|█████     | 24/48 [00:06<00:05,  4.39it/s]
Capturing CUDA graphs (FULL):  52%|█████▏    | 25/48 [00:06<00:05,  4.43it/s]
Capturing CUDA graphs (FULL):  54%|█████▍    | 26/48 [00:06<00:04,  4.50it/s]
Capturing CUDA graphs (FULL):  56%|█████▋    | 27/48 [00:07<00:04,  4.55it/s]
Capturing CUDA graphs (FULL):  58%|█████▊    | 28/48 [00:07<00:04,  4.59it/s]
Capturing CUDA graphs (FULL):  60%|██████    | 29/48 [00:07<00:04,  4.61it/s]
Capturing CUDA graphs (FULL):  62%|██████▎   | 30/48 [00:07<00:03,  4.58it/s]
Capturing CUDA graphs (FULL):  65%|██████▍   | 31/48 [00:07<00:03,  4.60it/s]
Capturing CUDA graphs (FULL):  67%|██████▋   | 32/48 [00:08<00:03,  4.62it/s]
Capturing CUDA graphs (FULL):  69%|██████▉   | 33/48 [00:08<00:03,  4.64it/s]
Capturing CUDA graphs (FULL):  71%|███████   | 34/48 [00:08<00:03,  4.65it/s]
Capturing CUDA graphs (FULL):  73%|███████▎  | 35/48 [00:08<00:02,  4.65it/s]
Capturing CUDA graphs (FULL):  75%|███████▌  | 36/48 [00:09<00:02,  4.67it/s]
Capturing CUDA graphs (FULL):  77%|███████▋  | 37/48 [00:09<00:02,  4.70it/s]
Capturing CUDA graphs (FULL):  79%|███████▉  | 38/48 [00:09<00:02,  4.71it/s]
Capturing CUDA graphs (FULL):  81%|████████▏ | 39/48 [00:09<00:01,  4.71it/s]
Capturing CUDA graphs (FULL):  83%|████████▎ | 40/48 [00:09<00:01,  4.73it/s]
Capturing CUDA graphs (FULL):  85%|████████▌ | 41/48 [00:10<00:01,  4.77it/s]
Capturing CUDA graphs (FULL):  88%|████████▊ | 42/48 [00:10<00:01,  4.79it/s]
Capturing CUDA graphs (FULL):  90%|████████▉ | 43/48 [00:10<00:01,  4.79it/s]
Capturing CUDA graphs (FULL):  92%|█████████▏| 44/48 [00:10<00:00,  4.81it/s]
Capturing CUDA graphs (FULL):  94%|█████████▍| 45/48 [00:10<00:00,  4.84it/s]
Capturing CUDA graphs (FULL):  96%|█████████▌| 46/48 [00:11<00:00,  4.85it/s]
Capturing CUDA graphs (FULL):  98%|█████████▊| 47/48 [00:11<00:00,  4.84it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 48/48 [00:11<00:00,  4.86it/s]
Capturing CUDA graphs (FULL): 100%|██████████| 48/48 [00:11<00:00,  4.17it/s]
(EngineCore pid=129) INFO 08-25 19:54:32 [speculator.py:137] Capturing model for DSpark speculator...
(EngineCore pid=129) 
Capturing dspark CUDA graphs (FULL):   0%|          | 0/48 [00:00<?, ?it/s]
Capturing dspark CUDA graphs (FULL):   2%|▏         | 1/48 [00:00<00:06,  7.78it/s]
Capturing dspark CUDA graphs (FULL):   6%|▋         | 3/48 [00:00<00:03, 12.50it/s]
Capturing dspark CUDA graphs (FULL):  10%|█         | 5/48 [00:00<00:02, 14.69it/s]
Capturing dspark CUDA graphs (FULL):  15%|█▍        | 7/48 [00:00<00:02, 16.06it/s]
Capturing dspark CUDA graphs (FULL):  21%|██        | 10/48 [00:00<00:02, 17.79it/s]
Capturing dspark CUDA graphs (FULL):  27%|██▋       | 13/48 [00:00<00:01, 19.20it/s]
Capturing dspark CUDA graphs (FULL):  33%|███▎      | 16/48 [00:00<00:01, 20.57it/s]
Capturing dspark CUDA graphs (FULL):  40%|███▉      | 19/48 [00:01<00:01, 22.09it/s]
Capturing dspark CUDA graphs (FULL):  46%|████▌     | 22/48 [00:01<00:01, 23.52it/s]
Capturing dspark CUDA graphs (FULL):  52%|█████▏    | 25/48 [00:01<00:00, 25.09it/s]
Capturing dspark CUDA graphs (FULL):  60%|██████    | 29/48 [00:01<00:00, 26.91it/s]
Capturing dspark CUDA graphs (FULL):  69%|██████▉   | 33/48 [00:01<00:00, 28.72it/s]
Capturing dspark CUDA graphs (FULL):  77%|███████▋  | 37/48 [00:01<00:00, 30.16it/s]
Capturing dspark CUDA graphs (FULL):  85%|████████▌ | 41/48 [00:01<00:00, 32.04it/s]
Capturing dspark CUDA graphs (FULL):  94%|█████████▍| 45/48 [00:01<00:00, 34.06it/s]
Capturing dspark CUDA graphs (FULL): 100%|██████████| 48/48 [00:01<00:00, 25.49it/s]
(EngineCore pid=129) INFO 08-25 19:54:34 [model_runner.py:791] Graph capturing finished in 64 secs, took 1.52 GiB
(EngineCore pid=129) INFO 08-25 19:54:34 [gpu_worker.py:789] Free memory on device (42.91/61.34 GiB) on startup. Desired GPU memory utilization is (0.63, 38.65 GiB). Actual usage is 24.95 GiB for consumed memory (weights + non-torch), 2.75 GiB for peak activation, and 1.52 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=9968454882` (9.28 GiB) to fit into requested memory, or `--kv-cache-memory=14540111360` (13.54 GiB) to fully utilize gpu memory. Current kv cache memory in use is 10.95 GiB.
(EngineCore pid=129) INFO 08-25 19:54:38 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=129) INFO 08-25 19:54:38 [core.py:348] init engine (profile, create kv cache, warmup model) took 86.25 s (compilation: 1.68 s)
(EngineCore pid=129) INFO 08-25 19:54:42 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) INFO 08-25 19:55:12 [api_server.py:678] Supported tasks: ['generate']
(APIServer pid=1) INFO 08-25 19:55:13 [parser_manager.py:37] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 08-25 19:55:17 [hf.py:540] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1) WARNING 08-25 19:55:17 [model.py:1637] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1) INFO 08-25 19:55:19 [api_server.py:682] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:37] Available routes are:
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1) INFO 08-25 19:55:19 [launcher.py:99] API server: waiting for HTTP server to start
(APIServer pid=1) INFO:     Started server process [1]
(APIServer pid=1) INFO:     Waiting for application startup.
(APIServer pid=1) INFO:     Application startup complete.
(APIServer pid=1) INFO 08-25 19:55:42 [launcher.py:105] API server: HTTP server started
(APIServer pid=1) INFO:     172.17.0.1:45382 - "GET /health HTTP/1.1" 200 OK
(EngineCore pid=129) WARNING 08-25 19:55:42 [jit_monitor.py:135] Triton kernel JIT compilation during inference: _prepare_dflash_inputs_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=129) WARNING 08-25 19:55:42 [jit_monitor.py:135] Triton kernel JIT compilation during inference: _compute_local_logits_stats_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=129) WARNING 08-25 19:55:42 [jit_monitor.py:135] Triton kernel JIT compilation during inference: _rejection_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=129) WARNING 08-25 19:55:42 [jit_monitor.py:135] Triton kernel JIT compilation during inference: _resample_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=1) INFO:     172.17.0.1:45390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-25 19:55:52 [loggers.py:310] Engine 000: Avg prompt throughput: 2.6 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 08-25 19:55:52 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.07, Accepted throughput: 1.57 tokens/s, Drafted throughput: 1.93 tokens/s, Accepted: 110 tokens, Drafted: 135 tokens, Per-position acceptance rate: 0.926, 0.852, 0.815, 0.778, 0.704, Avg Draft acceptance rate: 81.5%
