Reliability & architecture analysis for agentic systems · by ActaClad
| Severity | Rule | Location | What & why |
|---|---|---|---|
| Major Medium | PLB-OBS-001 · NIST-AI-RMF:MEASURE | llm/default_plugins/openai_models.py:961:26 | This project ships LLM/agent code with no in-code tracing or instrumentation; production failures will be undiagnosable. (If you enable tracing via env vars / auto-instrumentation, ignore this.) Note: harness rules reason about the whole repo — scan the project root (including tests/ and CI), not just src/. An LLM/agent app with no tracing is unobservable in production — failures can't be traced. How to fixInstrument the app so production runs are traceable. - An observability SDK: OpenTelemetry, LangSmith, Langfuse, Phoenix/Arize, Logfire, Helicone, Traceloop, … - or the framework's tracing callbacks (LangChain callbacks, CrewAI telemetry). If you already enable tracing via environment variables (LANGCHAIN_TRACING_V2=true, OTEL_* auto-instrumentation), this finding does not apply — Plumbline cannot see env-var activation; ignore or suppress it. |
| Critical High | PLB-OUT-001 | llm/default_plugins/openai_models.py · 2 sites 1027:35, 1143:35 | LLM output is parsed with json.loads and no error handling; a malformed generation will raise and crash the request. `json.loads` on raw LLM output with no guard crashes on the first malformed generation. How to fixValidate model output against a schema and handle parse failure (retry or fallback).
Bad:
data = json.loads(resp.choices[0].message.content)
Good:
try:
data = json.loads(resp.choices[0].message.content)
except json.JSONDecodeError:
data = repair_or_retry(...)
# better: use the provider's structured-output / response_format and validate |