Diagnose a failed material#

Materials reach FAILED for two very different reasons, and the fix differs completely. This page is how you tell them apart in about ten seconds.

Start with status#

$ matmelt status

Status of run nacl-demo:
  Materials by state
┏━━━━━━━━━━━━┳━━━━━━━┓
┃ State      ┃ Count ┃
┡━━━━━━━━━━━━╇━━━━━━━┩
│ EXPANDED   │     2 │
│ FAILED     │     3 │
│ STRUCTURED │    32 │
└────────────┴───────┘
                                Failed materials
┏━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Material    ┃ Stage  ┃ Attempts ┃ Reason                                     ┃
┡━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Cl1Na1_3D-1 │ screen │        3 │ stage 'screen' cannot run on item          │
│             │        │          │ 'Cl1Na1_3D-1': missing or empty required   │
│             │        │          │ input file(s) ['structure.cif'] in         │
│             │        │          │ materials-work/Cl1Na1_3D-1 (one candidate  │
│             │        │          │ crystal from the structure stage)          │
└─────────────┴────────┴──────────┴────────────────────────────────────────────┘

status exits 0 while listing failures. Failures are content, not a command failure.

Read the attempt count first#

Attempts: 3 (or whatever your retry_budget is) means a fault: something went wrong that might not have. MatMelt retried it to exhaustion. The cause is almost always environmental — a missing file, a crashed job, a backend that went away — and it is usually yours to fix.

Attempts: 1 means a science verdict: a deterministic rejection. Nothing was retried, because re-running reproduces it exactly. The material was screened out on its merits:

material Cl1Na1_3D-3 failed terminally at stage 'screen' (verdict): screened
out: energy -578480.6344 eV/atom outside the physical window [-35.0, 10.0]
eV/atom -- the model is almost certainly extrapolating; minimum interatomic
distance 0.8069 A < 1.2 A -- atoms are overlapping (unphysical geometry);
residual stress 627297760.068 GPa > 2.0 GPa -- unit cell not mechanically
equilibrated; final fmax 293121.41212 eV/A > acceptance gate 0.5 eV/A -- no
local minimum reached

Every clause is quantified against a stated bound, so you can judge the rejection rather than trust it. A verdict like this needs no action: it is the funnel doing its job. Retrying it only burns the same compute for the same answer.

The run itself exits 0 either way. A run in which materials were screened out did what you asked.

Then read the reason#

The reason is stored verbatim, so it names the stage, the material, the file and the directory. Two shapes come up constantly:

A contract refusal — the stage checked its inputs and would not start:

stage 'screen' cannot run on item 'Cl1Na1_3D-1': missing or empty required
input file(s) ['structure.cif'] in materials-work/Cl1Na1_3D-1

Note empty counts as missing: a truncated file fails the same way a deleted one does.

A semantic refusal — the inputs exist but say the wrong thing, e.g. extract asked to work on a material the screening stage rejected. Both are faults, both spend the budget, and both are fixed upstream rather than by retrying harder.

To list just the failures as ids — the picking list for a targeted re-run:

$ matmelt status --state FAILED

Fix, then retry#

retry resets FAILED materials so a later run picks them up again. Preview first; it changes nothing without --yes:

$ matmelt retry --dry-run

Run nacl-demo: 2 FAILED material(s) across 1 stage(s) to retry:
                           Failed materials by stage
┏━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Stage  ┃ Count ┃ Distinct reasons ┃ Example reason                           ┃
┡━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ screen │     2 │                2 │ stage 'screen' cannot run on item        │
│        │       │                  │ 'Cl1Na1_3D-6': missing or empty required │
│        │       │                  │ in...                                    │
└────────┴───────┴──────────────────┴──────────────────────────────────────────┘
retry --dry-run: nothing changed.

Narrow it with --stage (everything that failed at one stage) or --materials (a specific list), then apply:

$ matmelt retry --materials Cl1Na1_3D-6,Cl1Na1_3D-10 --yes

Reset 2 material(s) -- they will re-run at their failed stage on the next
'matmelt run'.

Each material resumes from its own timeline — the output state of the last stage it actually completed — so a reset is correct even if you have since changed the funnel. The completion history is preserved, and the lease, attempt count and error are cleared.

Then run again. In this capture the underlying cause (two structure.cif files truncated to zero bytes) had been fixed first, so the materials went straight through:

$ matmelt run --only screen --materials Cl1Na1_3D-6,Cl1Na1_3D-10 --yes

pass 23: completed=1
pass 33: completed=1
  Materials by state
┏━━━━━━━━━━━━┳━━━━━━━┓
┃ State      ┃ Count ┃
┡━━━━━━━━━━━━╇━━━━━━━┩
│ EXPANDED   │     2 │
│ FAILED     │     3 │
│ SCREENED   │     2 │
│ STRUCTURED │    32 │
└────────────┴───────┘

Retrying without fixing the cause reproduces the failure, one budget at a time. retry re-queues; it does not repair.

When a run refuses to start at all#

A missing dependency is caught before any material is touched:

$ matmelt run --yes

Run nacl-demo: compose -> structure
Seed enumeration: 2 element(s) -> 1 combination(s) (size 2: 1).
run: not ready -- missing requirements:
  compose: smact (ModuleNotFoundError: No module named 'smact')
To fix, run:
  /usr/bin/python3 -m pip install smact

Exit 1, nothing queued, nothing failed. The preflight covers only the stages this run would actually execute, so a missing calculate dependency cannot block a run that stops at screen. See Check that a machine can do the work.