Composable discrete-value model
Scalar Q, QR-DQN, IQN and FQF share one sequence-safe learner; head and strategy determine the value
representation.
Representation path
Head + strategy
Double-DQN update
after step
Sequence batch
n-step · masks ·
burn-in
Frame adapter
[B,T,…] → [B·T,…]
Sensor encoder
independent frames
Temporal core
Identity · GRU ·
Mamba
Value head
scalar · QR ·
IQN/FQF
Value
strategy
support ·
expectation
Action values
[..., quantiles,
actions]
Current
action
selected
distribution
Online next
state
all actions →
argmax
Target next
state
evaluate
selected argmax
Optimizers
main · FQF
fraction
Loss +
priority
TD/quantile
loss · PER
Target sync
parameters +
buffers
Valid pairs: ScalarQ + Scalar · FixedQuantile + Fixed · ImplicitQuantile + Random (IQN) or Learned
fractions (FQF).
Expectation owns the quantile axis. Selected-action priorities add a singleton action axis and are
validated before backward.
Download editable diagram