Recurrent replay sequence, n-step target and PER feedback
A sampled anchor is eligible only when its full episode-local history and n-step horizon are available; burn-in is context, not a loss region.
Replay eligibility
Sequence unroll
Value target
Prioritized replay
feedback
sample history
hidden state
next sampled
batch
Replay transitions
interleaved actor IDs
Episode-local chain
same episode + step
no boundary crossing
Eligible anchor
L unique history IDs +
complete n-step horizon
Sampled anchor weight
normalized IS weight
Sequence window
t0 … tL−1; priority anchor = tL−1
Burn-in [0, B)
state update only
detached gradient
Learning
positions
B … L−n−1 plus
final anchor
Validity mask
valid loss
positions only
Target history
next window keeps
context
Per-position loss
value regression
per valid position
Double-Q bootstrap
online selects
target evaluates
n-step target
Gₜ = Σ γⁱ rₜ₊ᵢ + γᵏ
Qtarget(sₜ₊ₖ)
Bootstrap contract
terminated: zero
truncated: bootstrap
Absolute value error
one |TD| per trained
position
Sequence priority
p = 0.9 · max(|TD|) +
0.1 · mean(|TD|)
Sampling probability
p^alpha + optional
uniform/elite mix
Importance
correction
w = (N · P)^−beta
normalized by max
Validated before training: n_step < sequence_length, burn_in < sequence_length, and sequence replay cannot be combined with a feature pipeline that
already stacks history.
Scope: R2D2-style recurrent replay. The package reconstructs episode-local histories and burn-in, but does not claim the complete distributed R2D2
system.
Download editable diagram