reference/generated/torchrl.modules.RunningValueNorm

Run in Google Colab

Colab

Download Notebook

Notebook

View on GitHub

GitHub

RunningValueNorm

class torchrl.modules.RunningValueNorm(*, shape: int | tuple[int, ...] = 1, epsilon: float = 1e-05, device: device | None = None)

Exact running mean / variance (Welford’s online algorithm).

Unlike PopArtValueNorm, this normaliser does not decay older samples — it accumulates the true sample mean and variance over every target it has ever seen. Useful when value targets are roughly stationary (no curriculum, no reward-shaping schedule), where the EMA’s adaptivity is unnecessary and the exact running stats give a slightly tighter estimate.

Keyword Arguments:

• shape – per-element shape of the value tensor. Defaults to 1.

• epsilon – numerical stabiliser added to the running variance. Defaults to 1e-5.

• device – device for the running-stats buffers.

Example

>>> vn = RunningValueNorm(shape=1)
>>> for _ in range(10):
...     vn.update(torch.randn(64, 1) * 3.0 + 1.0)
>>> normed = vn.normalize(torch.randn(8, 1))

denormalize(normalised_value: Tensor) → Tensor

Inverse of normalize() — recover real-scale values.

normalize(value_target: Tensor) → Tensor

Standardise value_target using the current running stats.

scale() → Tensor

Multiplicative scale currently applied by normalize().

Exposed separately so consumers can rescale quantities that must not be re-centred, e.g. advantages (already centred by the value baseline), for which only the division by the scale applies.

Deliberately not abstract until v0.16 so that subclasses written before it existed keep instantiating (with a DeprecationWarning); this default raises NotImplementedError when called.

update(value_target: Tensor) → None

Fold a batch of value targets into the running stats.