reference/objectives_value

Run in Google Colab

Colab

Download Notebook

Notebook

View on GitHub

GitHub

Value-Based Methods

Loss modules for value-based RL algorithms.

DQNLoss(*args, **kwargs)	The DQN Loss class.
DistributionalDQNLoss(*args, **kwargs)	A distributional DQN loss class.
IQLLoss(*args, **kwargs)	TorchRL implementation of the IQL loss.
DiscreteIQLLoss(*args, **kwargs)	TorchRL implementation of the discrete IQL loss.
CQLLoss(*args, **kwargs)	TorchRL implementation of the continuous CQL loss.
DiscreteCQLLoss(*args, **kwargs)	TorchRL implementation of the discrete CQL loss.

Parallel Q-Network lambda returns

DQNLoss supports the lambda-return target used by Parallel Q-Networks (PQN) through its existing value-estimator interface:

loss = DQNLoss(value_network, action_space=action_spec)
loss.make_value_estimator(
    ValueEstimators.TDLambda,
    gamma=0.99,
    lmbda=0.95,
)

The value network’s greedy action selection writes chosen_action_value, so the lambda return bootstraps from \(\max_a Q(s_{t+1}, a)\). It does not use the behavior action at the next step, and no ("next", "action") entry is required.