releso.agent.DQNAgent

class releso.agent.DQNAgent(*, save_location: Path, logger_name: str | None = None, tensorboard_log: str | None = None, policy: Literal['MlpPolicy', 'CnnPolicy', 'MultiInputPolicy'], use_custom_feature_extractor: Literal['resnet18', 'mobilenet_v2'] | None = None, cfe_without_linear: bool = False, policy_kwargs: Dict[str, Any] | None = None, type: Literal['DQN'], learning_rate: float = 0.0001, buffer_size: int = 1000000, learning_starts: int = 256, batch_size: int | None = 32, tau: float = 1.0, gamma: float = 0.99, train_freq: int = 4, gradient_steps: int = 1, optimize_memory_usage: bool = False, target_update_interval: int = 256, exploration_fraction: float = 0.1, exploration_initial_eps: float = 1.0, exploration_final_eps: float = 0.05, max_grad_norm: float = 10, seed: int | None = None, device: str = 'auto')

Bases: BaseTrainingAgent

DQN Agent definition.

DQN definition for the stable_baselines3 implementation for this algorithm. Variable comments are taken from the stable_baselines3 documentation.

__init__(**data: Any) → None: Constructor for the ReLeSO basemodel object.

Methods

`convert_to_pathlib_add_datetime`(v)	Add timestamp to save_location, of applicable.
`get_additional_kwargs`(**kwargs)	Add additional keyword arguments for agent instantiation.
`get_agent`(environment[, normalizer_divisor])	Creates the stable_baselines version of the wanted Agent.
`get_logger`()	Gets the currently defined environment logger.
`get_next_tensorboard_experiment_name`()	Return tensorboard experiment name.
`set_logger_name_recursively`(logger_name)	Set the logger_name variable for all child elements.

Attributes

`agent_type`	What RL algorithm was used to train the agent.
`learning_rate`	The learning rate, it can be a function of the current progress remaining (from 1 to 0)
`buffer_size`	size of the replay buffer
`learning_starts`	how many steps of the model to collect transitions for before learning starts
`batch_size`	Minibatch size for each gradient update
`tau`	the soft update coefficient ("Polyak update", between 0 and 1) default 1 for hard update
`gamma`	the discount factor
`train_freq`	Update the model every `train_freq` steps.
`gradient_steps`	How many gradient steps to do after each rollout (see `train_freq`) Set to `-1` means to do as many gradient steps as steps done in the environment during the rollout.
`optimize_memory_usage`
`target_update_interval`	update the target network every `target_update_interval`
`exploration_fraction`	fraction of entire training period over which the exploration rate is reduced
`exploration_initial_eps`	initial value of random action probability
`exploration_final_eps`	final value of random action probability
`max_grad_norm`	The maximum value for the gradient clipping
`seed`	Seed for the pseudo random generators
`device`	Device (cpu, cuda, …) on which the code should be run.
`policy`	policy defines the network structure which the agent uses
`use_custom_feature_extractor`	If given the str identifies the Custom Feature Extractor to be added.
`cfe_without_linear`	use the custom feature extractor with out a final linear layer
`policy_kwargs`	additional arguments to be passed to the policy on creation
`tensorboard_log`	base directory of the tensorboard logs if given an experiment name with a current timestamp is also added.
`save_location`	Definition of the save location of the logs and validation results.
`logger_name`	name of the logger.

agent_type: Literal['DQN']: What RL algorithm was used to train the agent. Needs to be know to correctly load the agent.

batch_size: int | None: Minibatch size for each gradient update

buffer_size: int: size of the replay buffer

device: str: Device (cpu, cuda, …) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible.

exploration_final_eps: float: final value of random action probability

exploration_fraction: float: fraction of entire training period over which the exploration rate is reduced

exploration_initial_eps: float: initial value of random action probability

gamma: float: the discount factor

get_agent(environment: GymEnvironment, normalizer_divisor: int = 1) → DQN

Creates the stable_baselines version of the wanted Agent.

Uses all Variables given in the object (except agent_type) as the input parameters of the agent object creation.

Parameters:

environment (GymEnvironment) – The environment the agent uses to
train.
normalizer_divisor (int) – Currently not used in this function.

Returns:

Initialized DQN agent.

Return type:

DQN

gradient_steps: int: How many gradient steps to do after each rollout (see train_freq) Set to -1 means to do as many gradient steps as steps done in the environment during the rollout.

learning_rate: float: The learning rate, it can be a function of the current progress remaining (from 1 to 0)

learning_starts: int: how many steps of the model to collect transitions for before learning starts

max_grad_norm: float: The maximum value for the gradient clipping

seed: int | None: Seed for the pseudo random generators

target_update_interval: int: update the target network every target_update_interval

tau: float: the soft update coefficient (“Polyak update”, between 0 and 1) default 1 for hard update

train_freq: int: Update the model every train_freq steps.