releso.agent.A2CAgent

class releso.agent.A2CAgent(*, save_location: Path, logger_name: str | None = None, tensorboard_log: str | None = None, policy: Literal['MlpPolicy', 'CnnPolicy', 'MultiInputPolicy'], use_custom_feature_extractor: Literal['resnet18', 'mobilenet_v2'] | None = None, cfe_without_linear: bool = False, policy_kwargs: Dict[str, Any] | None = None, type: Literal['A2C'], learning_rate: float = 0.0007, n_steps: int = 5, gamma: float = 0.99, gae_lambda: float = 1.0, ent_coef: float = 0.0, vf_coef: float = 0.5, seed: int | None = None, device: str = 'auto', max_grad_norm: float = 0.5, rms_prop_eps: float = 1e-05, use_rms_prop: bool = True, use_sde: bool = False, sde_sample_freq: int = -1, normalize_advantage: bool = False)

Bases: BaseTrainingAgent

A2c agent definition.

A2C definition for the stable_baselines3 implementation for this algorithm. Variable comments are taken from the stable_baselines3 documentation.

__init__(**data: Any) → None: Constructor for the ReLeSO basemodel object.

Methods

`convert_to_pathlib_add_datetime`(v)	Add timestamp to save_location, of applicable.
`get_additional_kwargs`(**kwargs)	Add additional keyword arguments for agent instantiation.
`get_agent`(environment[, normalizer_divisor])	Creates the stable_baselines version of the wanted Agent.
`get_logger`()	Gets the currently defined environment logger.
`get_next_tensorboard_experiment_name`()	Return tensorboard experiment name.
`set_logger_name_recursively`(logger_name)	Set the logger_name variable for all child elements.

Attributes

`agent_type`	What RL algorithm was used to train the agent.
`learning_rate`	The learning rate, it can be a function of the current progress remaining (from 1 to 0)
`n_steps`	The number of steps to run for each environment per update(i.e. rollout buffer size is n_steps * n_envs where n_envs is number of environment copies running in parallel) NOTE: n_steps * n_envs must be greater than 1 (because of the advantage normalization) See https://github.com/pytorch/pytorch/issues/29372.
`gamma`	Discount factor
`gae_lambda`	Factor for trade-off of bias vs variance for Generalized Advantage Estimator to classic advantage when set to 1
`ent_coef`	Entropy coefficient for the loss calculation
`vf_coef`	Value function coefficient for the loss calculation
`seed`	Seed for the pseudo random generators
`device`	Device (cpu, cuda, …) on which the code should be run.
`max_grad_norm`	The maximum value fot he gradient clipping
`rms_prop_eps`	RMSProp epsilon.
`use_rms_prop`	Whether to use RMSprop (default) or Adam as optimizer
`use_sde`	Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False)
`sde_sample_freq`	-1 (sample only at beginning of roll)
`normalize_advantage`	Whether to normalize or not the advantage
`policy`	policy defines the network structure which the agent uses
`use_custom_feature_extractor`	If given the str identifies the Custom Feature Extractor to be added.
`cfe_without_linear`	use the custom feature extractor with out a final linear layer
`policy_kwargs`	additional arguments to be passed to the policy on creation
`tensorboard_log`	base directory of the tensorboard logs if given an experiment name with a current timestamp is also added.
`save_location`	Definition of the save location of the logs and validation results.
`logger_name`	name of the logger.

agent_type: Literal['A2C']: What RL algorithm was used to train the agent. Needs to be know to correctly load the agent.

device: str: Device (cpu, cuda, …) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible.

ent_coef: float: Entropy coefficient for the loss calculation

gae_lambda: float: Factor for trade-off of bias vs variance for Generalized Advantage Estimator to classic advantage when set to 1

gamma: float: Discount factor

get_agent(environment: GymEnvironment, normalizer_divisor: int = 1) → A2C

Creates the stable_baselines version of the wanted Agent.

Uses all Variables given in the object (except agent_type) as the input parameters of the agent object creation.

Notes

The A2C variable n_steps scales with the amount of environments the agent is trained with. If this behavior is unwanted you can set the variable ‘’normalize_training_values’’ in BaseParser to true. This will set this variable to the number of environments so that the scaled values can be unscaled.

Parameters:

environment (GymEnvironment) – The environment the agent uses to train.
normalizer_divisor (int) – Divides the variable n_steps by this value to descale scaled values with n_environments unequal zero.

Returns:

Initialized A2C agent.

Return type:

A2C

learning_rate: float: The learning rate, it can be a function of the current progress remaining (from 1 to 0)

max_grad_norm: The maximum value fot he gradient clipping

n_steps: int: The number of steps to run for each environment per update(i.e. rollout buffer size is n_steps * n_envs where n_envs is number of environment copies running in parallel) NOTE: n_steps * n_envs must be greater than 1 (because of the advantage normalization) See https://github.com/pytorch/pytorch/issues/29372

normalize_advantage: Whether to normalize or not the advantage

rms_prop_eps: RMSProp epsilon. It stabilizes square root computation in denominator of of RMSProp update

sde_sample_freq

-1 (sample only at beginning of roll)

Type:: Sample a new noise matrix every n steps when using gSDE Default

seed: int | None: Seed for the pseudo random generators

use_rms_prop: Whether to use RMSprop (default) or Adam as optimizer

use_sde: Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False)

vf_coef: float: Value function coefficient for the loss calculation