releso.agent.SACAgent

class releso.agent.SACAgent(*, save_location: Path, logger_name: str | None = None, tensorboard_log: str | None = None, policy: Literal['MlpPolicy', 'CnnPolicy', 'MultiInputPolicy'], use_custom_feature_extractor: Literal['resnet18', 'mobilenet_v2'] | None = None, cfe_without_linear: bool = False, policy_kwargs: Dict[str, Any] | None = None, type: Literal['SAC'], learning_rate: float = 0.001, buffer_size: int = 1000000, learning_starts: int = 100, batch_size: int | None = 64, tau: float = 0.005, gamma: float = 0.99, optimize_memory_usage: bool = False, ent_coef: str | float = 'auto', target_update_interval: int = 1, target_entropy: str | float = 'auto', use_sde: bool = False, sde_sample_freq: int = -1, use_sde_at_warmup: bool = False, seed: int | None = None, device: str = 'auto')

Bases: BaseTrainingAgent

SAC Agent definition.

SAC definition for the stable_baselines3 implementation for this algorithm. Variable comments are taken from the stable_baselines3 documentation.

__init__(**data: Any) → None: Constructor for the ReLeSO basemodel object.

Methods

`convert_to_pathlib_add_datetime`(v)	Add timestamp to save_location, of applicable.
`get_additional_kwargs`(**kwargs)	Add additional keyword arguments for agent instantiation.
`get_agent`(environment[, normalizer_divisor])	Creates the stable_baselines version of the wanted Agent.
`get_logger`()	Gets the currently defined environment logger.
`get_next_tensorboard_experiment_name`()	Return tensorboard experiment name.
`set_logger_name_recursively`(logger_name)	Set the logger_name variable for all child elements.

Attributes

`agent_type`	What RL algorithm was used to train the agent.
`learning_rate`	The learning rate, it can be a function of the current progress remaining (from 1 to 0)
`buffer_size`	size of the replay buffer
`learning_starts`	how many steps of the model to collect transitions for before learning starts
`batch_size`	Minibatch size
`tau`	the soft update coefficient ("Polyak update", between 0 and 1)
`gamma`	Discount factor
`optimize_memory_usage`
`ent_coef`	Entropy regularization coefficient.
`target_update_interval`	update the target network every `target_network_update_freq` gradient steps.
`target_entropy`	target entropy when learning `ent_coef` (`ent_coef = 'auto'`)
`use_sde`	Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False)
`sde_sample_freq`	-1 (only sample at the beginning of the rollout)
`use_sde_at_warmup`	Whether to use gSDE instead of uniform sampling during the warm up phase (before learning starts)
`seed`	Seed for the pseudo random generators
`device`	Device (cpu, cuda, …) on which the code should be run.
`policy`	policy defines the network structure which the agent uses
`use_custom_feature_extractor`	If given the str identifies the Custom Feature Extractor to be added.
`cfe_without_linear`	use the custom feature extractor with out a final linear layer
`policy_kwargs`	additional arguments to be passed to the policy on creation
`tensorboard_log`	base directory of the tensorboard logs if given an experiment name with a current timestamp is also added.
`save_location`	Definition of the save location of the logs and validation results.
`logger_name`	name of the logger.

agent_type: Literal['SAC']: What RL algorithm was used to train the agent. Needs to be know to correctly load the agent.

batch_size: int | None: Minibatch size

buffer_size: int: size of the replay buffer

device: str: Device (cpu, cuda, …) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible.

ent_coef: str | float: Entropy regularization coefficient. (Equivalent to inverse of reward scale in the original SAC paper.) Controlling exploration/exploitation trade-off. Set it to ‘auto’ to learn it automatically (and ‘auto_0.1’ for using 0.1 as initial value)

gamma: float: Discount factor

get_agent(environment: GymEnvironment, normalizer_divisor: int = 1) → SAC

Creates the stable_baselines version of the wanted Agent.

Uses all Variables given in the object (except agent_type) as the input parameters of the agent object creation.

Parameters:

environment (GymEnvironment) – The environment the agent uses to
train.
normalizer_divisor (int) – Currently not used in this function.

Returns:

Initialized SAC agent.

Return type:

SAC

learning_rate: float: The learning rate, it can be a function of the current progress remaining (from 1 to 0)

learning_starts: int: how many steps of the model to collect transitions for before learning starts

sde_sample_freq: int

-1 (only sample at the beginning of the rollout)

Type:: Sample a new noise matrix every n steps when using gSDE Default

seed: int | None: Seed for the pseudo random generators

target_entropy: str | float: target entropy when learning ent_coef (ent_coef = 'auto')

target_update_interval: int: update the target network every target_network_update_freq gradient steps.

tau: float: the soft update coefficient (“Polyak update”, between 0 and 1)

use_sde: bool: Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False)

use_sde_at_warmup: bool: Whether to use gSDE instead of uniform sampling during the warm up phase (before learning starts)