# rcognita.controllers.CtrlRLStab

class rcognita.controllers.CtrlRLStab(dim_input, dim_output, mode='JACS', ctrl_bnds=[], action_init=[], t0=0, sampling_time=0.1, Nactor=1, pred_step_size=0.1, sys_rhs=[], sys_out=[], state_sys=[], prob_noise_pow=1, is_est_model=0, model_est_stage=1, model_est_period=0.1, buffer_size=20, model_order=3, model_est_checks=0, gamma=1, Ncritic=4, critic_period=0.1, critic_struct='quad-nomix', actor_struct='quad-nomix', stage_obj_struct='quadratic', stage_obj_pars=[], observation_target=[], safe_ctrl=[], safe_decay_rate=[])

  Class of reinforcement learning agents with stabilizing constraints.

  Sampling here is similar to the predictive controller agent `CtrlOptPred`

  Needs a nominal controller object `safe_ctrl` with a respective Lyapunov function.

  `w_actor` : weights.

  Feature structure is defined via a string flag `actor_struct`. Read more on features in class description of `controllers.CtrlOptPred`.

  `w_critic` : weights.

  Feature structure is defined via a string flag `critic_struct`. Read more on features in class description of `controllers.CtrlOptPred`.

  mode

    Controller mode. Currently available only JACS, joint actor-critic (stabilizing).

    Type

      : string

  Read more
  ---------
  Osinenko, P., Beckenbach, L., Göhrt, T., & Streif, S. (2020). A reinforcement learning method with closed-loop stability guarantee. IFAC-PapersOnLine
  __init__(dim_input, dim_output, mode='JACS', ctrl_bnds=[], action_init=[], t0=0, sampling_time=0.1, Nactor=1, pred_step_size=0.1, sys_rhs=[], sys_out=[], state_sys=[], prob_noise_pow=1, is_est_model=0, model_est_stage=1, model_est_period=0.1, buffer_size=20, model_order=3, model_est_checks=0, gamma=1, Ncritic=4, critic_period=0.1, critic_struct='quad-nomix', actor_struct='quad-nomix', stage_obj_struct='quadratic', stage_obj_pars=[], observation_target=[], safe_ctrl=[], safe_decay_rate=[])

    Parameter specification largely resembles that of `CtrlOptPred` class.

    Parameters

      - dim_input (: integer) – Dimension of input and output which should comply with the system-to-be-controlled.

      - dim_output (: integer) – Dimension of input and output which should comply with the system-to-be-controlled.

      - ctrl_bnds (: array of shape `[dim_input, 2]`) – Box control constraints. First element in each row is the lower bound, the second - the upper bound. If empty, control is unconstrained (default).

      - action_init (: array of shape `[dim_input, ]`) – Initial action to initialize optimizers.

      - t0 (: number) – Initial value of the controller’s internal clock.

      - sampling_time (: number) – Controller’s sampling time (in seconds).

      - sys_rhs (: functions) – Functions that represent the right-hand side, resp., the output of the exogenously passed model. The latter could be, for instance, the true model of the system. In turn, `state_sys` represents the (true) current state of the system and should be updated accordingly. Parameters `sys_rhs, sys_out, state_sys` are used in those controller modes which rely on them.

      - sys_out (: functions) – Functions that represent the right-hand side, resp., the output of the exogenously passed model. The latter could be, for instance, the true model of the system. In turn, `state_sys` represents the (true) current state of the system and should be updated accordingly. Parameters `sys_rhs, sys_out, state_sys` are used in those controller modes which rely on them.

      - prob_noise_pow (: number) – Power of probing noise during an initial phase to fill the estimator’s buffer before applying optimal control.

      - is_est_model (: number) – Flag whether to estimate a system model. See `_estimate_model()`.

      - model_est_stage (: number) – Initial time segment to fill the estimator’s buffer before applying optimal control (in seconds).

      - model_est_period (: number) – Time between model estimate updates (in seconds).

      - buffer_size (: natural number) – Size of the buffer to store data.

      - model_order (: natural number) –

        Order of the state-space estimation model

        \[\begin{array}{ll} \hat x^+ & = A \hat x + B action, \newline observation^+ & = C \hat x + D action. \end{array}\]

        See `_estimate_model()`. This is just a particular model estimator. When customizing, `_estimate_model()` may be changed and in turn the parameter `model_order` also. For instance, you might want to use an artifial neural net and specify its layers and numbers of neurons, in which case `model_order` could be substituted for, say, `Nlayers`, `Nneurons`.

      - model_est_checks (: natural number) – Estimated model parameters can be stored in stacks and the best among the `model_est_checks` last ones is picked. May improve the prediction quality somewhat.

      - gamma (: number in (0, 1]) – Discounting factor. Characterizes fading of stage objectives along horizon.

      - Ncritic (: natural number) – Critic stack size \(N_c\). The critic optimizes the temporal error which is a measure of critic’s ability to capture the optimal infinite-horizon objective (a.k.a. the value function). The temporal errors are stacked up using the said buffer.

      - critic_period (: number) – The same meaning as ``model_est_period`.`

      - critic_struct (: string) –

        Choice of the structure of the critic’s and actor’s features.

        Currently available:

        Feature structures
        Mode | Structure
        --- | ---
        ’quad-lin’ | Quadratic-linear
        ’quadratic’ | Quadratic
        ’quad-nomix’ | Quadratic, no mixed terms

        Add your specification into the table when customizing the actor and critic.

      - actor_struct (: string) –

        Choice of the structure of the critic’s and actor’s features.

        Currently available:

        Feature structures
        Mode | Structure
        --- | ---
        ’quad-lin’ | Quadratic-linear
        ’quadratic’ | Quadratic
        ’quad-nomix’ | Quadratic, no mixed terms

        Add your specification into the table when customizing the actor and critic.

      - stage_obj_struct (: string) –

        Choice of the stage objective structure.

        Currently available:

        Running objective structures
        Mode | Structure
        --- | ---
        ’quadratic’ | Quadratic \(\chi^\top R_1 \chi\), where \(\chi = [observation, action]\), `stage_obj_pars` should be `[R1]`
        ’biquadratic’ | 4th order \(\left( \chi^\top \right)^2 R_2 \left( \chi \right)^2 + \chi^\top R_1 \chi\), where \(\chi = [observation, action]\), `stage_obj_pars` should be `[R1, R2]`

  Methods

  `__init__`(dim_input, dim_output[, mode, …]) | Parameter specification largely resembles that of `CtrlOptPred` class.
  `compute_action`(t, observation) |
  `receive_sys_state`(state) | Fetch exogenous model state.
  `reset`(t0) | Resets agent for use in multi-episode simulation.
  `stage_obj`(observation, action) | Stage (equivalently, instantaneous or running) objective.
  `upd_accum_obj`(observation, action) | Sample-to-sample accumulated (summed up or integrated) stage objective.