### Navigation

- index
- modules |
- next |
- previous |
- PyBrain v0.3 documentation »

# `learners` – RL Components: Learners

## Abstract classes

The top of the learner hierarchy is more conceptual than functional. The different classes distinguish algorithms in such a way that we can automatically determine when an algorithm is not applicable for a problem.

class `pybrain.rl.learners.learner.``Learner`

  Top-level class for all reinforcement learning algorithms. Any learning algorithm changes a policy (in some way) in order to increase the expected reward/fitness.

  `learn`()
    The main method, that invokes a learning step.

class `pybrain.rl.learners.learner.``EpisodicLearner`

  Bases: `pybrain.rl.learners.learner.Learner`

  Assumes the task is episodic, not life-long, and therefore does a learning step only after the end of each episode.

class `pybrain.rl.learners.learner.``DataSetLearner`

  Bases: `pybrain.rl.learners.learner.EpisodicLearner`

  A class for learners that learn from a dataset, which has no target output but only a reinforcement signal for each sample. It requires a ReinforcementDataSet object (which provides state-action-reward tuples).

class `pybrain.rl.learners.learner.``ExploringLearner`

  Bases: `pybrain.rl.learners.learner.Learner`

  A Learner determines how to change the adaptive parameters of a module.

class `pybrain.rl.learners.directsearch.directsearch.``DirectSearchLearner`

  Bases: `pybrain.rl.learners.learner.Learner`

  The class of learners that (in contrast to value-based learners) searches directly in policy space.

## Value-based Learners

class `pybrain.rl.learners.valuebased.valuebased.``ValueBasedLearner`

  Bases: `pybrain.rl.learners.learner.ExploringLearner`, `pybrain.rl.learners.learner.DataSetLearner`, `pybrain.rl.learners.learner.EpisodicLearner`

  An RL algorithm based on estimating a value-function.

  `batchMode`
    Does the algorithm run in batch mode or online?
  `explorer`
    Return the internal explorer.
  `module`
    Return the internal module.
  `offPolicy`
    Does the algorithm work on-policy or off-policy?

class `pybrain.rl.learners.valuebased.``Q`(alpha=0.5, gamma=0.98999999999999999)

  Bases: `pybrain.rl.learners.valuebased.valuebased.ValueBasedLearner`

  `learn`()

    Learn on the current dataset, either for many timesteps and even episodes (batchMode = True) or for a single timestep (batchMode = False). Batch mode is possible, because Q-Learning is an off-policy method.

    In batchMode, the algorithm goes through all the samples in the history and performs an update on each of them. if batchMode is False, only the last data sample is considered. The user himself has to make sure to keep the dataset consistent with the agent’s history.

class `pybrain.rl.learners.valuebased.``QLambda`(alpha=0.5, gamma=0.98999999999999999, qlambda=0.90000000000000002)

  Bases: `pybrain.rl.learners.valuebased.valuebased.ValueBasedLearner`

  Q-lambda is a variation of Q-learning that uses an eligibility trace.

class `pybrain.rl.learners.valuebased.``SARSA`(alpha=0.5, gamma=0.98999999999999999)

  Bases: `pybrain.rl.learners.valuebased.valuebased.ValueBasedLearner`

  State-Action-Reward-State-Action (SARSA) algorithm.

  In batchMode, the algorithm goes through all the samples in the history and performs an update on each of them. if batchMode is False, only the last data sample is considered. The user himself has to make sure to keep the dataset consistent with the agent’s history.

class `pybrain.rl.learners.valuebased.``NFQ`

  Bases: `pybrain.rl.learners.valuebased.valuebased.ValueBasedLearner`

  Neuro-fitted Q-learning

## Direct-search Learners

class `pybrain.rl.learners.directsearch.policygradient.``PolicyGradientLearner`

  Bases: `pybrain.rl.learners.directsearch.directsearch.DirectSearchLearner`, `pybrain.rl.learners.learner.DataSetLearner`, `pybrain.rl.learners.learner.ExploringLearner`

  PolicyGradientLearner is a super class for all continuous direct search algorithms that use the log likelihood of the executed action to update the weights. Subclasses are ENAC, GPOMDP, or REINFORCE.

  `learn`()
    calls the gradient calculation function and executes a step in direction of the gradient, scaled with a small learning rate alpha.

class `pybrain.rl.learners.directsearch.reinforce.``Reinforce`

  Bases: `pybrain.rl.learners.directsearch.policygradient.PolicyGradientLearner`

  Reinforce is a gradient estimator technique by Williams (see “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning”). It uses optimal baselines and calculates the gradient with the log likelihoods of the taken actions.

class `pybrain.rl.learners.directsearch.enac.``ENAC`

  Bases: `pybrain.rl.learners.directsearch.policygradient.PolicyGradientLearner`

  Episodic Natural Actor-Critic. See J. Peters “Natural Actor-Critic”, 2005. Estimates natural gradient with regression of log likelihoods to rewards.

Note

Black-box optimization algorithms can also be seen as direct-search RL algorithms, but are not included here.

### Navigation

- index
- modules |
- next |
- previous |
- PyBrain v0.3 documentation »