There is a bunch of reported RNN problems: #795 #513 #486 and some others. Generally, performance of agents with RNN on top never seemed to be good. This should be looked into, compared with other libraries, and fixed.

@Trinkle23897 @araffin @BFAnas If you have ideas where to start or whom to ping, feel free to write here. I guess it won't be an easy one...

Until this is solved, it might make sense to document that using RNNs is currently discouraged.