This is a prerequisite for #134 .

Collector.collect should only reflect the stat of current call. If users want to track the stat (such as moving average) of Collector.collect, they can achieve it with minimal efforts, but this should not be done by Tianshou anyway. So I commited d3e23ac .

To me it is a great improvement. Reducing the number of branches is not easy but often it makes everything simpler and clearer. And especially this part of the code was unnecessarily complicated and hard to understand for no reason (probably an artefact of the past), since a single env is nothing conceptually different from a vector venv with a single instance. Good you took the time to clean that part of the code !

And especially this part of the code was unnecessarily complicated and hard to understand for no reason (probably an artefact of the past), since a single env is nothing conceptually different from a vector venv with a single instance.

I realized this when I want to support sync simulation, and get support from @Trinkle23897 . It is indeed an artefact of the past because he wants to support partial episodes, but later he realized that it is hardly necessary, so I removed this part of code.

ready for review now @Trinkle23897 @duburcqa

According to my experiments:
Even though I change the step_number within an episode (e.g. There is a 30 steps episode, within which I update the net every 5, 10, or 15 steps), the result may be different.
I do not understand. What is the reason for 'he realized that it is hardly necessary'?

We do some experiments on Bipedal-walker and Lunar-lander, the final result seems slightly different with each other. @imoneoi

Even though I change the step_number within an episode (e.g. There is a 30 steps episode, within which I update the net every 5, 10, or 15 steps), the result may be different.

The point of this pr is to enforce that there will only be full episodes in the replay buffer. If fully online learning really matters for you, I think you can use a single environment and try to write a collector with simple for-loops.

for xxx:
    data = []
    while len(data) < n:
        data.append(env.step())
    policy.learn(data)

In my opinion, if you need a replay buffer, fully online learning (learning after every several steps) will not matter so much. Current implementation (Tianshou 0.2.5) allows users to perform online learning at the episode level (learning after every several episodes) and I think it is enough.

However, feel free to provide feedback and to discuss :)