It's no difference with the previous version because self(batch).act is not used in learn(), since eps only changes act.
But pointing it out is much better.

Reshape is good, and would be better if you can avoid shape [bsz] by all means (reward has this shape) because of the confusing behavior.

Reshape is good, and would be better if you can avoid shape [bsz] by all means (reward has this shape) because of the confusing behavior.

I don't agree, usually I prefer to use .flatten(1, -1) since it makes it clear that you want to flatten but preserve batch processing.

Reshape is good, and would be better if you can avoid shape [bsz] by all means (reward has this shape) because of the confusing behavior.

I don't agree, usually I prefer to use .flatten(1, -1) since it makes it clear that you want to flatten but preserve batch processing.

But 1-dim tensor cannot apply flatten(1):

In [6]: b=torch.rand(3)

In [7]: b.flatten(1,-1)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-7-0fd10ca82bb9> in <module>
----> 1 b.flatten(1,-1)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

But 1-dim tensor cannot apply flatten(1):

OK so your example before edit "[bsz, ?] or [bsz, ?, ?]" was wrong. So yes if [bsz] can happen (which I highly doubt, since it means 0-dim input without batch-processing) then reshape must be used.

if [bsz] can happen (which I highly doubt, since it means 0-dim input without batch-processing)

reward is the case.

reward is the case.

The reward is used as input of its own neural network ? In which algorithm ?

The reward is used as input of its own neural network

I'm not talking about using reward as input, though. My suggestion is to reshape reward to be [bsz, 1] in Policy.learn :)

The reward is used as input of its own neural network

I'm not talking about using reward as input, though. My suggestion is to reshape reward to be [bsz, 1] in Policy.learn :)

I don't agree to reshape this as [bsz, 1] since it had already made a bug previously.

I'm not talking about using reward as input, though. My suggestion is to reshape reward to be [bsz, 1] in Policy.learn :)

We are talking about this initially.

Okay. This is because I write a small test with MyTestEnv under the test/base/ folder. I found that the obs in this env cannot be processed through the current network (because its shape [bsz]), so I modify this line of code.
I know it rarely happens :)