It is true that timeout should be supported for dynamic environments. In fact, action distribution depends on policy, which may change the time cost distribution of step functions.

I would add that in m'y opinion it is usually more common to deal with such dynamic environment than a static one. For example in my case I have to deal with dynamic environments, never static. It comes from the way the state is integrated over time. Some state requires to refine the computation to guarantee the accuracy, while for others it is not necessary.

Is it ready for review ? I didn't spent much time thinking to a better way to handle methods such as wait to avoid relying on static methods of the workers themself. I'm suite confident that using metaclass (in a way similar to metaprograming in c++) could do the trick but it is tricky to design and lack for clarity even for devs. So I don't know what to think about this.

Is it ready for review?

Yes, of course!

And I have another concern: can we reduce the async overhead as much as possible? I tested with wait_num == env_num for both sync and async mode (use test_drqn.py), and the result shows that the speed of async is 80% of sync. This could be the future work.

I would add that in m'y opinion it is usually more common to deal with such dynamic environment than a static one.

Agree. Only toy environments are static. Almost all realistic environments are dynamic.

I'll have a look later today

As I said, I should be clear from the documentation that the vector env are not usual pool, a sense that it does not really handle parallelism directly. Being asynchronous or not is to the resort of the sole worker and at its discretion, with no explicit mechanism for the pool to know this information in advance (hence the unusual wait mechanism).

Being asynchronous or not is to the resort of the sole worker and at its discretion, with no explicit mechanism for the pool to know this information in advance

I don't agree. The workers have send_action / get_results interfaces so that they support asynchronous simulation. It is up to the user / vec env whether to use asynchronous or not. I don't think "the unusual wait mechanism" exists.

The workers have send_action / get_results interfaces so that they support asynchronous simulation. It is up to the user / vec env whether to use asynchronous or not.

No it is not. It requires the user to use a worker that support async behavior. It is not asynchronous by itself.

I don't think "the unusual wait mechanism" exists.

Are you kidding me ? Show me a single well-known library using the same "calling static method from workers" strategy to implement wait method.

Are you kidding me ? Show me a single well-known library using the same "calling static method from workers" strategy to implement wait method.

I admit the current implementation requires the hack of calling static method from workers.

To remove the hack, each worker should implement a ready interface, which returns True if it is ready to return results. But I don't know how to implement it for ray, and I think it would be difficult to implement. Therefore, I rely on the readily implemented connection.wait and ray.wait.

The ready interface enables waiting by while-loop. An efficient wait should rely on something like semaphore, i.e. sleep on a set of semaphores. But I doubt something general like sleeping on a set of semaphores exists. Most likely, sleeping on some signal is worker-dependent.

I agree it is tricky to implement and I'm fine with the current implementation. It could be refactor later if necessary, or if we have time for that.