🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2859

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

In this PR, I add an example to show how to sync weights when the native weight update class doesn't work (eg serializing mps weights is not possible so you can cast them to cpu then recast to mps)
I also fix the collectors and test the solution.
This should also make other update methods easier to implement.

cc @mikaylagawarecki

We also need to find a way to pass the local update to the sub-collector. With the distributed ones it's easy to do (we can pass any kwarg). For mp ones not so much atm