Replies: 1 comment 2 replies
-
|
Hi @jhartman, thanks for your interest in Horovod and sorry for taking a while to answer. As of now you can indeed build tensor slicing yourself with Horovod 0.23, which introduced the process sets that I alluded to in that old answer. Basically, Horovod process sets allow you to run different collective operations (such as your gradient Allreduce) on different subset of ranks, concurrently. Previously, not being able to do so was a blocker for model parallelism in Horovod. Here is some intro documentation: https://horovod.readthedocs.io/en/stable/process_set_include.html Pipelining is also something that can be built in user code and I think process sets should also help you out here for efficient implementations. However, for now Horovod does not provide these high level features itself. I can imagine that contributions would be quite welcome here. What do you think, @tgaddair? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @maxhgerlach . I really appreciate the answer. I think a high-level feature would be really helpful. I can present this to my team as to whether we can help. Would you be willing to put together an example of how pipelining could be done in user code in TF or PyTorch? This isn't obvious to me. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I don't have any pipelining code that I would be able to share. But there are several open source repositories available, like GPipe (+ re-implementations) and PipeDream. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi folks,
Are there plans to support tensor slicing and pipeline parallelism? Microsoft and NVidia recently released a blog post about a collaboration to train a 500B parameter model. Are there plans to do the same within Horovod? Horovod is widely used so having this functionality would enable more people to work and research with larger models.
I noticed this discussion that seems to imply similar functionality coming but I didn't get a sense of :
If help is needed, I think some devs on my team would be interested in partnering next year.
Thanks!
Josh
Beta Was this translation helpful? Give feedback.
All reactions