Gaussian Relighting Support #200
Replies: 3 comments 5 replies
-
|
This would be really cool. For the proxy mesh route, I think that the API should expose the proxy mesh so that it can be set manually. For example, when doing a mesh-to-cloud conversion, one would likely want to use the converted mesh directly as the proxy mesh for lighting, so that the mesh generation could be skipped. |
Beta Was this translation helpful? Give feedback.
-
|
some initial sketching here (doesn't function yet, feel free to add to it): #201 I like the gaussian -> proxy mesh idea, it is useful for lighting, shadows, morph targets, and skeletons. I think there is a quality challenge in 3dgs -> mesh extraction. the proposed sketch api should support manual proxy mesh by adding the proxy mesh component directly. there is also work required to extract the proper material/textures for the proxy mesh. e.g. how to extract PBR planes from spherical harmonics, map to texture(s), etc. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Do you all think that heuristic-based approaches are a dead-end for achieving high-quality Gaussian relighting? Most of these approaches require the initial training images used to generate the gaussians. |
Beta Was this translation helpful? Give feedback.
-
There's a lot of edge cases there, right? Like, will it work with multiple gaussian clouds composited together? It also obviously won't work if the training images aren't available or if they never existed, like in generative use cases. |
Beta Was this translation helpful? Give feedback.
-
|
Would a |
Beta Was this translation helpful? Give feedback.
-
|
I like the idea of porting a lot of the capability/examples pioneered in spark. agent workflows keeping capabilities in sync would be really neat. most of it should be straightforward, there is some improvement that can be made in bevy_gaussian_splatting in the multi-cloud handling, e.g. spark aggregates all gaussians in the world where bevy_gaussian_splatting runs indirect drawing per cloud (relatively minor compute pipeline change). |
Beta Was this translation helpful? Give feedback.
-
Sounds good! I’m happy to help. If you’d like to create a GitHub project timeline to convert bevy_gaussian_splatting to be 1-to-1 compatible with Spark, I’d love to help and pick up some of the tasks on that roadmap. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Any thoughts on how to enable gaussian relighting using bevy_solari?
I would love if most of the raytracing / global illumination logic to handled by solari as this crate wouldn't have to maintain more code than it needs to but from some of the research papers i've skimmed through the most direct, possibly non optimal, method might be to create a proxy mesh.
Beta Was this translation helpful? Give feedback.
All reactions