bevy_light_field 💡🌾📷

rust bevy light field camera array tooling

capabilities

• grid view of light field camera array
• stream to files with recording controls
• person segmentation post-process (batch across streams)
• async segmentation model inference
• foreground extraction post-process and visualization mode
• camera array calibration (extrinsics, intrinsics, color)
• camera position visualization
• playback nersemble recordings with annotations
• 3d reconstruction dataset preparation
• real-time 3d reconstruction viewer

run the viewer

cargo run -- --help

the viewer opens a window and displays the light field camera array, with post-process options

see execution provider bevy_ort documentation for better performance

• windows: cargo run --release --features "ort/cuda"

controls

• r to start recording
• s to stop recording
• esc to exit
• UI controls

library usage

use bevy::{
    prelude::*,
    render::render_resource::{
        Extent3d,
        TextureDescriptor,
        TextureDimension,
        TextureFormat,
        TextureUsages,
    },
};

use bevy_light_field::stream::{
    RtspStreamDescriptor,
    RtspStreamPlugin,
    StreamId,
};


const RTSP_URIS: [&str; 1] = [
    "rtsp://localhost:554/lizard",
];


fn main() {
    App::new()
        .add_plugins((
            DefaultPlugins,
            RtspStreamPlugin,
        ))
        .add_systems(Startup, create_streams)
        .run();
}


fn create_streams(
    mut commands: Commands,
    mut images: ResMut<Assets<Image>>,
) {
    RTSP_URIS.iter()
        .enumerate()
        .for_each(|(index, &url)| {
            let entity = commands.spawn_empty().id();

            let size = Extent3d {
                width: 640,
                height: 360,
                ..default()
            };

            let mut image = Image {
                texture_descriptor: TextureDescriptor {
                    label: Some(url),
                    size,
                    dimension: TextureDimension::D2,
                    format: TextureFormat::Rgba8UnormSrgb,
                    mip_level_count: 1,
                    sample_count: 1,
                    usage: TextureUsages::COPY_DST
                        | TextureUsages::TEXTURE_BINDING
                        | TextureUsages::RENDER_ATTACHMENT,
                    view_formats: &[TextureFormat::Rgba8UnormSrgb],
                },
                ..default()
            };
            image.resize(size);

            let image = images.add(image);
            commands.spawn(SpriteBundle {
                sprite: Sprite {
                    custom_size: Some(Vec2::new(640.0, 360.0)),
                    ..default()
                },
                texture: image.clone(),
                ..default()
            });

            let rtsp_stream = RtspStreamDescriptor::new(
                url.to_string(),
                StreamId(index),
                image,
            );

            commands.entity(entity).insert(rtsp_stream);
        });
}

light field camera array

view the onshape model

• parts list

setup rtsp streaming server

it is useful to test the light field viewer with emulated camera streams

obs studio

• install https://obsproject.com/
• install rtsp plugin https://github.com/iamscottxu/obs-rtspserver/releases
• tools > rtsp server > start server

compatible bevy versions

bevy_light_field	bevy
0.1.0	0.13

credits

• bevy_video
• gaussian_avatars
• modnet
• nersemble
• paddle_seg_matting
• ray diffusion