+

US20190364264A1 - Method and system for unsynchronized structured lighting - Google Patents

Method and system for unsynchronized structured lighting Download PDF

Info

Publication number
US20190364264A1
US20190364264A1 US16/434,846 US201916434846A US2019364264A1 US 20190364264 A1 US20190364264 A1 US 20190364264A1 US 201916434846 A US201916434846 A US 201916434846A US 2019364264 A1 US2019364264 A1 US 2019364264A1
Authority
US
United States
Prior art keywords
pixel
image
image frames
synchronized
sequence
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US16/434,846
Inventor
Gabriel Taubin
Daniel Moreno
Fatih Calakli
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Brown University
Original Assignee
Brown University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Brown University filed Critical Brown University
Priority to US16/434,846 priority Critical patent/US20190364264A1/en
Publication of US20190364264A1 publication Critical patent/US20190364264A1/en
Assigned to UNITED STATES DEPARTMENT OF ENERGY reassignment UNITED STATES DEPARTMENT OF ENERGY CONFIRMATORY LICENSE (SEE DOCUMENT FOR DETAILS). Assignors: BROWN UNIVERSITY
Priority to US17/226,920 priority patent/US11425358B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/204Image signal generators using stereoscopic image cameras
    • H04N13/254Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03BAPPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B35/00Stereoscopic photography
    • G03B35/02Stereoscopic photography by sequential recording
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • G06T7/521Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106Processing image signals
    • H04N13/167Synchronising or controlling image signals
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10028Range image; Depth image; 3D point clouds

Definitions

  • This patent document relates generally to the field of three-dimensional shape capture of the surface geometry of an object, and more particularly to structured lighting three-dimensional shape capture.
  • Three-dimensional scanning and digitization of the surface geometry of objects is commonly used in many industries and services, and their applications are numerous.
  • a few examples of such applications are inspection and measurement of shape conformity in industrial production systems, digitization of clay models for industrial design and styling applications, reverse engineering of existing parts with complex geometry for three-dimensional printing, interactive visualization of objects in multimedia applications, three-dimensional documentation of artwork, historic and archaeological artifacts, human body scanning for better orthotics adaptation, biometry or custom-fit clothing, and three-dimensional forensic reconstruction of crime scenes.
  • Three dimensional shape capture systems based on structure lighting are more accurate than those based on time-of-flight (TOF) image sensors.
  • TOF time-of-flight
  • a pattern projector is used to illuminate the scene of interest with a sequence of known two-dimensional patterns, and a camera is used to capture a sequence of images, synchronized with the projected patterns. The camera captures one image for each projected pattern.
  • Each sequence of images captured by the camera is decoded by a computer processor into a dense set of projector-camera pixel correspondences, and subsequently into a three-dimensional range image, using the principles of optical triangulation.
  • the main limitation of three-dimensional shape capture systems is the required synchronization between projector and camera.
  • the sequence of patterns must be projected at a fast rate, the camera must capture image frames exactly at the same frame rate, and the camera has to start capturing the first frame of the sequence exactly when the projector starts to project the first pattern.
  • Rolling shutter is a method of image capture in which a still picture or each frame of a video is captured not by taking a snapshot of the entire scene at single instant in time but rather by scanning across the scene rapidly, either vertically or horizontally. In other words, not all parts of the image of the scene are recorded at exactly the same instant. This is in contrast with “global shutter” in which the entire frame is captured at the same instant.
  • global shutter Even though most image sensors in consumer devices are rolling shutter sensors, many image sensors used in industrial applications are global shutter sensors.
  • a system and method to capture the surface geometry a three-dimensional object in a scene using unsynchronized structured lighting solves the problems of the prior art.
  • the method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate; and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns. Because the method enables use of an unsynchronized pattern projector and camera significant cost savings can be achieved.
  • the method enables use of inexpensive cameras, such as smartphone cameras, webcams, point-and-shoot digital cameras, camcorders as well as industrial cameras. Furthermore, the method and system enable processing the images with a variety of computing hardware, such as computers, digital signal processors, smartphone processors and the like. Consequently, three-dimensional image capture using structured lighting may be used with relatively little capital investment.
  • FIG. 1 is an illustration of an exemplary embodiment of the method and system for unsynchronized structured lighting
  • FIG. 2 shows a flowchart of an exemplary embodiment of the method and system for unsynchronized structured lighting
  • FIG. 3 shows a flowchart of an exemplary embodiment of the method and system for unsynchronized structured lighting of synthesizing the synchronized sequence of image frames
  • FIG. 4 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a global shutter image sensor is used and where the image frame rate is identical to the pattern frame rate;
  • FIG. 5 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a rolling shutter image sensor is used and where the image frame rate is identical to the pattern frame rate;
  • FIG. 6 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a global shutter image sensor is used and where the image frame rate is higher or equal than the pattern frame rate
  • FIG. 7 is a chart illustrating the timing for method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a rolling shutter image sensor is used and where the image frame rate is higher or equal than the pattern frame rate;
  • FIG. 8 shows a chart illustrating a calibration pattern used to correct the time
  • FIG. 9 shows a chart illustrating image data normalized according to the method described herein.
  • FIG. 10 shows a chart illustrating a model estimating the pattern value for each pixel in a captured image.
  • FIGS. 1 and 2 A system and method to capture the surface geometry a three-dimensional object in a scene using unsynchronized structured lighting is shown generally in FIGS. 1 and 2 .
  • the method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate; and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns.
  • One object of the present invention is a system to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames, illustrated in FIG. 1 ; the unsynchronized image frames captured while a three-dimensional scene was illuminated by a sequence of patterns.
  • the system comprises a pattern projector 2020 and a camera 2030 .
  • An object 2010 is partially illuminated by the pattern projector 2020 , and partially visible by the camera 2030 .
  • the pattern projector projects a sequence of patterns 2021 , 2022 , 2023 at a certain pattern rate.
  • the pattern rate is measured in patterns per second.
  • the camera 2030 captures a sequence of unsynchronized image frames 2031 , 2032 , 2033 , 2034 , at a certain frame rate.
  • the frame rate is larger or equal than the pattern rate.
  • the number of unsynchronized image frames captured by the camera is larger or equal than the number of patterns in the sequence of patterns.
  • the camera starts capturing the first unsynchronized image frame not earlier than the time when the projector starts projecting the first pattern of the sequence of image patterns.
  • the camera ends capturing the last unsynchronized image frame not earlier than the time when the projector ends projecting the last pattern of the sequence of image patterns.
  • To capture a frame the camera opens the camera aperture, which it closes after an exposure time.
  • the camera determines the pixel values by integrating the incoming light while the aperture is open. Since camera and projector are not synchronized, the projector may switch patterns while the camera has the aperture open.
  • the pixels of the captured unsynchronized image frame 2032 will be partially exposed to two consecutive patterns 2021 and 2022 .
  • the resulting sequence of unsynchronized image frames are transmitted to a computer processor 2040 , which executes a method to synthesize a synchronized sequence of image frames from the unsynchronized sequence of image frames.
  • the number of frames in the synchronized sequence of image frames 2041 , 2042 , 2043 will be the same as the number of patterns 2021 , 2022 , 2023 and represent estimates of what the camera would have captured if it were synchronized with the projector.
  • the camera 2030 and the computer processor 2040 are components of a single device such a digital camera, smartphone, or computer tablet.
  • Another object of the invention is an unsynchronized three-dimensional shape capture system, comprising the system to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames described above, and further comprising prior art methods for decoding, three-dimensional triangulation, and optionally geometric processing, executed by the computer processor.
  • Another object of the invention is a three-dimensional snapshot camera comprising the unsynchronized three-dimensional shape capture system, where the projector has the means to select the pattern rate from a plurality of supported pattern rates, the camera has the means to select the frame rate from a plurality of supported frame rates, and the camera is capable of capturing the unsynchronized image frames in burst mode at a fast frame rate.
  • the projector has a knob to select the pattern rate.
  • the pattern rate is set by a pattern rate code sent to the projector through a communications link.
  • the system has means to set the pattern rate and the frame rate so that the frame rate is not slower than the pattern rate.
  • the user sets the pattern rate and the frame rate.
  • the camera has the means to receive a camera trigger signal, and the means to set the number of burst mode frames.
  • the camera trigger signal is generated by a camera trigger push-button. When the camera receives the trigger signal it starts capturing the unsynchronized image frames at the set frame rate, and it stops capturing unsynchronized image frames after capturing the set number of burst mode frames.
  • the projector continuously projects the sequence of patterns in a cyclic fashion.
  • the system has the means of detecting when the first pattern is about to be projected, and the camera trigger signal is delayed until that moment.
  • the projector has the means to receive a projector trigger signal.
  • the camera generates the projector trigger signal after receiving the camera trigger signal, and the camera has the means to send the projector trigger signal to the projector.
  • the camera has a flash trigger output, and it sends the projector trigger signal to the projector through the flash trigger output.
  • Another object of this invention is a method to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames, generating a number of frames in the synchronized sequence of image frames equal to the number of projected patterns, and representing estimates of what the camera would have captured if it were synchronized with the projector.
  • each of the synchronized image frames corresponds to one of a sequence of image patterns. Further, the synchronized image frames and unsynchronized image frames have the same width and height.
  • Each image frame includes a plurality of common pixels.
  • the plurality of common pixels is partitioned into a pixel partition.
  • the pixel partition includes a plurality of pixel sets, each of which is disjoint. Further each of the plurality of common pixels is a member of one of the pixel sets.
  • a step b) a pixel set is selected.
  • a pixel set measurement matrix is built.
  • a pixel set projection matrix is estimated.
  • the pixel set projection matrix projects a measurement space vector onto the column space of the pixel set measurement matrix.
  • a pixel set system matrix is estimated.
  • the system matrix is parameterized by a set of system matrix parameters.
  • a step f) a pixel set synchronized matrix as a function of the pixel set measurement matrix and the pixel set system matrix is estimated.
  • steps b) to f) are repeated until all the pixels sets have been selected.
  • a sequence of synchronized image frames from the pixel set synchronized matrices is constructed.
  • the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames applies to a global shutter image sensor where the image frame rate is identical to the pattern frame rate.
  • FIG. 4 illustrates the timing for this embodiment.
  • the projector projects N patterns at a fixed frame rate
  • a global shutter image sensor capture N images at identical frame rate. Capturing each image takes exactly one unit of time, normalized by the projector frame rate.
  • the actual value measured by the image sensor at the (x, y) pixel of the n-th image can be modeled as
  • I n ( x,y ) (1 ⁇ t 0 ) P n ( x,y )+ t 0 P n+1 ( x,y )
  • Projected patterns are known in advance, but since it is not known which projector pixel illuminates each image pixel, they have to be treated as unknown. To estimate the value of t.sub.0, the following expression is minimized
  • the N pattern pixel values P 1 (x, y), . . . , P n (x, y) can be estimated for each pixel (x, y) by minimizing the following expression
  • the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames applies to a rolling shutter image sensor where the image frame rate is identical to the pattern frame rate.
  • FIG. 5 illustrates the timing for this embodiment.
  • N patterns at fixed framerate a rolling shutter camera captures N images. Capture begins while pattern P 1 is being projected. Projector framerate is 1, pattern P n is projected between time n ⁇ 1 and n.
  • a camera frame is read every t f time, camera framerate is assumed equal to projector framerate but in practice may vary a little.
  • a camera row requires t, time to be readout from the sensor, thus, a sensor with Y rows needs a time Yt r to read a complete frame, Yt f less than or equal to t r .
  • Each camera frame is exposed t.sub.e time, its readout begins immediately after exposure ends, t e +t f less than or equal to t f .
  • Camera row y in image n begins being exposed at time t n,y
  • I n,y ( n ⁇ t n,y ) k n,y P n +( t n,y +t e ⁇ n ) k n,y P n+1 +C n,y ,
  • I ⁇ n , y - n ⁇ ⁇ ⁇ ⁇ P n ⁇ 1 t e + ( n - 1 ) ⁇ ⁇ ⁇ ⁇ P n ⁇ t f t e + y ⁇ ⁇ ⁇ ⁇ P n ⁇ t r t e + ⁇ ⁇ ⁇ P n ⁇ t 0 t e + P n + 1
  • I n ( x,y ) (1 ⁇ t 0 ⁇ yd ) P n ( x,y )+( t 0 +yd ) P n +1( x,y ),
  • [ t 0 d ] ( ⁇ n , x , y ⁇ ( P n + 1 - P n ) 2 ⁇ [ 1 y y y 2 ] ) - 1 ⁇ ( ⁇ n , x , y ⁇ ( P n + 1 - P n ) ⁇ ( I n - Pn ) ⁇ [ 1 y ] )
  • Equation 29 we use Equation 29 to compute to and d when we have some known (or estimated) pattern values.
  • the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames applies to a global shutter image sensor where the image frame rate is higher or equal than the pattern frame rate.
  • FIG. 6 illustrates the timing for this embodiment.
  • I n 1 ⁇ ⁇ ⁇ t ⁇ ⁇ n ⁇ ( t 0 , ⁇ ⁇ ⁇ t ) T ⁇ p
  • ⁇ ( n,m,t 0, ⁇ f ) max(0,min( m,t n ) ⁇ max( m ⁇ 1, t n ⁇ 1 ))
  • P n (x, y) represents a pattern value contributing to camera pixel (x, y)
  • P n+1 P 1
  • P 0 P N
  • I n+1 I 1
  • I 0 I n
  • I will omit pixel (x, y) to simplify the notation.
  • t ( i + 1 ) A t ⁇ ( t ( i ) ) - 1 ⁇ b t ⁇ ( t ( i ) )
  • Matrix V A (n,t) and vector V b (n,t) are defined such as
  • ⁇ n ⁇ ( t ) V A ⁇ ( n , t ) ⁇ [ t 0 ⁇ ⁇ ⁇ t ] + V b ⁇ ( n , t )
  • the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames applies to a rolling shutter image sensor where the image frame rate is higher or equal than the pattern frame rate.
  • FIG. 7 illustrates the timing for this embodiment.
  • Projector frame rate is 1
  • pattern P m is projected between time m ⁇ 1 and m.
  • a camera frame is read every t f .
  • a camera row requires t r time to be readout from the sensor, thus, a sensor with Y rows needs a time Yt r to read a complete frame, t f ⁇ Yt r .
  • Each camera frame is exposed t.sub.e time, its readout begins immediately after exposure ends, t e +t r >t f .
  • Camera row y in image n begins being exposed at time t n, y
  • Equation E(h) cannot be minimized in closed form because the values matrix V n,y depends on the unknown values. Using an iterative approach the current value h (i) is used to compute V n,y (i) and the next value h (i+1)pl .
  • Decoding is done in two steps: 1) the time offset t 0 need to be estimated for this particular sequence; 2) the pattern values are estimated for each camera pixel, as shown in FIG. 10 .
  • Value t 0 is estimated using Equation E(h) where the known components of h are fixed, but some pattern values are required to be known, specially we need to know for some pixels whether they are transitioning from ‘black’ to ‘white’ or the opposite. Non-transitioning pixels provided no information in this step. Until now, we have projected a couple of black's and white's at the beginning of the sequence to ensure we can normalized all pixels correctly and to simplify t 0 estimation. We will revisit this point in the future for other pattern sequences.
  • pattern values are estimated by minimizing the following energy
  • the exemplary embodiments of the method and system provides a unique solution to the problem of using structure lighting for three-dimensional image capture where the camera and projector are unsychronized.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Electromagnetism (AREA)
  • Image Processing (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

A system and method to capture the surface geometry a three-dimensional object in a scene using unsynchronized structured lighting is disclosed. The method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate, and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene. Each of the synchronized image frames corresponds to one image pattern of the sequence of image patterns.

Description

    CROSS REFERENCE TO RELATED APPLICATIONS
  • This application is a divisional application of U.S. patent application Ser. No. 15/124,176, which is a national phase filing under 35 U.S.C. § 371 of International Application No. PCT/US2015/019357 filed Mar. 9, 2015, which claims priority to earlier filed U.S. Provisional Application Ser. No. 61/949,529, filed Mar. 7, 2014, the contents of which are incorporated herein by reference.
  • STATEMENT REGARDING GOVERNMENT INTEREST
  • This Invention was made with government support under grant number DE-FG02-08ER15937 awarded by the Department of Energy. The government has certain rights in the invention.
  • BACKGROUND OF THE INVENTION 1. Field of the Invention
  • This patent document relates generally to the field of three-dimensional shape capture of the surface geometry of an object, and more particularly to structured lighting three-dimensional shape capture.
  • 2. Background of the Related Art
  • Three-dimensional scanning and digitization of the surface geometry of objects is commonly used in many industries and services, and their applications are numerous. A few examples of such applications are inspection and measurement of shape conformity in industrial production systems, digitization of clay models for industrial design and styling applications, reverse engineering of existing parts with complex geometry for three-dimensional printing, interactive visualization of objects in multimedia applications, three-dimensional documentation of artwork, historic and archaeological artifacts, human body scanning for better orthotics adaptation, biometry or custom-fit clothing, and three-dimensional forensic reconstruction of crime scenes.
  • One technology for three-dimensional shape capture is based on structured lighting. Three dimensional shape capture systems based on structure lighting are more accurate than those based on time-of-flight (TOF) image sensors. In a standard structured lighting 3D shape capture system a pattern projector is used to illuminate the scene of interest with a sequence of known two-dimensional patterns, and a camera is used to capture a sequence of images, synchronized with the projected patterns. The camera captures one image for each projected pattern. Each sequence of images captured by the camera is decoded by a computer processor into a dense set of projector-camera pixel correspondences, and subsequently into a three-dimensional range image, using the principles of optical triangulation.
  • The main limitation of three-dimensional shape capture systems is the required synchronization between projector and camera. To capture a three-dimensional snapshot of a moving scene, the sequence of patterns must be projected at a fast rate, the camera must capture image frames exactly at the same frame rate, and the camera has to start capturing the first frame of the sequence exactly when the projector starts to project the first pattern.
  • Therefore, there is a need for three-dimensional shape measurement methods and systems based on structure lighting where the camera and the pattern projector are not synchronized.
  • Further complicating matters, image sensors generally use one of two different technologies to capture an image, referred to as “rolling shutter” and “global shutter”. “Rolling shutter” is a method of image capture in which a still picture or each frame of a video is captured not by taking a snapshot of the entire scene at single instant in time but rather by scanning across the scene rapidly, either vertically or horizontally. In other words, not all parts of the image of the scene are recorded at exactly the same instant. This is in contrast with “global shutter” in which the entire frame is captured at the same instant. Even though most image sensors in consumer devices are rolling shutter sensors, many image sensors used in industrial applications are global shutter sensors.
  • Therefore, there is a need for three-dimensional shape measurement methods and systems based on structure lighting where the camera and the pattern projector are not synchronized, supporting both global shutter and rolling shutter image sensors.
  • SUMMARY OF THE INVENTION
  • A system and method to capture the surface geometry a three-dimensional object in a scene using unsynchronized structured lighting solves the problems of the prior art. The method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate; and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns. Because the method enables use of an unsynchronized pattern projector and camera significant cost savings can be achieved. The method enables use of inexpensive cameras, such as smartphone cameras, webcams, point-and-shoot digital cameras, camcorders as well as industrial cameras. Furthermore, the method and system enable processing the images with a variety of computing hardware, such as computers, digital signal processors, smartphone processors and the like. Consequently, three-dimensional image capture using structured lighting may be used with relatively little capital investment.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • These and other features, aspects, and advantages of the method and system will become better understood with reference to the following description, appended claims, and accompanying drawings where:
  • FIG. 1 is an illustration of an exemplary embodiment of the method and system for unsynchronized structured lighting;
  • FIG. 2 shows a flowchart of an exemplary embodiment of the method and system for unsynchronized structured lighting;
  • FIG. 3 shows a flowchart of an exemplary embodiment of the method and system for unsynchronized structured lighting of synthesizing the synchronized sequence of image frames;
  • FIG. 4 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a global shutter image sensor is used and where the image frame rate is identical to the pattern frame rate;
  • FIG. 5 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a rolling shutter image sensor is used and where the image frame rate is identical to the pattern frame rate;
  • FIG. 6 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a global shutter image sensor is used and where the image frame rate is higher or equal than the pattern frame rate
  • FIG. 7 is a chart illustrating the timing for method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a rolling shutter image sensor is used and where the image frame rate is higher or equal than the pattern frame rate;
  • FIG. 8 shows a chart illustrating a calibration pattern used to correct the time;
  • FIG. 9 shows a chart illustrating image data normalized according to the method described herein; and
  • FIG. 10 shows a chart illustrating a model estimating the pattern value for each pixel in a captured image.
  • DETAILED DESCRIPTION
  • A system and method to capture the surface geometry a three-dimensional object in a scene using unsynchronized structured lighting is shown generally in FIGS. 1 and 2. The method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate; and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns.
  • One object of the present invention is a system to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames, illustrated in FIG. 1; the unsynchronized image frames captured while a three-dimensional scene was illuminated by a sequence of patterns. The system comprises a pattern projector 2020 and a camera 2030. An object 2010 is partially illuminated by the pattern projector 2020, and partially visible by the camera 2030. The pattern projector projects a sequence of patterns 2021, 2022, 2023 at a certain pattern rate. The pattern rate is measured in patterns per second. The camera 2030 captures a sequence of unsynchronized image frames 2031, 2032, 2033, 2034, at a certain frame rate. The frame rate is larger or equal than the pattern rate. The number of unsynchronized image frames captured by the camera is larger or equal than the number of patterns in the sequence of patterns. The camera starts capturing the first unsynchronized image frame not earlier than the time when the projector starts projecting the first pattern of the sequence of image patterns. The camera ends capturing the last unsynchronized image frame not earlier than the time when the projector ends projecting the last pattern of the sequence of image patterns. To capture a frame the camera opens the camera aperture, which it closes after an exposure time. The camera determines the pixel values by integrating the incoming light while the aperture is open. Since camera and projector are not synchronized, the projector may switch patterns while the camera has the aperture open. As a result, some or all of the pixels of the captured unsynchronized image frame 2032 will be partially exposed to two consecutive patterns 2021 and 2022. The resulting sequence of unsynchronized image frames, are transmitted to a computer processor 2040, which executes a method to synthesize a synchronized sequence of image frames from the unsynchronized sequence of image frames. The number of frames in the synchronized sequence of image frames 2041, 2042, 2043 will be the same as the number of patterns 2021, 2022, 2023 and represent estimates of what the camera would have captured if it were synchronized with the projector. In a preferred embodiment the camera 2030 and the computer processor 2040 are components of a single device such a digital camera, smartphone, or computer tablet.
  • Another object of the invention is an unsynchronized three-dimensional shape capture system, comprising the system to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames described above, and further comprising prior art methods for decoding, three-dimensional triangulation, and optionally geometric processing, executed by the computer processor.
  • Another object of the invention is a three-dimensional snapshot camera comprising the unsynchronized three-dimensional shape capture system, where the projector has the means to select the pattern rate from a plurality of supported pattern rates, the camera has the means to select the frame rate from a plurality of supported frame rates, and the camera is capable of capturing the unsynchronized image frames in burst mode at a fast frame rate. In a preferred embodiment the projector has a knob to select the pattern rate. In another preferred embodiment the pattern rate is set by a pattern rate code sent to the projector through a communications link. Furthermore, the system has means to set the pattern rate and the frame rate so that the frame rate is not slower than the pattern rate. In a more preferred embodiment the user sets the pattern rate and the frame rate.
  • In a more preferred embodiment of the snapshot camera, the camera has the means to receive a camera trigger signal, and the means to set the number of burst mode frames. In an even more preferred embodiment, the camera trigger signal is generated by a camera trigger push-button. When the camera receives the trigger signal it starts capturing the unsynchronized image frames at the set frame rate, and it stops capturing unsynchronized image frames after capturing the set number of burst mode frames.
  • In a first preferred embodiment of the snapshot camera with camera trigger signal, the projector continuously projects the sequence of patterns in a cyclic fashion. In a more preferred embodiment the system has the means of detecting when the first pattern is about to be projected, and the camera trigger signal is delayed until that moment.
  • In a second preferred embodiment of the snapshot camera with camera trigger signal, the projector has the means to receive a projector trigger signal. In a more preferred embodiment the camera generates the projector trigger signal after receiving the camera trigger signal, and the camera has the means to send the projector trigger signal to the projector. In an even more preferred embodiment the camera has a flash trigger output, and it sends the projector trigger signal to the projector through the flash trigger output. When the projector receives the trigger signal it starts projecting the sequence of patterns at the set pattern rate, and it stops projecting patterns after it projects the last pattern.
  • Another object of this invention is a method to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames, generating a number of frames in the synchronized sequence of image frames equal to the number of projected patterns, and representing estimates of what the camera would have captured if it were synchronized with the projector.
  • As will be described in greater detail below in the associated proofs, the method to synthesize the synchronized sequence of image frames from the unsynchronized sequence of image frames is shown generally in FIG. 3. Each of the synchronized image frames corresponds to one of a sequence of image patterns. Further, the synchronized image frames and unsynchronized image frames have the same width and height. Each image frame includes a plurality of common pixels. In a first step a) the plurality of common pixels is partitioned into a pixel partition. The pixel partition includes a plurality of pixel sets, each of which is disjoint. Further each of the plurality of common pixels is a member of one of the pixel sets. In a step b) a pixel set is selected. In a step c) a pixel set measurement matrix is built. In a step d) a pixel set projection matrix is estimated. The pixel set projection matrix projects a measurement space vector onto the column space of the pixel set measurement matrix. In a step e) a pixel set system matrix is estimated. The system matrix is parameterized by a set of system matrix parameters. In a step f) a pixel set synchronized matrix as a function of the pixel set measurement matrix and the pixel set system matrix is estimated. In a step g) steps b) to f) are repeated until all the pixels sets have been selected. In a step h) a sequence of synchronized image frames from the pixel set synchronized matrices is constructed.
  • In a preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a global shutter image sensor where the image frame rate is identical to the pattern frame rate. FIG. 4 illustrates the timing for this embodiment. In this embodiment, the projector projects N patterns at a fixed frame rate, a global shutter image sensor capture N images at identical frame rate. Capturing each image takes exactly one unit of time, normalized by the projector frame rate. The start time for the first image capture t.sub.0 is unknown, but the start time for the n-th image capture is related to the start time for the first image capture tn=t0+n−1. The actual value measured by the image sensor at the (x, y) pixel of the n-th image, can be modeled as

  • I n(x,y)=(1−t 0)P n(x,y)+t 0 P n+1(x,y)
  • where Pn(x, y) and Pn−1(x, y) represent the pattern values to be estimated that contribute to the image pixel (x, y) and Pn+1=P1. Projected patterns are known in advance, but since it is not known which projector pixel illuminates each image pixel, they have to be treated as unknown. To estimate the value of t.sub.0, the following expression is minimized
  • E ( t 0 ) = 1 2 n = 1 N ( x , y ) ( ( 1 - t 0 ) P n ( x , y ) + I 0 P n + 1 ( x , y ) - I n ( x , y ) ) 2
      • with respect to t0, where the sum is over a subset of pixels (x, y) for which the corresponding pattern pixel values Pn(x, y) and Pn−1(x, y) are known. Differentiating E(t0) with respect to t0 and equating the result to zero, an expression to estimate t0 is obtained
  • I 0 = n = 1 N ( x , y ) ( P n + 1 ( x , y ) - P n ( x , y ) ( I n ( x , y ) - P n ( x , y ) ) n = 1 N ( x , y ) ( P n + 1 ( x , y ) - P n ( x , y ) ) 2
  • Once the value of t0 has been estimated, the N pattern pixel values P1(x, y), . . . , Pn(x, y) can be estimated for each pixel (x, y) by minimizing the following expression
  • E ( P 1 ( x , y ) , , P N ( x , y ) ) = 1 2 n = 1 N ( ( 1 - t 0 ) P n ( x , y ) + t 0 P n + 1 ( x , y ) - I n ( x , y ) ) 2
      • which reduces to solving the following system of N linear equations
  • β P n - 1 ( x , y ) + α P n ( x , y ) + β P n + 1 ( x , y ) = t 0 I n - 1 ( x , y ) + ( 1 - t 0 ) I n ( x , y )
      • for n=1, . . . , N, where α=t2 0+(1−t0)2 and β=t0(1−t0).
  • In another preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a rolling shutter image sensor where the image frame rate is identical to the pattern frame rate. FIG. 5 illustrates the timing for this embodiment. We project N patterns at fixed framerate, a rolling shutter camera captures N images. Capture begins while pattern P1 is being projected. Projector framerate is 1, pattern Pn is projected between time n−1 and n. A camera frame is read every tf time, camera framerate is assumed equal to projector framerate but in practice may vary a little. A camera row requires t, time to be readout from the sensor, thus, a sensor with Y rows needs a time Ytr to read a complete frame, Ytf less than or equal to tr. Each camera frame is exposed t.sub.e time, its readout begins immediately after exposure ends, te+tf less than or equal to tf.
  • Camera row y in image n begins being exposed at time tn,y

  • t n,y =t 0+(n−1)t f +yt r ,y:0 . . . Y−1,
      • and exposition ends at time tn,y+te
  • In this model image n is exposed while pattern Pn and Pn+1 are being projected. Intensity level measured at a pixel in row y is given by

  • I n,y=(n−t n,y)k n,y P n+(t n,y +t e −n)k n,y P n+1 +C n,y,
  • The constants kn,y and Cn,y are scene dependent.
  • Let be min {In,y} a pixel being being exposed while P(t)=0, and max {In,y} a pixel being exposed while P(t)=1, max {In,y}=te kn,y+Cn,y. Now, we define a normalized image In,y as,
  • I ~ n , y = ( n - t n , y ) P n + ( t n , y + t e - n ) P n + 1 t e
  • A normalized image is completely defined by the time variables and pattern values. In this section we want to estimate the time variables. Let's rewrite Equation 58 as
  • I ~ n , y = - n Δ P n 1 t e + ( n - 1 ) Δ P n t f t e + y Δ P n t r t e + Δ P n t 0 t e + P n + 1
      • being t0 and d unknown. Image pixel values are given by

  • I n(x,y)=(1−t 0 −yd)P n(x,y)+(t 0 +yd)P n+1(x,y),
  • Same as before, Pn(x, y) and Pn+1(x, y) represent the pattern values contributing to camera pixel (x, y), we define Pn+1=P1, P0=Pn, In+1=I1, and I0=IN, and I will omit pixel (x, y) to simplify the notation. We now minimize the following energy to find the time variables t0 and d
  • E ( t 0 , d ) = 1 2 n = 1 N x , y ( ( 1 - t 0 - y d ) P n + ( t 0 + y d ) P n + 1 - I n ) 2
  • The partial derivatives are given by
  • E ( t 0 , d ) t 0 = n = 1 N x , y ( P n + 1 - P n ) ( ( 1 - t 0 - y d ) P n + ( t 0 + y d ) P n + 1 - I n ) E ( t 0 , d ) d = n = 1 N x , y y ( P n + 1 - P n ) ( ( 1 - t 0 - y d ) P n + ( t 0 + y d ) P n + 1 - I n )
  • We set the gradient equal to the null vector and reorder as
  • [ t 0 d ] = ( n , x , y ( P n + 1 - P n ) 2 [ 1 y y y 2 ] ) - 1 ( n , x , y ( P n + 1 - P n ) ( I n - Pn ) [ 1 y ] )
  • We use Equation 29 to compute to and d when we have some known (or estimated) pattern values.
  • With known t0 and d we estimate pattern values minimizing
  • E ( P 1 , , P N ) = 1 2 n = 1 N ( ( 1 - t 0 - y d ) P n + ( t 0 + y d ) P n + 1 - I n ) 2 .
  • Analogous as in Case 1 we obtain that Ap=b with A as in Equation 12 and α, β, and b defined as

  • a=(1t 0 −yd)2+(t 0 +yd)2, β=(1=t 0 −yd)(t 0 +yd)

  • b=(1−t 0 −yd)(I 1 I 2 . . . , I N)r+(t 0 +yd)(I N ,I 1 , . . . I N−1)r
  • Pattern values for each pixel are given by p=A−1 b.
  • In another preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a global shutter image sensor where the image frame rate is higher or equal than the pattern frame rate. FIG. 6 illustrates the timing for this embodiment. We now project M patterns at fixed framerate and we capture N images with a global shutter camera, also at a fixed framerate. We require that N≥M. Capture begins while pattern P1 is being projected. We introduce a new variable d which is the camera capture delay from one row to the next. Same as in Case 1, up to two patterns may contribute to each image but here we do not know which ones are because the camera frame rate is unknown. The new image equation is
  • I n = t n - 1 t n m = 1 M f m ( t ) P m dt = m = 1 M P m t n - 1 t n f m ( t ) dt f m ( t ) = { 1 if m - 1 t m 0 otherwise
  • Let Δt=tn−1−tn the time between image frames, let p=(P1 . . . , PM)T and Φn(t0, Δt)=(Φn, 1, t0, Δt), . . . , Φ(n, M, t0, Δt))T and rewrite Equation 33 as
  • I n = 1 Δ t φ n ( t 0 , Δ t ) T p
  • Each function Φ(n, m, t0, Δt)=∫n−1 tnfm(t)dt can be written as

  • Φ(n,m,t0,Δf)=max(0,min(m,t n)−max(m−1,t n−1))
  • Same as before, Pn(x, y) represents a pattern value contributing to camera pixel (x, y), we define Pn+1=P1, P0=PN, In+1=I1, and I0=In, and I will omit pixel (x, y) to simplify the notation.
  • We now minimize the following energy to find the time variables t0 and Δt
  • E ( t 0 , Δ t ) = 1 2 n = 1 N x , y ( 1 Δ t φ n ( t 0 , Δ t ) T - I n ) 2
  • We solve for t0 and Δt by making
  • E ( t 0 , Δ t ) = 0 E ( t 0 , Δ t ) = n = 1 N x , y 1 Δ t J φ n ( t 0 , Δ t ) T p ( 1 Δ t φ n ( t 0 , Δ t ) T p - I n )
  • Because JΦn(t0, Δt) depends on the unknown value t=(t0, Δt)T we solve for them iteratively
  • t ( i + 1 ) = A t ( t ( i ) ) - 1 b t ( t ( i ) ) A t ( t ) = n = 1 N x , y 1 Δ t 2 J φ n ( t ) T pp T V A ( n , t ) b t ( t ) = n = 1 N x , y 1 Δ t J φ n ( t ) T p ( I n - 1 Δ t V b ( n , t ) T p )
  • Matrix VA(n,t) and vector Vb(n,t) are defined such as
  • φ n ( t ) = V A ( n , t ) [ t 0 Δ t ] + V b ( n , t )
  • For completeness we include the following definitions:
  • V A ( , n , t ) = [ v A ( n , 1 , t ) , , v A ( n , M , t ) ] T V b ( n , t ) = [ v b ( n , 1 , t ) , , v b ( n , M , t ) ] T t diff min ( m , t n ) - max ( m - 1 , t n - 1 ) v A ( n , m , t ) = { [ 0 , 1 ] T if m - 1 t n - 1 t n m and t diff > 0 [ - 1 , 1 - n ] T if m - 1 t n - 1 m t n and t diff > 0 [ 1 , n ] T if t n - 1 m - 1 t n m and t diff > 0 [ 0 , 0 ] T if t n - 1 m - 1 m t n and t diff > 0 [ 0 , 0 ] T otherwise v b ( n , m , t ) = { 0 if m - 1 t n - 1 t n m and t diff > 0 m if m - 1 t n - 1 m t n and t diff > 0 1 - m if t n - 1 m - 1 t n m and t diff > 0 1 if t n - 1 m - 1 m t n and t diff > 0 0 otherwise J φ n ( t ) = V A ( n , t )
  • With known t0 and 66 t we estimate pattern values minimizing
  • E ( p ) = 1 2 n = 1 N ( φ n ( t ) T p - I n ) 2
  • Analogous as in Case 1 we obtain that Ap=b with
  • A = Φ T Φ , b = Φ T I Φ = [ φ 1 ( t ) T φ N ( t ) T ]
  • Pattern values for each pixel are given by p=A−1b.
  • In another preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a rolling shutter image sensor where the image frame rate is higher or equal than the pattern frame rate. FIG. 7 illustrates the timing for this embodiment. Projector frame rate is 1, pattern Pm is projected between time m−1 and m. A camera frame is read every tf. A camera row requires tr time to be readout from the sensor, thus, a sensor with Y rows needs a time Ytr to read a complete frame, tf≥Ytr. Each camera frame is exposed t.sub.e time, its readout begins immediately after exposure ends, te+tr>tf.
  • Camera row y in image n begins being exposed at time tn, y

  • t n,y =t 0+(n−1)t f +yt r ,y:0 . . . Y−1
      • and exposition ends at time tn,y+te.
  • In this model a pixel intensity in image n at row y is given by
  • I n , y = t n , y t n , y + t e k n , y P ( t ) dt + C n , y I n , y = m = 1 M max ( 0 , max ( t n , y , m - 1 ) min ( t n , y + t e , m ) k n , y P m dt ) + C n , y
  • The constants kn, y and Cn, y are scene dependent, P.sub.m is either 0 or 1.
  • Let be min{sub.n, y} a pixel being exposed while P(t)=0, and max{I.sub.n, y} a pixel being exposed while P(t)=1,

  • min{I n,y }=C n,y

  • max{I n,y }=t e k n,y +C n,y
  • Now, we define a normalized image .sub.n, y as,
  • I ^ n , y = I n , y - min { I n , y } max [ I n , y ] min { I n , y } I ~ n , y = 1 t e m = 1 M max ( 0 , max ( t n , y , m - 1 ) min ( t n , y + t e , m ) P m dt )
  • A normalized image is completely defined by the time variables and pattern values. In this section we want to estimate the time variables. Let's rewrite the previous equation as,
  • I ~ n , y = m = 1 M φ ( n , m , y ) P m φ ( n , m , y ) = 1 t e max ( 0 , min ( t n , y + t e , m ) - max ( t n , y , m - 1 ) ) Let be h = 1 t e ( 1 , t f , t r , t 0 ) T , now we write φ ( n , m , y ) = v nmy T h with v nmy defined as v nmy T = { [ t e , 0 , 0 , 0 ] if t n 2 < t m 2 t n 1 > t m 1 t n 2 > t n 1 [ t e - m + 1 , n - 1 , y , 1 ] if t n 2 < t m 2 t n 1 t m 1 t n 2 > t m 1 [ m , 1 - n , - y , - 1 ] if t n 2 t m 2 t n 1 > t m 1 t m 2 > t n 1 [ 1 , 0 , 0 , 0 ] if t n 2 t m 2 t n 1 t m 1 t m 2 > t m 1 [ 0 , 0 , 0 , 0 ] otherwise t n 1 = t n , y , t n 2 = t n , y + t e , t m 1 = m - 1 , t m 2 = m Let be p = ( P 1 , , P M ) T and let V ny be V ny = [ - v n 1 y T - v nMy T ] .
  • We now minimize the following energy to find the unknown h
  • E ( h ) = 1 2 n = 1 N x , y ( p ( x , y ) T V ny h - I ~ n ( x , y ) ) 2
      • with the following constraints
  • h > [ 0 1 0 0 ] and [ 0 - 1 Y 0 0 - 1 1 - ] h [ 0 - 1 ]
      • or equivalently
  • t r e - t f t e - 1 t r + t e t f
  • Equation E(h) cannot be minimized in closed form because the values matrix Vn,y depends on the unknown values. Using an iterative approach the current value h(i) is used to compute Vn,y (i) and the next value h(i+1)pl.
  • Up to this point we have assumed that the only unknown is h, meaning that pattern values are known for all image pixels. The difficulty lies is knowing which pattern pixel is being observed by each camera pixel. We simplify this issue by making calibration patterns all ‘black or all ‘white’, best seen in FIG. 8. For example, a sequence of four patterns ‘{black, black, white, white}’ will produce images with completely black and completely white pixels, as well as pixels in transition from black to white and vice versa. The all black or white pixels are required to produce normalized images, as shown in FIG. 9, and the pixels in transition constrain the solution of the parameter h in Equation E(h).
  • Decoding is done in two steps: 1) the time offset t0 need to be estimated for this particular sequence; 2) the pattern values are estimated for each camera pixel, as shown in FIG. 10. Value t0 is estimated using Equation E(h) where the known components of h are fixed, but some pattern values are required to be known, specially we need to know for some pixels whether they are transitioning from ‘black’ to ‘white’ or the opposite. Non-transitioning pixels provided no information in this step. Until now, we have projected a couple of black's and white's at the beginning of the sequence to ensure we can normalized all pixels correctly and to simplify t0 estimation. We will revisit this point in the future for other pattern sequences.
  • Similarly as for the time variables, pattern values are estimated by minimizing the following energy
  • E ( p ) = 1 2 n = 1 N ( h T V ny T p ( x , y ) - I ~ n ( x , y ) ) 2 , s . t . p m 1
  • The matrix hT Vn,y T is bi-diagonal for N=M and it is fixed if h is known.
  • Therefore, it can be seen that the exemplary embodiments of the method and system provides a unique solution to the problem of using structure lighting for three-dimensional image capture where the camera and projector are unsychronized.
  • It would be appreciated by those skilled in the art that various changes and modifications can be made to the illustrated embodiments without departing from the spirit of the present invention. All such modifications and changes are intended to be within the scope of the present invention except as limited by the scope of the appended claims.

Claims (5)

What is claimed is:
1. A method to synthesize a sequence of synchronized image frames synchronized from a sequence of unsynchronized image frames;
each of the synchronized image frames corresponding to one of a sequence of image patterns;
the synchronized image frames and unsynchronized image frames being image frames of the same width and height;
the image frames comprising a plurality of common pixels;
the method comprising the steps of:
a) partitioning the plurality of common pixels into a pixel partition; the pixel partition comprising a plurality of pixel sets; the pixel sets being disjoint; each of the plurality of common pixels being a member of one of the pixel sets;
b) selecting a pixel set;
c) building a pixel set measurement matrix;
d) estimating a pixel set projection matrix; the pixel set projection matrix projecting a measurement space vector onto the column space of the pixel set measurement matrix;
e) estimating a pixel set system matrix; the system matrix being parameterized by a set of system matrix parameters;
f) estimating a pixel set synchronized matrix as a function of the pixel set measurement matrix and the pixel set system matrix;
g) repeating steps b) to f) until all the pixels sets have been selected; and
h) constructing a sequence of synchronized image frames from the pixel set synchronized matrices.
2. A method as in claim 1, where the pixel partition comprises a single pixel set, and the single pixel set contains all the pixels of the image frames.
3. A method as in claim 1, where the number of pixel sets in the pixel partition is equal to the height of the image frames, and each row of the image frames is a pixel set.
4. A method as in claim 1, where the pixel set measurement matrix is modeled as the product of the system matrix times the pixel set synchronized matrix, and the step of estimating the pixel set synchronized matrices reduces to the solution of a linear least-squares problem.
5. A method as in claim 1, where the system matrix is parameterized by an image frame period parameter, an integration time parameter, and a first pattern delay parameter.
US16/434,846 2014-03-07 2019-06-07 Method and system for unsynchronized structured lighting Abandoned US20190364264A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/434,846 US20190364264A1 (en) 2014-03-07 2019-06-07 Method and system for unsynchronized structured lighting
US17/226,920 US11425358B2 (en) 2014-03-07 2021-04-09 Method and system for unsynchronized structured lighting

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201461949529P 2014-03-07 2014-03-07
PCT/US2015/019357 WO2015134961A1 (en) 2014-03-07 2015-03-09 Method and system for unsynchronized structured lighting
US201615124176A 2016-09-07 2016-09-07
US16/434,846 US20190364264A1 (en) 2014-03-07 2019-06-07 Method and system for unsynchronized structured lighting

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
US15/124,176 Division US20170078649A1 (en) 2014-03-07 2015-03-09 Method and system for unsynchronized structured lighting
PCT/US2015/019357 Division WO2015134961A1 (en) 2014-03-07 2015-03-09 Method and system for unsynchronized structured lighting

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/226,920 Continuation US11425358B2 (en) 2014-03-07 2021-04-09 Method and system for unsynchronized structured lighting

Publications (1)

Publication Number Publication Date
US20190364264A1 true US20190364264A1 (en) 2019-11-28

Family

ID=54055938

Family Applications (3)

Application Number Title Priority Date Filing Date
US15/124,176 Abandoned US20170078649A1 (en) 2014-03-07 2015-03-09 Method and system for unsynchronized structured lighting
US16/434,846 Abandoned US20190364264A1 (en) 2014-03-07 2019-06-07 Method and system for unsynchronized structured lighting
US17/226,920 Active US11425358B2 (en) 2014-03-07 2021-04-09 Method and system for unsynchronized structured lighting

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US15/124,176 Abandoned US20170078649A1 (en) 2014-03-07 2015-03-09 Method and system for unsynchronized structured lighting

Family Applications After (1)

Application Number Title Priority Date Filing Date
US17/226,920 Active US11425358B2 (en) 2014-03-07 2021-04-09 Method and system for unsynchronized structured lighting

Country Status (2)

Country Link
US (3) US20170078649A1 (en)
WO (1) WO2015134961A1 (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6582805B2 (en) * 2015-09-25 2019-10-02 富士通株式会社 Encoding apparatus, encoding system, encoding method, and program
EP3577415A4 (en) 2017-02-03 2020-12-30 Modit3d, Inc. THREE-DIMENSIONAL SCANNER AND PROCESS
EP3577416B1 (en) 2017-02-06 2022-03-30 Riven, Inc. System and method for 3d scanning
JP7029654B2 (en) 2017-07-12 2022-03-04 パナソニックIpマネジメント株式会社 Projection imaging system, measuring device and projection imaging method
EP3474541A1 (en) * 2017-10-17 2019-04-24 ams AG 3d camera system with rolling-shutter image sensor and array of addressable light sources
US10791277B2 (en) * 2018-09-11 2020-09-29 Cognex Corporation Methods and apparatus for optimizing image acquisition of objects subject to illumination patterns
CN112188181B (en) * 2019-07-02 2023-07-04 中强光电股份有限公司 Image display device, stereoscopic image processing circuit and synchronization signal correction method thereof
US11223808B1 (en) * 2020-06-30 2022-01-11 Christie Digital Systems Usa, Inc. Device, system and method for generating a mapping of projector pixels to camera pixels and/or object positions using alternating patterns

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100141780A1 (en) * 2008-12-09 2010-06-10 Kar-Han Tan View Projection Matrix Based High Performance Low Latency Display Pipeline
US7929751B2 (en) * 2005-11-09 2011-04-19 Gi, Llc Method and apparatus for absolute-coordinate three-dimensional surface imaging
US9256943B2 (en) * 2013-03-28 2016-02-09 Texas Instruments Incorporated Projector-camera misalignment correction for structured light systems

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6388654B1 (en) * 1997-10-03 2002-05-14 Tegrity, Inc. Method and apparatus for processing, displaying and communicating images
US8659668B2 (en) * 2005-10-07 2014-02-25 Rearden, Llc Apparatus and method for performing motion capture using a random pattern on capture surfaces
US8016434B2 (en) * 2008-06-05 2011-09-13 Disney Enterprises, Inc. Method and system for projecting an animated object and concurrently moving the object's projection area through an animation pattern
US9066087B2 (en) * 2010-11-19 2015-06-23 Apple Inc. Depth mapping using time-coded illumination
CN102760234B (en) * 2011-04-14 2014-08-20 财团法人工业技术研究院 Depth image acquisition device, system and method
JP5898484B2 (en) 2011-12-19 2016-04-06 キヤノン株式会社 Information processing apparatus, information processing apparatus control method, and program
KR102009292B1 (en) * 2012-05-11 2019-08-12 한국전자통신연구원 Apparatus and method for reconstructing three dimensional face based on multiple cameras
US20140035480A1 (en) * 2012-08-02 2014-02-06 Tsvi Blumin Method for the control of luminance of gas discharge lamps
US20140307055A1 (en) 2013-04-15 2014-10-16 Microsoft Corporation Intensity-modulated light pattern for active stereo
US9569892B2 (en) * 2013-09-26 2017-02-14 Qualcomm Incorporated Image capture input and projection output

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7929751B2 (en) * 2005-11-09 2011-04-19 Gi, Llc Method and apparatus for absolute-coordinate three-dimensional surface imaging
US20100141780A1 (en) * 2008-12-09 2010-06-10 Kar-Han Tan View Projection Matrix Based High Performance Low Latency Display Pipeline
US9256943B2 (en) * 2013-03-28 2016-02-09 Texas Instruments Incorporated Projector-camera misalignment correction for structured light systems

Also Published As

Publication number Publication date
US20210258558A1 (en) 2021-08-19
WO2015134961A1 (en) 2015-09-11
US20170078649A1 (en) 2017-03-16
US11425358B2 (en) 2022-08-23

Similar Documents

Publication Publication Date Title
US11425358B2 (en) Method and system for unsynchronized structured lighting
US10161745B2 (en) Real-time-measurement projection device and three-dimensional-projection measurement device
EP3673646B1 (en) Image stitching with electronic rolling shutter correction
JP6132510B2 (en) 3D video acquisition device and depth information calculation method in 3D video acquisition device
US11330208B2 (en) Image signal processing for reducing lens flare
CN1846447B (en) Image processing methodand image processing apparatus
Vo et al. Spatiotemporal bundle adjustment for dynamic 3d reconstruction
US8350922B2 (en) Method to compensate the effect of the rolling shutter effect
US20160328882A1 (en) Pass-through display of captured imagery
US9946955B2 (en) Image registration method
US10872434B2 (en) Image processing apparatus and method
US9766057B1 (en) Characterization of a scene with structured light
KR101932547B1 (en) Camera system and Method of image registration thereof
CN107967697B (en) Three-dimensional measurement method and system based on color random binary coding structure illumination
JP4831760B2 (en) 3D information detection method and apparatus
US10165257B2 (en) Robust disparity estimation in the presence of significant intensity variations for camera arrays
US20210241474A1 (en) System for image compositing including training with custom synthetic data
JP2004229093A (en) Method, device, and program for generating stereoscopic image, and recording medium
CN100481944C (en) Method and device for the computer-assisted motion compensation of a digitalized image and computer program products and computer-readable storage media
JPH07505033A (en) Mechanical method for compensating nonlinear image transformations, e.g. zoom and pan, in video image motion compensation systems
JP2001330417A (en) Three-dimensional shape measuring method and apparatus using color pattern light projection
CN111373222A (en) Light projection system
CN110383829B (en) Image processing apparatus and method
CN109120846A (en) Image processing method and device, electronic equipment and computer readable storage medium
WO2021154099A1 (en) System for image compositing including training with synthetic data

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

AS Assignment

Owner name: UNITED STATES DEPARTMENT OF ENERGY, DISTRICT OF COLUMBIA

Free format text: CONFIRMATORY LICENSE;ASSIGNOR:BROWN UNIVERSITY;REEL/FRAME:055546/0227

Effective date: 20201120

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载