US20080103699A1 - Method and apparatus for performing wide area terrain mapping - Google Patents
Method and apparatus for performing wide area terrain mapping Download PDFInfo
- Publication number
- US20080103699A1 US20080103699A1 US11/352,434 US35243406A US2008103699A1 US 20080103699 A1 US20080103699 A1 US 20080103699A1 US 35243406 A US35243406 A US 35243406A US 2008103699 A1 US2008103699 A1 US 2008103699A1
- Authority
- US
- United States
- Prior art keywords
- frames
- lidar
- camera
- frame
- dem
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000013507 mapping Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims description 20
- 230000033001 locomotion Effects 0.000 claims description 11
- 238000011217 control strategy Methods 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 10
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 8
- 230000008676 import Effects 0.000 claims description 7
- 238000001514 detection method Methods 0.000 claims description 4
- 238000012360 testing method Methods 0.000 claims description 4
- 238000013519 translation Methods 0.000 claims description 3
- 238000012417 linear regression Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 21
- 238000010586 diagram Methods 0.000 description 6
- 238000003780 insertion Methods 0.000 description 5
- 230000037431 insertion Effects 0.000 description 5
- 238000013459 approach Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000000844 transformation Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000011439 discrete element method Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000010006 flight Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000012876 topography Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
Definitions
- Embodiments of the present invention generally relate to an imaging system and, more particularly, the present invention relates to wide area terrain mapping.
- the geographic system industry has begun to exploit scanning LIDAR (Light Detection and Ranging) techniques to directly collect three-dimensional topography.
- LIDAR Light Detection and Ranging
- a single laser beam is moved over the ground using aircraft motion and scanning mirrors.
- range from the laser to the ground (or objects thereon) is estimated.
- Range samples are transformed to three-dimensional points in ground coordinates using high accuracy, onboard GPS/INS data and scan angles (collectively, “metadata”).
- the points are then connected into a three-dimensional mesh surface or re-gridded to a uniformly sampled digital elevation map (DEM) for display to a user.
- DEM digital elevation map
- the present invention is a method and apparatus for performing wide area terrain mapping.
- the system comprises a digital elevation map (DEM) and mosaic generation engine that processes images that are simultaneously captured by an electro-optical camera (RGB camera) and a LIDAR sensor.
- the LIDAR sensor and camera are generally mounted on a scanner within an aerial platform.
- the image data collected by both the camera and the LIDAR sensor are processed to create a geometrically accurate three-dimensional view of objects viewed from the aerial platform.
- FIG. 1 depicts one embodiment of an aerial platform for carrying a sensor suite in accordance with the invention
- FIG. 2 depicts a scan pattern used by the present invention
- FIG. 3 depicts a block diagram of a sensor suite that is mounted to the aerial platform in accordance with the present invention
- FIG. 4 depicts a functional block diagram of software used for processing the information collected by the sensor suite
- FIG. 5 depicts a flow diagram of the processing used by the present invention
- FIGS. 6A-6D depict a graphical representations of a process used to process image data in the present invention.
- FIG. 7 depicts a block diagram of a computer system that is used to implement the invention.
- the present invention is a method and apparatus for performing wide area terrain mapping using a combination of LIDAR and video technology within a single platform.
- FIG. 1 depicts an aerial platform 100 that carries a sensor suite 102 that performs aerial imaging of objects on the ground.
- the platform 100 may be a manned airplane or helicopter, or the platform 100 may be an unmanned aerial vehicle (UAV).
- UAV unmanned aerial vehicle
- the suite 100 is used to collect range data, video imagery, and metadata comprising the location of the platform (e.g., Universal Transverse Mercator (UTM) coordinates) and the orientation of the sensor suite 102 (e.g., a scan angle).
- UTM Universal Transverse Mercator
- This information is stored within the sensor suite 102 for subsequent processing.
- the information could also be transmitted to a ground station (not shown) for processing.
- FIG. 2 depicts a flight path 200 that is flown by the aerial platform while the sensor suite 102 is performing a scan pattern 202 that zigzags about the flight path 200 .
- a swath of terrain is imaged from the air with each pass of the platform.
- the aerial platform can image a large area of terrain.
- FIG. 3 depicts a block diagram of a sensor suite 102 that is carried by the platform 100 of FIG. 1 .
- the sensor suite 102 comprises a sensor assembly 300 coupled to a computer 306 .
- the sensor assembly 300 comprises an electro-optic sensor (e.g., a video camera 302 ) and a LIDAR sensor 304 that are coupled to one another by a scanner 322 .
- the camera 302 and sensor 304 are physically aligned to have a common bore sight.
- the video camera 302 is, in one embodiment of the invention, an RGB camera. However, other camera may be used such as an infrared camera.
- the LIDAR sensor 304 is a flash LIDAR focal plane array. By mounting the LIDAR sensor 304 and video camera 302 upon the same scanning platform (scanner 322 ), the video camera 302 and the LIDAR sensor 304 image objects on the ground at the same time.
- the output from sensor assembly 300 is coupled to the computer 306 for storing the information from the assembly 300 .
- a GPS/INS unit 308 for providing position and orientation information for each of the images captured by the sensor assembly 300 .
- the computer 306 comprises a CPU 310 , a support circuits 312 and memory 314 .
- the CPU 310 may be any one of many commercially available processors or microcontrollers.
- the support circuits 312 include various well known circuitry for supporting the operation of the CPU 310 .
- the support circuits 312 may include clock circuits, cache, power supplies, input/output interfaces, and the like.
- the memory 314 may include random access memory, read only memory, disk drive memory, removable storage, and the like.
- the memory 314 stores the image data 316 created by the sensor assembly as well as metadata 318 created by the GPS/INS unit 308 and the scanner, e.g., the metadata comprises platform location, heading, and scanner direction (scan angle).
- Control software 320 is also stored in memory for controlling the scanner 322 .
- the image data 316 and metadata 318 is generally stored on a removable PCMCIA flash card that can be easily removed from the platform and inserted into a processing system on the ground. Alternatively, the image data 316 and metadata 318 can be transmitted to a ground station while the platform is airborne.
- the sensor suite 102 provides the mapping data capability.
- the LIDAR sensor 304 is capable of acquiring, for example, 128 ⁇ 128 intensity and range images with a resolution of 1 m per pixel at a capture rate of up to 20 Hz.
- the camera 302 captures, for example, 480 ⁇ 480 pixel frames at the same frame rate as the LIDAR sensor 304 .
- the entire sensor suite 102 occupies a small volume, approximately 1 cubic foot, for a payload weight of 26 kg. This is a significant improvement over the existing airborne scanning laser systems that can occupy a volume as large as 7.7 cubic feet with a payload weight of 76.6 kg for the laser system alone.
- the aircraft flies several long parallel passes over the area of interest, while a scanning mirror oscillates the sensor across track. This pattern is shown in FIG. 2 .
- the resulting data follows a two-dimensional zigzagged topology. With this acquisition pattern, temporally successive frames overlap by around 30%, and frames in successive rows of the zigzagged topology overlap by around 50%.
- approximately 7 GB of data are collected.
- FIG. 7 depicts a computer system 700 comprising a computer 702 and disk drive 704 .
- the computer 702 comprises at least one central processing unit (CPU) 704 , support circuits 706 and memory 708 .
- the computer 702 is a dual processor personal computer to achieve a processing rate equivalent to the data acquisition rate. In other words, the application will process one hour of data in the same amount of time that was required to collect the data.
- the CPU 704 may comprise one or more commercially available microprocessors or microcontrollers.
- the support circuits 706 are well-known support circuits that support the operation of the CPU 704 .
- the support circuits 706 may include one or more of clock circuits, power supplies, cache, input/output circuits, network connection circuits and the like.
- the memory 708 comprises software to implement the present invention.
- the software applications include a user interface 410 , a data import module 418 and a DEM and mosaic generation engine 430 ; each of which are discussed in detail below.
- the collected data coupled to the computer 702 via a PCMCIA memory connection, is processed by the various software applications that generate a wide area digital elevation model (DEM) and orthorectified image mosaics from the collected data.
- the generated products e.g., DEM or mosaics
- the application software is backed by the relational database server “postgreSQL” in order to preserve information across work sessions. It also seamlessly links to a three-dimensional point cloud visualization module.
- Typical workflow of a ground analyst to generate mapping products comprises the following steps: one, import the collected data from the memory card into the software; two, visualize the footprints of the LIDAR frames on a map and the 3-D point cloud in the three-dimensional viewer; three, optionally select region of interest for processing; four, register, adjust and grid the LIDAR frames to generate a DEM and an intensity orthorectified mosaic; five, orthorectify and mosaic the RGB visible data; six, view on a map and/or export the generated products.
- FIG. 4 depicts a functional block diagram of the application components involved as an operator progresses through the workflow while the application is executing on the CPU 432 .
- data is imported from the memory flash card 314 into the application 408 , through an intuitive wizard-based user interface 410 .
- the operator invokes the DEM generation and mosaicking engine 430 , again through an intuitive wizard-based user interface 410 .
- the engine 430 relies on a control strategy module 420 that orchestrates the data processing and dispatches data to the various data processing servers 428 , 426 , 424 and 422 .
- results are saved to disk 404 and the appropriate database entries are inserted in the database 406 .
- the data import module 418 converts the LIDAR range data collected during the flight session to LIDAR height field frames in a plane-plus-parallax form that will lead to an efficient frame registration framework.
- the height field frames represent the relative heights of LIDAR point samples with respect to a reference plane located at the mean elevation in the frame.
- the conversion step maps LIDAR frame points (u,v,r) where r is the LIDAR range to plane-plus-parallax points (u,v,p), p being the parallax computed for pixel (u,v) with respect to a reference plane.
- LIDAR range points to a projected space related to the chosen world coordinate system by a three-dimensional projective transformation (represented by a 4 ⁇ 4 matrix).
- a three-dimensional projective transformation represented by a 4 ⁇ 4 matrix.
- two LIDAR height field frames will be related by a three-dimensional projective transformation.
- affine transformation is sufficient to relate two frames.
- Such transformations can be estimated very efficiently using direct methods. Further detail on the registration process follows.
- the data preprocessing expresses the input data with respect to a local vertical coordinate system (LVCS).
- LVCS local vertical coordinate system
- the coordinates of the LVCS origin are that of the centroid of the collected data, rounded to the nearest multiple of 5,000 meters.
- the introduction of an LVCS serves two purposes: first, all the data processing is performed with respect to a Cartesian coordinate system; second, the local origin ensures that the numbers are kept small enough for accurate representation in a 32 bit float.
- LIDAR frames are grouped into segments, for efficient storage and access purposes.
- the segment's boundaries preserve flight line boundaries, so that segments will not straddle two flight lines.
- the following will be stored to disk: a sequence of height field frames, a sequence of intensity frames, a sequence of color visible frames and metadata records for each frame in the segment.
- FIG. 5 is a flowchart depicting the registration process 500 as performed by the registration server 424 and the bundle adjustment server 424 .
- Flash LIDAR registration consists of two major processes 501 and 503 : first, in process 501 , pairwise registration is performed on temporally adjacent frame pairs as well as other overlapping frame pairs that are selected by the topology inference process described below. Then, in process 503 , all the pairwise registration motion parameters in a region are combined with metadata based constraints by least squares adjustment to achieve global consistency.
- frames are supplied to the method 500 .
- the three-dimensional mapping between corresponding three-dimensional points in a pair of LIDAR height maps can be represented by a three-dimensional projective transformation.
- the effect of parallax on the mapping can be disregarded, allowing motion to be approximated by a two-dimensional affine transformation in the x-y plane composed with a translation and a shear in the z dimension.
- two test points are defined at the x,y center of one frame with minimum and maximum height values of that frame.
- test points are mapped using the metadata to the other frame in the pair, and the x,y distance between the two warped points forms an upper bound on the amount of parallax between the two frames. If not negligible, then the original LIDAR height fields can be prewarped at step 506 using metadata into a common orthorectified (nadir view) coordinate system, upon which the previous approximations hold for the warped frames.
- the two-dimensional affine transformation between two frames of height fields is estimated by a multiresolution direct alignment technique.
- a shifted and tilted plane model is fit by linear regression to the residual height between the affine registered frames.
- the final estimated three-dimensional mapping for this pair is then the composition of the x, y and z transforms (if prewarping was performed, the mapping is additionally composed with the prewarp transformations).
- estimated pairwise motion information is output from the process 501 .
- the terrain model is created one tile at a time using the estimated motion information 512 , metadata constraints 516 and reference motion information 520 .
- a least square adjustment is formed to estimate the optimum placement of each frame (three-dimensional rotation and translation) in the current tile.
- Each three-dimensional mapping is represented as a set of four virtual three-dimensional tie point correspondences. Due to noise, the constraints cannot all be exactly satisfied simultaneously, so the placement parameters are jointly estimated to minimize a robust total error criterion.
- the result of process 503 is final motion information 518 .
- Each constraint is inversely weighted by its uncertainty so that three-dimensional mappings with high expected noise to not adversely affect the global solution.
- the noise level of metadata is assumed to be constant over time and thus a fixed weight of one is assigned to all the constraints from metadata.
- a weight of infinity is assigned to constraints on the frames in previously finished tiles, since their placements can no longer be allowed to change.
- a weight between 0 and 10 is defined by the sharpness of the correlation peak.
- the weight function favors high center correlation and penalizes high shifted correlations, which indicate uncertainty in the motion estimation.
- a weight of 0 indicates that the pairwise registration failed.
- the invention uses an approach that generates DEMs and mosaics on a tile by tile basis.
- the size of all the tiles is the same, and user specified.
- the origins of the tiles are defined along a fixed partition of the UTM coordinate system, so that the tiles generated from different overlapping flights are consistent. Also, overlap exists between contiguous tiles, to maintain continuity across tiles, and so that the edge artifacts that occur during gridding and orthorectification can be trimmed off.
- the control strategy module orchestrates the processing of data on a tile by tile basis. For each tile, the control strategy module finds the segments that fall into or overlaps with that tile and proceeds to pull together all the frames of those segments for processing. In order to generate the constraints for the bundle adjustment as described above, frames that overlap must be identified for registration. For a given frame, there will be several types of overlapping frames that are candidate for registration: temporally adjacent frames, and spatially adjacent frames. The second category will include frames that belong to the same flight line as the given frame, and frames that belong to a different flight line. This process of discovering the relative placement of frames, or framed topology, is referred to as a topology inference. While for a given frame, there may be several overlapping frames that fit for registration performing all possible registrations would counter the objective of the data processing rate equivalent to the data acquisition rate. As such, only a subset of registration will actually be performed based on inference modules selection.
- the topology inference module (located within the control strategy module) operates by building an undirected weighted graph 600 for each tile, where each node (dot) is a frame relevant to that tile, and edges (lines) are weighted by the overlap between the two frames they link.
- the mapping from a frame to the LVCS as derived from the metadata is made available during the inference process.
- the topology inference module computes the overlap between two frames as the distance between their centers expressed in the coordinate system of one of the frames.
- the graph 600 of FIG. 6A is first initialized with all the frames relevant to a tile. Edges are inserted between frames that are temporally adjacent, without duplication.
- each node is visited and all frames overlapping with corresponding frames are identified, whether they are from the same flight line or not. Overlapping frames are visited (dashed arrowed lines of FIG. 6B ) by descending order of overlap, to preferentially insert edges with the most overlapping frames first.
- F 1 is referred to as the frame represented by the currently visited node N 1
- F 2 as one of the frames overlapping F 1 .
- the node representing F 2 is referred to node N 2 . If F 2 is temporally adjacent to frame F 1 , no edge is inserted since it already exists following initialization (e.g., nodes N 1 and N 3 ). However, if it isn't, as shown in FIG.
- an insertion cost is computed to determine whether an edge should be inserted.
- this cost weighs the shortest path 602 between node N 1 and node N 2 before insertion, against the shortest path 604 , effectively the overlap between frame F 1 and frame F 2 , that would result from the edge insertion.
- it is computed as the ratio of the Dijkstra shortest path between node N. and node N 2 , to the overlap between frame F 1 and frame F 2 . If the computed ratio exceeds a certain threshold, an edge 606 is inserted. This insertion strategy heuristically insures that edges are inserted sparsely along the graph 600 .
- the edge 606 has been inserted into the graph 600 and the process continues to the next node N 3 .
- the process repeats until all the nodes (frames) are processed.
- the list of frames to register and include in the bundle adjustment is given by the graph's edges.
- the control strategy module can then invoke the registration and bundle adjustment servers to perform the frame alignment.
- the frames pulled for a tile are inserted in a spatial index maintained in memory to speed up the identification of overlapping frames.
- the spatial index is maintained for the processing of an entire data collection.
- the LIDAR data is resampled using the regridding server 426 along a uniformed grid, a process also referred to as “gridding”, to generate a DEM.
- the gridding approach performs a simple averaging: pixels from the LIDAR height field frames are warped to points in the UTM coordinate system, then each point is stored in the proper bin. If there are multiple points in a bin, they are added together and divided by the bin count.
- an orthorectified mosaic can be generated from the intensity data the same time as the DEM. Once a DEM is available, the color visible frames can be orthorectified and mosaicked.
- the mosaicking server 428 uses a backward warping strategy for mosaicking both the intensity and color frames.
- the process first computes the bounding box of a given height field frame in the UTM coordinate system. Each DEM pixel (x, y, z) in the bounding box is warped back to a pixel (u, v, p) in the height field frame.
- the process compares the parallax value P, computed from warping, to the parallax value P′ at position (u, v), interpolated from the height field frame: if P′ differs significantly from P, then the DEM pixel is occluded from the view of the height field frame and should be skipped. Otherwise, the intensity or RGB value at position (u, v) is interpolated and inserted in the mosaic.
- the insertion position is related to the position (x, y, z) by an offset and scaling factor.
- the invention is useful in rapidly creating a wide area terrain map using both LIDAR and visible sensor information.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Electromagnetism (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
Abstract
Description
- This application claims benefit of U.S. provisional patent application Ser. No. 60/651,685, filed Feb. 10, 2005, which is incorporated by reference.
- This invention was made with U.S. government support under contract number GS-10F-0197L-84658. The U.S. government has certain rights in this invention.
- 1. Field of the Invention
- Embodiments of the present invention generally relate to an imaging system and, more particularly, the present invention relates to wide area terrain mapping.
- 2. Description of the Related Art
- The geographic system industry has begun to exploit scanning LIDAR (Light Detection and Ranging) techniques to directly collect three-dimensional topography. Typically, a single laser beam is moved over the ground using aircraft motion and scanning mirrors. Using a measure of the time of flight for each pulse of a laser, range from the laser to the ground (or objects thereon) is estimated. Range samples are transformed to three-dimensional points in ground coordinates using high accuracy, onboard GPS/INS data and scan angles (collectively, “metadata”). The points are then connected into a three-dimensional mesh surface or re-gridded to a uniformly sampled digital elevation map (DEM) for display to a user. However, commercial scanning LIDAR instruments are relatively expensive and bulky such that aerial imagery (video imagery) must be acquired separately and then “draped” over the three-dimensional terrain that has been produced by the LIDAR technique. Such a process is costly, labor intensive, and time consuming.
- Therefore, there is a need in the art for a comprehensive wide area terrain mapping system that combines both LIDAR information and electro-optic information, e.g., video images, in a single platform.
- The present invention is a method and apparatus for performing wide area terrain mapping. The system comprises a digital elevation map (DEM) and mosaic generation engine that processes images that are simultaneously captured by an electro-optical camera (RGB camera) and a LIDAR sensor. The LIDAR sensor and camera are generally mounted on a scanner within an aerial platform. The image data collected by both the camera and the LIDAR sensor are processed to create a geometrically accurate three-dimensional view of objects viewed from the aerial platform.
- So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.
-
FIG. 1 depicts one embodiment of an aerial platform for carrying a sensor suite in accordance with the invention; -
FIG. 2 depicts a scan pattern used by the present invention; -
FIG. 3 depicts a block diagram of a sensor suite that is mounted to the aerial platform in accordance with the present invention; -
FIG. 4 depicts a functional block diagram of software used for processing the information collected by the sensor suite; -
FIG. 5 depicts a flow diagram of the processing used by the present invention; -
FIGS. 6A-6D depict a graphical representations of a process used to process image data in the present invention; and -
FIG. 7 depicts a block diagram of a computer system that is used to implement the invention. - The present invention is a method and apparatus for performing wide area terrain mapping using a combination of LIDAR and video technology within a single platform.
-
FIG. 1 depicts anaerial platform 100 that carries asensor suite 102 that performs aerial imaging of objects on the ground. Theplatform 100 may be a manned airplane or helicopter, or theplatform 100 may be an unmanned aerial vehicle (UAV). Whatever form the platform takes, thesuite 100 is used to collect range data, video imagery, and metadata comprising the location of the platform (e.g., Universal Transverse Mercator (UTM) coordinates) and the orientation of the sensor suite 102 (e.g., a scan angle). This information is stored within thesensor suite 102 for subsequent processing. Of course, the information could also be transmitted to a ground station (not shown) for processing. -
FIG. 2 depicts aflight path 200 that is flown by the aerial platform while thesensor suite 102 is performing ascan pattern 202 that zigzags about theflight path 200. As such, a swath of terrain is imaged from the air with each pass of the platform. By flying in a serpentine path, the aerial platform can image a large area of terrain. -
FIG. 3 depicts a block diagram of asensor suite 102 that is carried by theplatform 100 ofFIG. 1 . Thesensor suite 102 comprises asensor assembly 300 coupled to acomputer 306. Thesensor assembly 300 comprises an electro-optic sensor (e.g., a video camera 302) and aLIDAR sensor 304 that are coupled to one another by ascanner 322. Thecamera 302 andsensor 304 are physically aligned to have a common bore sight. Thevideo camera 302 is, in one embodiment of the invention, an RGB camera. However, other camera may be used such as an infrared camera. In one embodiment of the invention, the LIDARsensor 304 is a flash LIDAR focal plane array. By mounting the LIDARsensor 304 andvideo camera 302 upon the same scanning platform (scanner 322), thevideo camera 302 and the LIDARsensor 304 image objects on the ground at the same time. - The output from
sensor assembly 300 is coupled to thecomputer 306 for storing the information from theassembly 300. Also included on theplatform 100 is a GPS/INS unit 308 for providing position and orientation information for each of the images captured by thesensor assembly 300. Thecomputer 306 comprises aCPU 310, asupport circuits 312 andmemory 314. TheCPU 310 may be any one of many commercially available processors or microcontrollers. Thesupport circuits 312 include various well known circuitry for supporting the operation of theCPU 310. Thesupport circuits 312 may include clock circuits, cache, power supplies, input/output interfaces, and the like. Thememory 314 may include random access memory, read only memory, disk drive memory, removable storage, and the like. Thememory 314 stores theimage data 316 created by the sensor assembly as well asmetadata 318 created by the GPS/INS unit 308 and the scanner, e.g., the metadata comprises platform location, heading, and scanner direction (scan angle).Control software 320 is also stored in memory for controlling thescanner 322. Theimage data 316 andmetadata 318 is generally stored on a removable PCMCIA flash card that can be easily removed from the platform and inserted into a processing system on the ground. Alternatively, theimage data 316 andmetadata 318 can be transmitted to a ground station while the platform is airborne. - More specifically, the
sensor suite 102 provides the mapping data capability. TheLIDAR sensor 304 is capable of acquiring, for example, 128×128 intensity and range images with a resolution of 1 m per pixel at a capture rate of up to 20 Hz. Thecamera 302 captures, for example, 480×480 pixel frames at the same frame rate as theLIDAR sensor 304. - The
entire sensor suite 102 occupies a small volume, approximately 1 cubic foot, for a payload weight of 26 kg. This is a significant improvement over the existing airborne scanning laser systems that can occupy a volume as large as 7.7 cubic feet with a payload weight of 76.6 kg for the laser system alone. - During a typical flight, the aircraft flies several long parallel passes over the area of interest, while a scanning mirror oscillates the sensor across track. This pattern is shown in
FIG. 2 . The resulting data follows a two-dimensional zigzagged topology. With this acquisition pattern, temporally successive frames overlap by around 30%, and frames in successive rows of the zigzagged topology overlap by around 50%. During a typical 1 hour flight plan, approximately 7 GB of data are collected. -
FIG. 7 depicts acomputer system 700 comprising acomputer 702 anddisk drive 704. Thecomputer 702 comprises at least one central processing unit (CPU) 704,support circuits 706 andmemory 708. In one embodiment of the invention, thecomputer 702 is a dual processor personal computer to achieve a processing rate equivalent to the data acquisition rate. In other words, the application will process one hour of data in the same amount of time that was required to collect the data. TheCPU 704 may comprise one or more commercially available microprocessors or microcontrollers. Thesupport circuits 706 are well-known support circuits that support the operation of theCPU 704. Thesupport circuits 706 may include one or more of clock circuits, power supplies, cache, input/output circuits, network connection circuits and the like. Thememory 708 comprises software to implement the present invention. The software applications include auser interface 410, adata import module 418 and a DEM andmosaic generation engine 430; each of which are discussed in detail below. The collected data, coupled to thecomputer 702 via a PCMCIA memory connection, is processed by the various software applications that generate a wide area digital elevation model (DEM) and orthorectified image mosaics from the collected data. The generated products (e.g., DEM or mosaics) are expressed with respective to UTM coordinates. The application software is backed by the relational database server “postgreSQL” in order to preserve information across work sessions. It also seamlessly links to a three-dimensional point cloud visualization module. - Typical workflow of a ground analyst to generate mapping products comprises the following steps: one, import the collected data from the memory card into the software; two, visualize the footprints of the LIDAR frames on a map and the 3-D point cloud in the three-dimensional viewer; three, optionally select region of interest for processing; four, register, adjust and grid the LIDAR frames to generate a DEM and an intensity orthorectified mosaic; five, orthorectify and mosaic the RGB visible data; six, view on a map and/or export the generated products.
-
FIG. 4 depicts a functional block diagram of the application components involved as an operator progresses through the workflow while the application is executing on the CPU 432. First, data is imported from thememory flash card 314 into theapplication 408, through an intuitive wizard-baseduser interface 410. Then, after visualization on a map or in the three-dimensional viewer 402, the operator invokes the DEM generation andmosaicking engine 430, again through an intuitive wizard-baseduser interface 410. Theengine 430 relies on acontrol strategy module 420 that orchestrates the data processing and dispatches data to the variousdata processing servers disk 404 and the appropriate database entries are inserted in thedatabase 406. - The data import
module 418 converts the LIDAR range data collected during the flight session to LIDAR height field frames in a plane-plus-parallax form that will lead to an efficient frame registration framework. Essentially, the height field frames represent the relative heights of LIDAR point samples with respect to a reference plane located at the mean elevation in the frame. To be precise, the conversion step maps LIDAR frame points (u,v,r) where r is the LIDAR range to plane-plus-parallax points (u,v,p), p being the parallax computed for pixel (u,v) with respect to a reference plane. The benefit of this conversion is to map LIDAR range points to a projected space related to the chosen world coordinate system by a three-dimensional projective transformation (represented by a 4×4 matrix). Thus, two LIDAR height field frames will be related by a three-dimensional projective transformation. In fact, for frames within a row of the sensor scanning pattern, and affine transformation is sufficient to relate two frames. Such transformations can be estimated very efficiently using direct methods. Further detail on the registration process follows. - Although the final products are generated with respect to UTM coordinates, the data preprocessing expresses the input data with respect to a local vertical coordinate system (LVCS). The coordinates of the LVCS origin are that of the centroid of the collected data, rounded to the nearest multiple of 5,000 meters. The introduction of an LVCS serves two purposes: first, all the data processing is performed with respect to a Cartesian coordinate system; second, the local origin ensures that the numbers are kept small enough for accurate representation in a 32 bit float.
- During import, successive LIDAR frames are grouped into segments, for efficient storage and access purposes. The segment's boundaries preserve flight line boundaries, so that segments will not straddle two flight lines. For a given segment, the following will be stored to disk: a sequence of height field frames, a sequence of intensity frames, a sequence of color visible frames and metadata records for each frame in the segment. Description information about the segments, including where to access the data sequences on disk and bounding rectangles, is saved in the database.
-
FIG. 5 is a flowchart depicting theregistration process 500 as performed by theregistration server 424 and thebundle adjustment server 424. Flash LIDAR registration consists of twomajor processes 501 and 503: first, inprocess 501, pairwise registration is performed on temporally adjacent frame pairs as well as other overlapping frame pairs that are selected by the topology inference process described below. Then, inprocess 503, all the pairwise registration motion parameters in a region are combined with metadata based constraints by least squares adjustment to achieve global consistency. - At
step 502, frames are supplied to themethod 500. The three-dimensional mapping between corresponding three-dimensional points in a pair of LIDAR height maps can be represented by a three-dimensional projective transformation. For frames taken from nearly the same aerial viewpoint, the effect of parallax on the mapping can be disregarded, allowing motion to be approximated by a two-dimensional affine transformation in the x-y plane composed with a translation and a shear in the z dimension. To check if parallax is negligible, atstep 504, two test points are defined at the x,y center of one frame with minimum and maximum height values of that frame. The test points are mapped using the metadata to the other frame in the pair, and the x,y distance between the two warped points forms an upper bound on the amount of parallax between the two frames. If not negligible, then the original LIDAR height fields can be prewarped atstep 506 using metadata into a common orthorectified (nadir view) coordinate system, upon which the previous approximations hold for the warped frames. - At
steps step 512, estimated pairwise motion information is output from theprocess 501. - At
step 514, the terrain model is created one tile at a time using the estimatedmotion information 512,metadata constraints 516 andreference motion information 520. Given all the three dimensional mappings estimated for the selected frame pairs, the metadata for all the frames in a tile, and the placements of the adjoining frames in previously finished tiles, a least square adjustment is formed to estimate the optimum placement of each frame (three-dimensional rotation and translation) in the current tile. Each three-dimensional mapping is represented as a set of four virtual three-dimensional tie point correspondences. Due to noise, the constraints cannot all be exactly satisfied simultaneously, so the placement parameters are jointly estimated to minimize a robust total error criterion. The result ofprocess 503 isfinal motion information 518. - Each constraint is inversely weighted by its uncertainty so that three-dimensional mappings with high expected noise to not adversely affect the global solution. The noise level of metadata is assumed to be constant over time and thus a fixed weight of one is assigned to all the constraints from metadata. A weight of infinity is assigned to constraints on the frames in previously finished tiles, since their placements can no longer be allowed to change. For estimated pair wise motions, a weight between 0 and 10 is defined by the sharpness of the correlation peak. Specifically the normalized correlation between two height fields registered by two-dimensional affine is computed and then also computed as the registration is perturbed by 4 pixels in 8 compass directions. The weight function favors high center correlation and penalizes high shifted correlations, which indicate uncertainty in the motion estimation. A weight of 0 indicates that the pairwise registration failed.
- Given the sheer amount of data to be processed for a single flight, the invention uses an approach that generates DEMs and mosaics on a tile by tile basis. The size of all the tiles is the same, and user specified. The origins of the tiles are defined along a fixed partition of the UTM coordinate system, so that the tiles generated from different overlapping flights are consistent. Also, overlap exists between contiguous tiles, to maintain continuity across tiles, and so that the edge artifacts that occur during gridding and orthorectification can be trimmed off.
- The control strategy module orchestrates the processing of data on a tile by tile basis. For each tile, the control strategy module finds the segments that fall into or overlaps with that tile and proceeds to pull together all the frames of those segments for processing. In order to generate the constraints for the bundle adjustment as described above, frames that overlap must be identified for registration. For a given frame, there will be several types of overlapping frames that are candidate for registration: temporally adjacent frames, and spatially adjacent frames. The second category will include frames that belong to the same flight line as the given frame, and frames that belong to a different flight line. This process of discovering the relative placement of frames, or framed topology, is referred to as a topology inference. While for a given frame, there may be several overlapping frames that fit for registration performing all possible registrations would counter the objective of the data processing rate equivalent to the data acquisition rate. As such, only a subset of registration will actually be performed based on inference modules selection.
- As depicted in
FIGS. 6A-6D , the topology inference module (located within the control strategy module) operates by building an undirectedweighted graph 600 for each tile, where each node (dot) is a frame relevant to that tile, and edges (lines) are weighted by the overlap between the two frames they link. The mapping from a frame to the LVCS as derived from the metadata is made available during the inference process. By cascading the initial mappings of two frames F1 and F2, the topology inference module computes the overlap between two frames as the distance between their centers expressed in the coordinate system of one of the frames. Thegraph 600 ofFIG. 6A is first initialized with all the frames relevant to a tile. Edges are inserted between frames that are temporally adjacent, without duplication. - From this initial configuration, each node is visited and all frames overlapping with corresponding frames are identified, whether they are from the same flight line or not. Overlapping frames are visited (dashed arrowed lines of
FIG. 6B ) by descending order of overlap, to preferentially insert edges with the most overlapping frames first. To simplify, the subsequent explanation, F1 is referred to as the frame represented by the currently visited node N1, and F2 as one of the frames overlapping F1. The node representing F2 is referred to node N2. If F2 is temporally adjacent to frame F1, no edge is inserted since it already exists following initialization (e.g., nodes N1 and N3). However, if it isn't, as shown inFIG. 6C , an insertion cost is computed to determine whether an edge should be inserted. Intuitively, this cost weighs the shortest path 602 between node N1 and node N2 before insertion, against theshortest path 604, effectively the overlap between frame F1 and frame F2, that would result from the edge insertion. Practically, it is computed as the ratio of the Dijkstra shortest path between node N. and node N2, to the overlap between frame F1 and frame F2. If the computed ratio exceeds a certain threshold, anedge 606 is inserted. This insertion strategy heuristically insures that edges are inserted sparsely along thegraph 600. AtFIG. 6D , theedge 606 has been inserted into thegraph 600 and the process continues to the next node N3. The process repeats until all the nodes (frames) are processed. Once the graph is complete, the list of frames to register and include in the bundle adjustment is given by the graph's edges. The control strategy module can then invoke the registration and bundle adjustment servers to perform the frame alignment. - Given that the database does not store geographic bounds for individual frames but whole segments, the frames pulled for a tile are inserted in a spatial index maintained in memory to speed up the identification of overlapping frames. The spatial index is maintained for the processing of an entire data collection.
- After bundle adjustment, the LIDAR data is resampled using the
regridding server 426 along a uniformed grid, a process also referred to as “gridding”, to generate a DEM. The gridding approach performs a simple averaging: pixels from the LIDAR height field frames are warped to points in the UTM coordinate system, then each point is stored in the proper bin. If there are multiple points in a bin, they are added together and divided by the bin count. Optionally, an orthorectified mosaic can be generated from the intensity data the same time as the DEM. Once a DEM is available, the color visible frames can be orthorectified and mosaicked. - The
mosaicking server 428 uses a backward warping strategy for mosaicking both the intensity and color frames. The process first computes the bounding box of a given height field frame in the UTM coordinate system. Each DEM pixel (x, y, z) in the bounding box is warped back to a pixel (u, v, p) in the height field frame. The process then compares the parallax value P, computed from warping, to the parallax value P′ at position (u, v), interpolated from the height field frame: if P′ differs significantly from P, then the DEM pixel is occluded from the view of the height field frame and should be skipped. Otherwise, the intensity or RGB value at position (u, v) is interpolated and inserted in the mosaic. For the color mosaic, the insertion position is related to the position (x, y, z) by an offset and scaling factor. - The invention is useful in rapidly creating a wide area terrain map using both LIDAR and visible sensor information.
- While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
Claims (24)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/352,434 US7363157B1 (en) | 2005-02-10 | 2006-02-10 | Method and apparatus for performing wide area terrain mapping |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US65168505P | 2005-02-10 | 2005-02-10 | |
US11/352,434 US7363157B1 (en) | 2005-02-10 | 2006-02-10 | Method and apparatus for performing wide area terrain mapping |
Publications (2)
Publication Number | Publication Date |
---|---|
US7363157B1 US7363157B1 (en) | 2008-04-22 |
US20080103699A1 true US20080103699A1 (en) | 2008-05-01 |
Family
ID=39310242
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/352,434 Active US7363157B1 (en) | 2005-02-10 | 2006-02-10 | Method and apparatus for performing wide area terrain mapping |
Country Status (1)
Country | Link |
---|---|
US (1) | US7363157B1 (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102447911A (en) * | 2010-10-01 | 2012-05-09 | 魏载荣 | Image acquisition unit, acquisition method, and associated control unit |
WO2013109742A1 (en) * | 2012-01-18 | 2013-07-25 | Logos Technologies, Inc. | Method, device, and system for computing a spherical projection image based on two-dimensional images |
US8818722B2 (en) | 2011-11-22 | 2014-08-26 | Honeywell International Inc. | Rapid lidar image correlation for ground navigation |
US8855911B2 (en) | 2010-12-09 | 2014-10-07 | Honeywell International Inc. | Systems and methods for navigation using cross correlation on evidence grids |
US9157743B2 (en) | 2012-07-18 | 2015-10-13 | Honeywell International Inc. | Systems and methods for correlating reduced evidence grids |
US20160209846A1 (en) * | 2015-01-19 | 2016-07-21 | The Regents Of The University Of Michigan | Visual Localization Within LIDAR Maps |
CN107833278A (en) * | 2017-10-25 | 2018-03-23 | 中煤航测遥感集团有限公司 | Terrain simulation method, apparatus and electronic equipment |
WO2018061010A1 (en) * | 2016-09-28 | 2018-04-05 | Pixtier Maps Ltd. | Point cloud transforming in large-scale urban modelling |
US10970815B2 (en) * | 2018-07-10 | 2021-04-06 | Raytheon Company | Multi-source image fusion |
US11127145B2 (en) | 2018-07-10 | 2021-09-21 | Raytheon Company | Image registration to a 3D point set |
US11619712B2 (en) * | 2017-09-22 | 2023-04-04 | Leica Geosystems Ag | Hybrid LiDAR-imaging device for aerial surveying |
Families Citing this family (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
IL169934A (en) * | 2005-07-27 | 2013-02-28 | Rafael Advanced Defense Sys | Real-time geographic information system and method |
IL175835A0 (en) * | 2006-05-22 | 2007-07-04 | Rafael Armament Dev Authority | Methods and systems for communicating and displaying points-of-interest |
WO2008100308A1 (en) * | 2006-12-19 | 2008-08-21 | Weil Gary J | Remote sensing of subsurface artifacts by use of visual and thermal imagery |
US7962265B2 (en) * | 2007-11-28 | 2011-06-14 | Honeywell International Inc. | Vehicular linear sensor system |
US8244026B2 (en) * | 2008-01-09 | 2012-08-14 | Tiltan Systems Engineering Ltd. | Apparatus and method for automatic airborne LiDAR data processing and mapping using data obtained thereby |
CN102105810B (en) * | 2008-07-24 | 2013-10-23 | 皇家飞利浦电子股份有限公司 | Three-dimensional (3-D) picture capture |
ES2366717B1 (en) * | 2008-09-03 | 2012-08-30 | Universidad De Sevilla | INFORMATION COLLECTION EQUIPMENT IN WORKS AND INFRASTRUCTURES BASED ON AN UNTRIPULATED AIR VEHICLE. |
US8290294B2 (en) * | 2008-09-16 | 2012-10-16 | Microsoft Corporation | Dehazing an image using a three-dimensional reference model |
US8619071B2 (en) * | 2008-09-16 | 2013-12-31 | Microsoft Corporation | Image view synthesis using a three-dimensional reference model |
US8284250B2 (en) * | 2009-01-16 | 2012-10-09 | Microsoft Corporation | Determining trigger rate for a digital camera |
US20100204964A1 (en) * | 2009-02-09 | 2010-08-12 | Utah State University | Lidar-assisted multi-image matching for 3-d model and sensor pose refinement |
IL198883A (en) * | 2009-05-21 | 2015-03-31 | Israel Aerospace Ind Ltd | Method and system for stereoscopic scanning |
US8473187B2 (en) * | 2009-06-01 | 2013-06-25 | Robert Bosch Gmbh | Method and apparatus for combining three-dimensional position and two-dimensional intensity mapping for localization |
FR2950438B1 (en) * | 2009-09-18 | 2011-10-14 | Eads Defence & Security Sys | THREE-DIMENSIONAL LOCATION OF GROUND TARGET ZONE BY MERGING IMAGES TAKEN BY TWO SATELLITE SENSORS |
US8159524B2 (en) * | 2009-11-09 | 2012-04-17 | Google Inc. | Orthorectifying stitched oblique imagery to a nadir view, and applications thereof |
US8542286B2 (en) * | 2009-11-24 | 2013-09-24 | Microsoft Corporation | Large format digital camera with multiple optical systems and detector arrays |
US8665316B2 (en) * | 2009-11-24 | 2014-03-04 | Microsoft Corporation | Multi-resolution digital large format camera with multiple detector arrays |
US20110221934A1 (en) * | 2010-03-12 | 2011-09-15 | Ideal Innovations Incorporated | Ground-Based Instrumentation Operating with Airborne Wave Reflectors |
US8390444B2 (en) | 2010-04-30 | 2013-03-05 | Hewlett-Packard Development Company, L.P. | Sensor-location system for locating a sensor in a tract covered by an earth-based sensor network |
KR101665567B1 (en) * | 2010-05-20 | 2016-10-12 | 삼성전자주식회사 | Temporal interpolation of three dimension depth image method and apparatus |
US9229106B2 (en) | 2010-08-13 | 2016-01-05 | Ryan Dotson | Enhancement of range measurement resolution using imagery |
ES2390441B1 (en) * | 2010-12-01 | 2013-10-02 | Sacyr, S.A. (Sociedad Unipersonal) | SYSTEM FOR TOPOGRAPHICAL STUDIES AND WORK INSPECTION |
KR20140030183A (en) * | 2011-04-14 | 2014-03-11 | 가부시키가이샤 니콘 | Image processing apparatus and image processing program |
US9746988B2 (en) * | 2011-05-23 | 2017-08-29 | The Boeing Company | Multi-sensor surveillance system with a common operating picture |
US9811880B2 (en) | 2012-11-09 | 2017-11-07 | The Boeing Company | Backfilling points in a point cloud |
US9121719B2 (en) * | 2013-03-15 | 2015-09-01 | Abalta Technologies, Inc. | Vehicle range projection |
US11022681B2 (en) * | 2014-05-06 | 2021-06-01 | Okkola Technologies, Llc | System and method for measuring and rendering terrain on a display |
US9678506B2 (en) | 2014-06-19 | 2017-06-13 | Skydio, Inc. | Magic wand interface and other user interaction paradigms for a flying digital assistant |
US12007763B2 (en) | 2014-06-19 | 2024-06-11 | Skydio, Inc. | Magic wand interface and other user interaction paradigms for a flying digital assistant |
US9798322B2 (en) | 2014-06-19 | 2017-10-24 | Skydio, Inc. | Virtual camera interface and other user interaction paradigms for a flying digital assistant |
US10203692B2 (en) | 2014-07-21 | 2019-02-12 | King Abdullah University Of Science And Technology | Structure from motion (SfM) processing for unmanned aerial vehicle (UAV) |
CN107155341B (en) * | 2014-07-31 | 2020-01-10 | 惠普发展公司,有限责任合伙企业 | Three-dimensional scanning system and frame |
US9446851B2 (en) * | 2014-11-18 | 2016-09-20 | Rapid Imaging Software, Inc. | Landing hazard avoidance display |
US20160238388A1 (en) * | 2015-02-17 | 2016-08-18 | Vivint Solar, Inc. | Solar system installation |
US9947232B2 (en) * | 2015-12-08 | 2018-04-17 | Honeywell International Inc. | Methods and apparatus for identifying terrain suitable for aircraft landing |
US10157497B2 (en) * | 2016-04-22 | 2018-12-18 | Utah State University | Textured digital elevation model generation |
US10520943B2 (en) | 2016-08-12 | 2019-12-31 | Skydio, Inc. | Unmanned aerial image capture platform |
US9734397B1 (en) | 2016-11-04 | 2017-08-15 | Loveland Innovations, LLC | Systems and methods for autonomous imaging and structural analysis |
US9886632B1 (en) | 2016-11-04 | 2018-02-06 | Loveland Innovations, LLC | Systems and methods for autonomous perpendicular imaging of test squares |
US9823658B1 (en) | 2016-11-04 | 2017-11-21 | Loveland Innovations, LLC | Systems and methods for adaptive property analysis via autonomous vehicles |
US10521664B2 (en) | 2016-11-04 | 2019-12-31 | Loveland Innovations, LLC | Systems and methods for autonomous perpendicular imaging of test squares |
US11295458B2 (en) | 2016-12-01 | 2022-04-05 | Skydio, Inc. | Object tracking by an unmanned aerial vehicle using visual sensors |
US10602056B2 (en) * | 2017-01-13 | 2020-03-24 | Microsoft Technology Licensing, Llc | Optimal scanning trajectories for 3D scenes |
US9805261B1 (en) | 2017-02-27 | 2017-10-31 | Loveland Innovations, LLC | Systems and methods for surface and subsurface damage assessments, patch scans, and visualization |
US10012735B1 (en) | 2017-05-04 | 2018-07-03 | Loveland Innovations, LLC | GPS offset calibrations for UAVs |
US10984182B2 (en) | 2017-05-12 | 2021-04-20 | Loveland Innovations, LLC | Systems and methods for context-rich annotation and report generation for UAV microscan data |
US10488521B2 (en) * | 2017-06-13 | 2019-11-26 | TuSimple | Sensor calibration and time method for ground truth static scene sparse flow generation |
US10866101B2 (en) * | 2017-06-13 | 2020-12-15 | Tusimple, Inc. | Sensor calibration and time system for ground truth static scene sparse flow generation |
US10481267B2 (en) * | 2017-06-13 | 2019-11-19 | TuSimple | Undistorted raw LiDAR scans and static point extractions method for ground truth static scene sparse flow generation |
US20180357314A1 (en) * | 2017-06-13 | 2018-12-13 | TuSimple | Time synchronization and data acquisition system for ground truth static scene sparse flow generation |
CN109087338B (en) * | 2017-06-13 | 2021-08-24 | 北京图森未来科技有限公司 | A method and device for extracting sparse optical flow from images |
US10447973B2 (en) * | 2017-08-08 | 2019-10-15 | Waymo Llc | Rotating LIDAR with co-aligned imager |
US10364027B2 (en) | 2017-10-24 | 2019-07-30 | Loveland Innovations, LLC | Crisscross boustrophedonic flight patterns for UAV scanning and imaging |
US10872534B2 (en) * | 2017-11-01 | 2020-12-22 | Kespry, Inc. | Aerial vehicle inspection path planning |
US10991258B2 (en) * | 2018-04-13 | 2021-04-27 | Suzhou Eavision Robotic Technologies Co., Ltd. | Real-time learning and detection of a border in a flight path |
CN112513594B (en) * | 2018-08-06 | 2024-05-17 | 以色列国家农业部、农村发展农业研究组织·沃尔卡尼中心 | Hyperspectral Scanner |
US10366287B1 (en) | 2018-08-24 | 2019-07-30 | Loveland Innovations, LLC | Image analysis and estimation of rooftop solar exposure |
US11205072B2 (en) | 2018-08-24 | 2021-12-21 | Loveland Innovations, LLC | Solar ray mapping via divergent beam modeling |
US11210514B2 (en) | 2018-08-24 | 2021-12-28 | Loveland Innovations, LLC | Image analysis and estimation of rooftop solar exposure via solar ray mapping |
WO2020154966A1 (en) * | 2019-01-30 | 2020-08-06 | Baidu.Com Times Technology (Beijing) Co., Ltd. | A rgb point clouds based map generation system for autonomous vehicles |
WO2020226187A1 (en) * | 2019-05-03 | 2020-11-12 | 엘지전자 주식회사 | Robot generating map on basis of multi-sensor and artificial intelligence and traveling by using map |
WO2021081368A1 (en) | 2019-10-25 | 2021-04-29 | Pictometry International Corp. | System using image connectivity to reduce bundle size for bundle adjustment |
US11532116B2 (en) | 2020-10-30 | 2022-12-20 | Loveland Innovations, Inc. | Graphical user interface for controlling a solar ray mapping |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5963664A (en) * | 1995-06-22 | 1999-10-05 | Sarnoff Corporation | Method and system for image combination using a parallax-based technique |
US6664529B2 (en) * | 2000-07-19 | 2003-12-16 | Utah State University | 3D multispectral lidar |
US6757445B1 (en) * | 2000-10-04 | 2004-06-29 | Pixxures, Inc. | Method and apparatus for producing digital orthophotos using sparse stereo configurations and external models |
US20050052714A1 (en) * | 2003-07-24 | 2005-03-10 | Zebra Imaging, Inc. | Enhanced environment visualization using holographic stereograms |
US20050195096A1 (en) * | 2004-03-05 | 2005-09-08 | Ward Derek K. | Rapid mobility analysis and vehicular route planning from overhead imagery |
US7127348B2 (en) * | 2002-09-20 | 2006-10-24 | M7 Visual Intelligence, Lp | Vehicle based data collection and processing system |
US7187809B2 (en) * | 2004-06-10 | 2007-03-06 | Sarnoff Corporation | Method and apparatus for aligning video to three-dimensional point clouds |
US7194389B2 (en) * | 2003-03-25 | 2007-03-20 | The United States Of America As Represented By The Secretary Of The Army | Fusion of data from differing mathematical models |
-
2006
- 2006-02-10 US US11/352,434 patent/US7363157B1/en active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5963664A (en) * | 1995-06-22 | 1999-10-05 | Sarnoff Corporation | Method and system for image combination using a parallax-based technique |
US6664529B2 (en) * | 2000-07-19 | 2003-12-16 | Utah State University | 3D multispectral lidar |
US6757445B1 (en) * | 2000-10-04 | 2004-06-29 | Pixxures, Inc. | Method and apparatus for producing digital orthophotos using sparse stereo configurations and external models |
US7127348B2 (en) * | 2002-09-20 | 2006-10-24 | M7 Visual Intelligence, Lp | Vehicle based data collection and processing system |
US7194389B2 (en) * | 2003-03-25 | 2007-03-20 | The United States Of America As Represented By The Secretary Of The Army | Fusion of data from differing mathematical models |
US20050052714A1 (en) * | 2003-07-24 | 2005-03-10 | Zebra Imaging, Inc. | Enhanced environment visualization using holographic stereograms |
US7190496B2 (en) * | 2003-07-24 | 2007-03-13 | Zebra Imaging, Inc. | Enhanced environment visualization using holographic stereograms |
US20050195096A1 (en) * | 2004-03-05 | 2005-09-08 | Ward Derek K. | Rapid mobility analysis and vehicular route planning from overhead imagery |
US7187809B2 (en) * | 2004-06-10 | 2007-03-06 | Sarnoff Corporation | Method and apparatus for aligning video to three-dimensional point clouds |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102447911A (en) * | 2010-10-01 | 2012-05-09 | 魏载荣 | Image acquisition unit, acquisition method, and associated control unit |
US8855911B2 (en) | 2010-12-09 | 2014-10-07 | Honeywell International Inc. | Systems and methods for navigation using cross correlation on evidence grids |
US8818722B2 (en) | 2011-11-22 | 2014-08-26 | Honeywell International Inc. | Rapid lidar image correlation for ground navigation |
WO2013109742A1 (en) * | 2012-01-18 | 2013-07-25 | Logos Technologies, Inc. | Method, device, and system for computing a spherical projection image based on two-dimensional images |
GB2512242A (en) * | 2012-01-18 | 2014-09-24 | Logos Technologies Llc | Method, device, and system for computing a spherical projection image based on two-dimensional images |
US9157743B2 (en) | 2012-07-18 | 2015-10-13 | Honeywell International Inc. | Systems and methods for correlating reduced evidence grids |
US20160209846A1 (en) * | 2015-01-19 | 2016-07-21 | The Regents Of The University Of Michigan | Visual Localization Within LIDAR Maps |
US9989969B2 (en) * | 2015-01-19 | 2018-06-05 | The Regents Of The University Of Michigan | Visual localization within LIDAR maps |
WO2018061010A1 (en) * | 2016-09-28 | 2018-04-05 | Pixtier Maps Ltd. | Point cloud transforming in large-scale urban modelling |
US11619712B2 (en) * | 2017-09-22 | 2023-04-04 | Leica Geosystems Ag | Hybrid LiDAR-imaging device for aerial surveying |
CN107833278A (en) * | 2017-10-25 | 2018-03-23 | 中煤航测遥感集团有限公司 | Terrain simulation method, apparatus and electronic equipment |
US10970815B2 (en) * | 2018-07-10 | 2021-04-06 | Raytheon Company | Multi-source image fusion |
US11127145B2 (en) | 2018-07-10 | 2021-09-21 | Raytheon Company | Image registration to a 3D point set |
Also Published As
Publication number | Publication date |
---|---|
US7363157B1 (en) | 2008-04-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7363157B1 (en) | Method and apparatus for performing wide area terrain mapping | |
US9798928B2 (en) | System for collecting and processing aerial imagery with enhanced 3D and NIR imaging capability | |
US10303966B2 (en) | Method and system of image-based change detection | |
US10182212B2 (en) | Wide area intermittent video using non-orthorectified feature matching in long period aerial image capture with pixel-based georeferencing | |
US8315477B2 (en) | Method and apparatus of taking aerial surveys | |
WO2007094765A2 (en) | Method and apparatus for performing wide area terrain mapping | |
EP2111530B1 (en) | Automatic stereo measurement of a point of interest in a scene | |
EP2791868B1 (en) | System and method for processing multi-camera array images | |
US10740608B2 (en) | Wide area intermittent video using non-orthorectified feature matching in long period aerial image capture with pixel-based georeferencing | |
GB2561329A (en) | Method and system for creating images | |
JPWO2012141235A1 (en) | 3D point cloud position data processing apparatus, 3D point cloud position data processing system, 3D point cloud position data processing method and program | |
KR20120105452A (en) | Multi-resolution digital large format camera with multiple detector arrays | |
US20110170800A1 (en) | Rendering a continuous oblique image mosaic | |
Moussa et al. | A fast approach for stitching of aerial images | |
US10453178B2 (en) | Large scale image mosaic construction for agricultural applications | |
US20250109940A1 (en) | System and method for providing improved geocoded reference data to a 3d map representation | |
Elbahnasawy et al. | Multi-sensor integration onboard a UAV-based mobile mapping system for agricultural management | |
CN112632415B (en) | Web map real-time generation method and image processing server | |
JP2020165778A (en) | Feature management system | |
Hanna et al. | Wide-area terrain mapping by registration of flash LIDAR imagery | |
Hsu | Geocoded terrestrial mosaics using pose sensors and video registration | |
AU2013260677B2 (en) | Method and apparatus of taking aerial surveys | |
Gabara et al. | Kortowo test field for testing photogrammetric products accuracy–design and first evaluation | |
Trask et al. | Automated model-based image registration for tactical image exploitation | |
CN115967354A (en) | Photovoltaic fault detection method based on unmanned aerial vehicle cruising |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SARNOFF CORPORATION, NEW JERSEY Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HANNA, BARBARA;CHAI, BING-BING;HSU, STEPHEN;REEL/FRAME:017608/0293;SIGNING DATES FROM 20060331 TO 20060425 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: PAT HOLDER CLAIMS SMALL ENTITY STATUS, ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: LTOS); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2553); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY Year of fee payment: 12 |