+

WO2018170918A1 - Multi-camera system for low light - Google Patents

Multi-camera system for low light Download PDF

Info

Publication number
WO2018170918A1
WO2018170918A1 PCT/CN2017/078167 CN2017078167W WO2018170918A1 WO 2018170918 A1 WO2018170918 A1 WO 2018170918A1 CN 2017078167 W CN2017078167 W CN 2017078167W WO 2018170918 A1 WO2018170918 A1 WO 2018170918A1
Authority
WO
WIPO (PCT)
Prior art keywords
color image
pixel array
color
pixel
sensor
Prior art date
Application number
PCT/CN2017/078167
Other languages
French (fr)
Inventor
Hajime Numata
Original Assignee
Huawei Technologies Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co., Ltd. filed Critical Huawei Technologies Co., Ltd.
Priority to PCT/CN2017/078167 priority Critical patent/WO2018170918A1/en
Priority to CN201780085025.8A priority patent/CN110235441B/en
Publication of WO2018170918A1 publication Critical patent/WO2018170918A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/13—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths with multiple sensors

Definitions

  • Embodiments of the present invention relate to multi-camera systems for low light.
  • smartphones require the use of small image sensors, as well as the resolution which is higher than 8MP or 12MP. To achieve both requirements, the size of one pixel is getting smaller, such as 1.4um, 1.12um, 1um or less. Because of the small pixel size, the sensitivity of the sensors/pixels is lower, and the picture quality in low light of the smartphone cameras is worse than that of the DSLRs or higher-quality still cameras.
  • the smartphone camera image sensors have evolved with improvements in efficiency (i.e., QE: Quantum efficiency) .
  • QE Quantum efficiency
  • W white color filter sensors
  • Patent Citation 1 discloses an image sensor having an RGBW array including all-wavelength type white (W) pixels in addition to RGB colors that was suggested as a method of solving the problem that the pixel density of image sensor has increased, the amount of incident light per pixel has decreased, and thus relative noise has increased. Portions with from upper-left to lower-right slanted lines represent blue (B) , portions with from lower-left to upper-right slanted lines represent red (R) , and dotted portions represent green (G) in FIG. 4.
  • B blue
  • R red
  • G green
  • Patent Citation 2 also discloses a decrease in a density of color pixels in the RGBW arrangement relative to the RGB (or Bayer) pixel arrangement.
  • FIG. 5 is a set of diagrams illustrating an example of related-art combined mono and color cameras.
  • FIG. 5 shows the mono (or B/W (black/white) ) camera image sensor as having the same pixel size as the color camera image sensor. Portions with upper-left to lower-right slanted lines represent blue (B) , portions with from lower-left to upper-right slanted lines represent red (R) , and dotted portions represent green (G) in FIG. 5.
  • the rate of improvement in QE is not so large.
  • the QE of the state-of-the-art sensor is sufficiently high. Thus, even if the QE improves, the rate is within a few percentage points, so that the sensitivity does not improve much.
  • An object of an embodiment of the present invention is to use relatively large white pixels on a second camera in different pixel arrays while maintaining the same resolution as that of the conventional color photograph cameras to thereby improve the low-light performance.
  • a multi-camera system including: a color image sensor configured to generate a color image; a single-color filter image sensor configured to generate a single-color image, the single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, wherein the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1, and an output color image is produced by fusion of the color image and the single-color image.
  • the multi-camera system makes it possible to use a second sensor whose sensitivity is 3 to 4 times the sensitivity of a color image sensor and makes it possible for the output of the fusion to achieve superior performance in low light.
  • FIG. 1 is a diagram illustrating a comparison of horizontal/vertical spatial sampling.
  • FIG. 2 is a set of diagrams illustrating zigzag pixel array processing.
  • FIG. 3 is a diagram illustrating an image processing pipeline.
  • FIG. 4 is a diagram illustrating an exemplary prior-art WRGB sensor.
  • FIG. 5 is a set of diagrams illustrating an example of prior-art combined mono and color camera.
  • FIG. 6 is a diagram illustrating a comparison of square/Zigzag pixel arrays.
  • FIG. 7 is a diagram showing the use of the ZigZag pixel array with the pixel size of x1.4.
  • Portions with upper-left to lower-right slanted lines represent blue (B)
  • portions with from lower-left to upper-right slanted lines represent red (R)
  • dotted portions represent green (G) in FIGS. 2-5.
  • a multi-camera system including: a color image sensor configured to generate a color image; a single-color filter image sensor configured to generate a single-color image, the single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, wherein the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1, an output color image is produced by fusion of the color image and the single-color filter image.
  • FIG. 1 shows the spatial sampling interval/frequency of a Bayer color filter sensor and a proposed one-color filter image sensor (here, an example of a BW (we refer to the one-color filter sensor as a BW (Black/White) filter sensor) is used) .
  • the Bayer color filter sensor has a checkered pattern with double density green pixels and less red and blue pixels since green information contributes to strengthening the luminance signal (Y signal) .
  • An output of the color filter sensor is processed by an ISP (image signal processor) , and a Red/Green/Blue (RGB) three color plane is generated by interpolation (de-mosaic) processing. Please see Fig. 2.
  • FIG. 6 is a diagram illustrating a comparison of square/zigzag pixel arrays.
  • the BW filter sensor has a zigzag pixel array such as shown in FIG. 6, which is a pixel array rotated by 45 degrees relative to a square pixel array, or a normal array. If appropriate interpolation is applied (if the same interpolation as G (Green) interpolation is applied) , horizontal and vertical (H/V) resolutions would be the same as those for the color filter sensor even if a 1.4-fold size pixel is used (FIG. 7) .
  • FIG. 6 is a diagram illustrating a comparison of square/zigzag pixel arrays.
  • the BW filter sensor has a zigzag pixel array such as shown in FIG. 6, which is a pixel array rotated by 45 degrees relative to a square pixel array, or a normal array. If appropriate interpolation is applied (if the same interpolation as G (Green) interpolation is applied) , horizontal and vertical (H/V
  • FIG. 7 is a diagram showing the use of the zigzag pixel array with the pixel size of x1.4.
  • the notation “x1.4” is relative to the color sensor pixel.
  • the 1.4-fold sized pixel may be used as shown in FIG. 7.
  • the area of the pixel is twice as large, which means that the sensitivity is also twice.
  • the sensitivity is 1.5 to 2 times.
  • the total sensitivity is 3 to 4 times. This is a larger improvement than that for other solutions.
  • FIG. 3 An example of a total image processing pipeline is shown in FIG. 3.
  • a multi-camera system includes a combination of a plurality of cameras including a plurality of color image sensors, each of which color image sensors is configured to generate a color image ; and a single-color filter image sensor configured to generate a single-color image .
  • the single-color filter image sensor has a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2.
  • the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1.
  • An output color image is produced by fusion of the plurality of color images and the single-color image. The same interpolation is applied for a green pixel on the color image and for the single color image.
  • the second sensor may have 3 to 4 times the sensitivity.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

A multi-camera system is provided. The multi-camera system includes a color image sensor configured to generate a color image; and a single-color filter image sensor configured to generate a single-color image, the single color filter image sensor having a single-color filter image sensor, the single-color filter image sensor including a zigzag pixel array with the pixel size of xl.4, or the square root of 2, wherein the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of xl, an output color image is produced by fusion of the color image and the single-color image.

Description

MULTI-CAMERA SYSTEM FOR LOW LIGHT TECHNICAL FIELD
Embodiments of the present invention relate to multi-camera systems for low light.
BACKGROUND
The performance and technology of smartphone cameras have been improving. The quality of the smartphone cameras are now almost the same as that of DSCs (Digital still cameras) /DSLRs (Digital single-lens reflex cameras) . Because of the demand for further convenience, the demand for higher sensitivity is always strong.
Because of size restrictions, smartphones require the use of small image sensors, as well as the resolution which is higher than 8MP or 12MP. To achieve both requirements, the size of one pixel is getting smaller, such as 1.4um, 1.12um, 1um or less. Because of the small pixel size, the sensitivity of the sensors/pixels is lower, and the picture quality in low light of the smartphone cameras is worse than that of the DSLRs or higher-quality still cameras. The smartphone camera image sensors have evolved with improvements in efficiency (i.e., QE: Quantum efficiency) . For the image sensors for the smartphone cameras such as the so-called second cameras which use WRGB (or RGBW) arrays or W sensors, W (white) color filter sensors may be used. FIG. 4is a diagram illustrating an exemplary prior-art WRGB (or RGBW) sensor. Patent Citation 1 discloses an image sensor having an RGBW array including all-wavelength type white (W) pixels in addition to RGB colors that was suggested as a method of solving the problem that the pixel density of image sensor has increased, the amount of incident light per pixel has decreased, and thus relative noise has increased. Portions with from upper-left to lower-right slanted lines represent blue (B) , portions with from lower-left to upper-right slanted lines represent red (R) , and dotted portions represent green (G) in FIG. 4.
Patent Citation 2 also discloses a decrease in a density of color pixels in the RGBW  arrangement relative to the RGB (or Bayer) pixel arrangement. FIG. 5is a set of diagrams illustrating an example of related-art combined mono and color cameras. FIG. 5shows the mono (or B/W (black/white) ) camera image sensor as having the same pixel size as the color camera image sensor. Portions with upper-left to lower-right slanted lines represent blue (B) , portions with from lower-left to upper-right slanted lines represent red (R) , and dotted portions represent green (G) in FIG. 5.
CITATION LIST
Patent Documents
[PATENT DOCUMENT 1] US9147230B2
[PATENT DOCUMENT 2] US9253459B2
SUMMARY
Problems to be solved by the invention
The technical problems of the related art may be summarized as follows:
(1) The rate of improvement in QE is not so large. The QE of the state-of-the-art sensor is sufficiently high. Thus, even if the QE improves, the rate is within a few percentage points, so that the sensitivity does not improve much.
(2-1) While white pixels of the WRGB array may have 1.3 to 2 times the sensitivity of color pixels, the WRGB array require complicated interpolation algorithms, and false color due to aliasing may occur due to the lower color resolution. Moreover, a 1.5 to 2-fold improvement is not so large for the actual usage.
(2-2) While the second camera which uses the W sensor would reduce the interpolation and color resolution issues, the rate of improvement in sensitivity is the same as the rate of improvement in sensitivity for the second camera which uses the WRGB.
Means for Solving the Problems
An object of an embodiment of the present invention is to use relatively large white pixels on a second camera in different pixel arrays while maintaining the same resolution as that of the conventional color photograph cameras to thereby  improve the low-light performance.
According to an embodiment of the present invention, a multi-camera system is provided including: a color image sensor configured to generate a color image； a single-color filter image sensor configured to generate a single-color image, the single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, wherein the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1, and an output color image is produced by fusion of the color image and the single-color image.
Effects of the Invention
The multi-camera system according to embodiments of the present invention makes it possible to use a second sensor whose sensitivity is 3 to 4 times the sensitivity of a color image sensor and makes it possible for the output of the fusion to achieve superior performance in low light.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a diagram illustrating a comparison of horizontal/vertical spatial sampling.
FIG. 2 is a set of diagrams illustrating zigzag pixel array processing.
FIG. 3 is a diagram illustrating an image processing pipeline.
FIG. 4 is a diagram illustrating an exemplary prior-art WRGB sensor.
FIG. 5 is a set of diagrams illustrating an example of prior-art combined mono and color camera.
FIG. 6 is a diagram illustrating a comparison of square/Zigzag pixel arrays.
FIG. 7 is a diagram showing the use of the ZigZag pixel array with the pixel size of x1.4.
Portions with upper-left to lower-right slanted lines represent blue (B) , portions with from lower-left to upper-right slanted lines represent red (R) , and dotted portions represent green (G) in FIGS. 2-5.
DESCRIPTION OF EMBODIMENTS
Mode 1 for Carrying Out the Invention
According to an embodiment of the present invention, a multi-camera system is provided, including: a color image sensor configured to generate a color image； a single-color filter image sensor configured to generate a single-color image, the single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, wherein the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1, an output color image is produced by fusion of the color image and the single-color filter image. FIG. 1 shows the spatial sampling interval/frequency of a Bayer color filter sensor and a proposed one-color filter image sensor (here, an example of a BW (we refer to the one-color filter sensor as a BW (Black/White) filter sensor) is used) .
The Bayer color filter sensor has a checkered pattern with double density green pixels and less red and blue pixels since green information contributes to strengthening the luminance signal (Y signal) . An output of the color filter sensor is processed by an ISP (image signal processor) , and a Red/Green/Blue (RGB) three color plane is generated by interpolation (de-mosaic) processing. Please see Fig. 2.
The ISPs (image signal processors) for current camera products have good interpolation algorithms, and the same resolution can be generated in the directions of horizontal and vertical lines. FIG. 6 is a diagram illustrating a comparison of square/zigzag pixel arrays. The BW filter sensor has a zigzag pixel array such as shown in FIG. 6, which is a pixel array rotated by 45 degrees relative to a square pixel array, or a normal array. If appropriate interpolation is applied (if the same interpolation as G (Green) interpolation is applied) , horizontal and vertical (H/V) resolutions would be the same as those for the color filter sensor even if a 1.4-fold size pixel is used (FIG. 7) . FIG. 7 is a diagram showing the use of the zigzag pixel array with the pixel size of x1.4. The notation “x1.4” is relative to the color sensor pixel. When the zigzag pixel array (45-degree rotated pixel array) is applied and horizontal/vertical resolutions are to be obtained as the color sensor with the pixel size of “x1” , the 1.4-fold sized pixel may be used as shown in FIG. 7. When the 1.4-fold sized pixel is used, the area of the pixel is twice as large, which means that the sensitivity is also twice. Moreover, when the BW pixel is used, the sensitivity is 1.5 to 2 times.
The total sensitivity is 3 to 4 times. This is a larger improvement than that for other solutions.
An example of a total image processing pipeline is shown in FIG. 3.
Mode 2 for Carrying Out the Invention
According to an embodiment of the present invention, a multi-camera system is provided. The multi-camera system includes a combination of a plurality of cameras including a plurality of color image sensors, each of which color image sensors is configured to generate a color image； and a single-color filter image sensor configured to generate a single-color image. The single-color filter image sensor has a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2. The zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1. An output color image is produced by fusion of the plurality of color images and the single-color image. The same interpolation is applied for a green pixel on the color image and for the single color image. The second sensor may have 3 to 4 times the sensitivity.

Claims (8)

  1. A multi-camera system comprising:
    a color image sensor configured to generate a color image； and
    a single-color filter image sensor configured to generate a single-color image, the single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, wherein the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1 and
    an output color image is produced by fusion of the color image and the single-color image.
  2. A multi-camera system comprising:
    a plurality of color image sensors configured to generate a color image； and
    a single-color filter image sensor configured to generate a single-color image, the single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, wherein the zigzag pixel array is a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1,
    an output color image is produced by fusion of the color image and the single-color image, and
    the same interpolation is applied for a green pixel on the color image and for the single-color image.
  3. A camera module, comprising
    the multi-camera system as claimed in claim 1 or 2.
  4. A smartphone comprising
    the camera module as claimed in claim 3.
  5. A mobile device comprising
    the camera module as claimed in claim 3.
  6. An integrated circuit comprising
    the multi-camera system as claimed in claim 1 or 2.
  7. A method using a multi-camera system including a color image sensor； and a single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, the zigzag pixel array being a pixel array  rotated by 45 degrees relative to a square pixel array with the pixel size of x1, the method comprising:
    generating, by the color image sensor, a color image；
    generating, by the single-color filter image sensor, a single-color image； and
    producing, by fusion of the color image and the single-color image, an output color image.
  8. A method using a multi-camera system including a plurality of cameras including a plurality of color image sensors； and a single-color filter image sensor having a sensor including a zigzag pixel array with the pixel size of x1.4, or the square root of 2, the zigzag pixel array being a pixel array rotated by 45 degrees relative to a square pixel array with the pixel size of x1, the method comprising:
    generating, by each of the plurality of color image sensors, a color image；
    generating, by the single-color filter image sensor, a single-color image；
    producing, by fusion of the color image and the single-color image sensor camera, an output color image； and
    applying the same interpolation for a green pixel on the color image and for the single-color image.
PCT/CN2017/078167 2017-03-24 2017-03-24 Multi-camera system for low light WO2018170918A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2017/078167 WO2018170918A1 (en) 2017-03-24 2017-03-24 Multi-camera system for low light
CN201780085025.8A CN110235441B (en) 2017-03-24 2017-03-24 Multi-camera system for low illumination

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/078167 WO2018170918A1 (en) 2017-03-24 2017-03-24 Multi-camera system for low light

Publications (1)

Publication Number Publication Date
WO2018170918A1 true WO2018170918A1 (en) 2018-09-27

Family

ID=63583882

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/078167 WO2018170918A1 (en) 2017-03-24 2017-03-24 Multi-camera system for low light

Country Status (2)

Country Link
CN (1) CN110235441B (en)
WO (1) WO2018170918A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022011506A1 (en) * 2020-07-13 2022-01-20 深圳市汇顶科技股份有限公司 Image processing method and image processing apparatus

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140160326A1 (en) * 2012-12-06 2014-06-12 Aptina Imaging Corporation Color filter arrangements for fused array imaging systems
CN105580347A (en) * 2013-07-31 2016-05-11 德克索实验室 Photographing device including multiple camera modules
CN105721841A (en) * 2014-12-18 2016-06-29 全视科技有限公司 High resolution array camera

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5472584B2 (en) * 2008-11-21 2014-04-16 ソニー株式会社 Imaging device
WO2010116370A1 (en) * 2009-04-07 2010-10-14 Nextvision Stabilized Systems Ltd Camera systems having multiple image sensors combined with a single axis mechanical gimbal
JP5150796B2 (en) * 2011-03-30 2013-02-27 富士フイルム株式会社 Solid-state imaging device driving method, solid-state imaging device, and imaging apparatus
US9344639B2 (en) * 2014-08-12 2016-05-17 Google Technology Holdings LLC High dynamic range array camera

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140160326A1 (en) * 2012-12-06 2014-06-12 Aptina Imaging Corporation Color filter arrangements for fused array imaging systems
CN105580347A (en) * 2013-07-31 2016-05-11 德克索实验室 Photographing device including multiple camera modules
CN105721841A (en) * 2014-12-18 2016-06-29 全视科技有限公司 High resolution array camera

Also Published As

Publication number Publication date
CN110235441A (en) 2019-09-13
CN110235441B (en) 2021-01-15

Similar Documents

Publication Publication Date Title
US9210391B1 (en) Sensor data rescaler with chroma reduction
EP2642757B1 (en) Imaging systems with clear filter pixels
EP3093819B1 (en) Imaging apparatus, imaging system, and signal processing method
US10572974B2 (en) Image demosaicer and method
US9681059B2 (en) Image-capturing device
US20160110843A1 (en) Sensor data rescaler for image signal processing
US8457433B2 (en) Methods and systems for image noise filtering
TWI386049B (en) A solid-state imaging device, and a device using the solid-state imaging device
TWI549519B (en) High resolution array camera
US8508625B2 (en) Image processing apparatus
JP2022071177A (en) Multiplexed high dynamic range image
EP2720455B1 (en) Image pickup device imaging three-dimensional moving image and two-dimensional moving image, and image pickup apparatus mounting image pickup device
CN104412581B (en) Color image sensor and camera head
JP2017112401A (en) Image sensor, image processing apparatus and method, and program
JP5414691B2 (en) Image processing apparatus and image processing method
US8270047B2 (en) Image processing method, image capture device, and computer readable medium for forming an image based on spatial frequency components
EP3275169B1 (en) Downscaling a digital raw image frame
US20160277721A1 (en) Color filtered area processing method for improving image processing
CN108605091A (en) Signal processing apparatus and imaging device
US20140037207A1 (en) System and a method of adaptively suppressing false-color artifacts
WO2018170918A1 (en) Multi-camera system for low light
WO2016167140A1 (en) Image-capturing device, image-capturing method, and program
US20150124114A1 (en) Imaging apparatus and method of controlling imaging apparatus
JP2006087088A (en) Imaging device, and imaging method
WO2018234616A1 (en) Image processing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17901505

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17901505

Country of ref document: EP

Kind code of ref document: A1

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载