CN111756974A - Image Sensors and Electronics - Google Patents

Abstract

Provided are an image sensor and an electronic device capable of improving the performance of the image sensor. The image sensor includes: a filter cell array including a plurality of filter cell groups, each of the plurality of filter cell groups including 4 × 4 filter cells; wherein, in 4 × 4 filtering units, each row, each column and each diagonal line include 2 white filtering units and 2 color filtering units; the pixel unit array comprises a plurality of pixel units, the pixel unit array is positioned below the light filtering unit array, and the plurality of pixel units in the pixel unit array correspond to the plurality of light filtering units in the light filtering unit array one by one. By the scheme, the white light filtering units and the color light filtering units are uniformly distributed in the light filtering unit array, so that image processing is conveniently performed by a subsequent image algorithm, the image color recovery is facilitated, the generation of color moire fringes is avoided, and the image quality parameters such as SNR (signal to noise ratio), resolution and the like of the image can be further improved.

Description

Image Sensors and Electronics

This application claims the priority of the Chinese invention application filed on May 15, 2020 with the application number 202010410639.2 and the invention title "Image Sensor and Electronic Device", the entire contents of which are incorporated into this application by application.

technical field

The present application relates to the field of sensors, and more particularly, to an image sensor and an electronic device.

Background technique

An image sensor is an electronic device that converts an optical image into a digital signal, and generally includes a pixel array composed of a plurality of pixel units, each pixel unit in the pixel array is used to form a pixel value in the image. In order to enable the image sensor to collect color images, a color filter (CF) may be disposed above the pixel unit, so that the pixel unit can receive a light signal of a specific color and form a pixel value corresponding to the light signal of the specific color.

However, when the color filter is set, the amount of light received by each pixel unit decreases, resulting in a decrease in the signal-to-noise ratio (SNR) of the image, which affects the image quality. And if the image sensor is applied to a mobile device, the size of the image sensor is limited, and the photosensitive area of the corresponding pixel array is also limited, so in a low-light environment, the image quality will be further limited.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image sensor and an electronic device, aiming at solving the problem of poor image quality obtained by the image sensor in a low-light environment.

In a first aspect, an image sensor is provided, comprising: a filter unit array including a plurality of filter unit groups, and each filter unit group in the multiple filter unit groups includes 4×4 filter units; wherein, In the 4×4 filter units, each row, each column and each diagonal line includes 2 white filter units and 2 color filter units;

The pixel unit array includes a plurality of pixel units, the pixel unit array is located below the filter unit array, and the pixel units in the pixel unit array correspond one-to-one with the filter units in the filter unit array.

Through the solution of the embodiment of the present application, since each filter unit group includes 2 white filter units and 2 color filter units in each row, each column and each diagonal line, therefore, multiple filter units The filter unit array formed by the light unit group also includes two types of filter units: white filter unit and color filter unit in each row, each column and oblique direction. The white filter unit and the color filter unit are in the filter unit. The uniform distribution in the cell array not only facilitates image processing by subsequent image algorithms, but also facilitates image color recovery, reduces image color loss, and avoids the generation of colored moire fringes. It can also obtain more accurate full-resolution grayscale images and further improve image image quality parameters such as SNR and resolution.

In some possible implementations, the filter unit group includes a first color filter unit, a second color filter unit and a third color filter unit of different colors, and the number of the first color filter unit is equal to the number of the first color filter unit. The sum of the numbers of the second color filter unit and the third color filter unit.

In some possible implementations, in the filter unit group, the number of the second color filter units is equal to the number of the third color filter units.

In some possible implementations, in the filter unit group, the colors of the two color filter units in each row are different, and the colors of the two color filter units in each column are different.

In some possible implementations, in each row of the filter unit group, 2 white filter units and 2 color filter units are arranged at intervals; in each column of the filter unit group, 2 white filter units The light units are arranged consecutively, and the 2 color filter units are arranged consecutively.

In some possible implementations, the color filter units of the same color in the filter unit group are arranged in a center-symmetric arrangement.

In some possible implementations, two target filter unit sets are arranged in the middle two rows of the filter unit group, and the target filter unit set includes two color filter units of the same color arranged at a common vertex angle.

In some possible implementation manners, in the filter unit group, the white filter units are respectively located in the first row and the second column, the first row and the fourth column, the second row and the second column, and the second row and the fourth column. , the third upper first column, the third row third column, the fourth row first column, the fourth row third column; the second color filter unit is located in the second row first column and the third row second column; the first color filter units are respectively located in the first row, the first column, the second row, the third column, the third row, the fourth column, and the fourth row and the second column; the third color filter units are respectively located in the first column Row third column and fourth row fourth column.

In some possible implementations, in each row of the filter unit group, two white filter units and two color filter units are arranged at intervals; in the two columns of the filter unit group interval, two white filter units are The filter units are arranged continuously, and the two color filter units are arranged separately; in the other two columns of the filter unit group interval, the two white filter units are arranged separately, and the two color filter units are arranged continuously.

In some possible implementations, two target filter unit sets are provided in the middle two columns of the filter unit group, and the target filter unit set includes two color filter units of the same color arranged at a common vertex angle.

In some possible implementations, the filter unit group is provided with 4 target filter unit sets, and the target filter unit set includes 2 color filter units of the same color arranged at a common vertex angle.

In some possible implementation manners, in the filter unit group, the white filter units are located in the first row and the second column, the first row and the fourth column, the second row and the first column, and the second row and the third column respectively. , the first column on the third, the third column on the third row, the second column on the fourth row, and the fourth column on the fourth row; the second color filter unit is located in the first row, the third column, and the second row column; the first color filter unit is located in the first row, the first column, the second row, the second column, the third row, the fourth column, and the fourth row and the third column; the third color filter unit is located in the third column, respectively Row first column and fourth row first column.

In some possible implementations, the first color filter unit, the second color filter unit, and the third color filter unit are used to pass light signals of three colors, and the wavelength bands of the light signals of the three colors Covers the visible light band.

In some possible implementations, the colors of the first color filter unit, the second color filter unit, and the third color filter unit are three of red, green, blue, cyan, magenta, and yellow, respectively. colors.

In some possible implementations, the first color filter unit is a green filter unit, the second color filter unit and the third color filter unit are a red filter unit and a blue filter unit, respectively.

In some possible implementations, the image sensor further includes: a microlens array, including a plurality of microlenses, the microlens array is located above the filter unit array, and is used for condensing the light signal returned by the photographed object to the filter A unit array, wherein a plurality of microlenses in the microlens array are in one-to-one correspondence with a plurality of filter units in the filter unit array.

In some possible implementations, the pixel values of the white pixel units in the pixel array are used to generate the first image data of the photographed object, and the pixel values of the color pixel units in the pixel array are used to generate the second image data of the photographed object. Image data, the first image data and the second image data are used to synthesize the target image of the shooting object; wherein, the white pixel unit is a pixel unit corresponding to the white filter unit, and the color pixel unit is corresponding to the The pixel unit of the color filter unit.

In some possible implementations, the pixel values of the color pixel units in the pixel array are used to generate an intermediate image through interpolation processing, and the intermediate image is used to generate the second image data in Bayer Bayer format through Remosaic processing.

In some possible implementations, in the 2×2 pixel values of the intermediate image, 2 pixel values are the original pixel values of the color pixel unit, and the other 2 pixel values are pixel values obtained by interpolation processing.

In some possible implementations, the first image data and the second image data have the same resolution.

In some possible implementations, the image sensor is a complementary metal oxide semiconductor CMOS image sensor, or a charge coupled device CCD image sensor.

In a second aspect, an electronic device is provided, including: the image sensor in the first aspect or any possible implementation manner of the first aspect.

Description of drawings

FIG. 1 is a schematic structural diagram of an image sensor provided by an embodiment of the present application.

FIG. 2 is a schematic top view of another image sensor provided by an embodiment of the present application.

Fig. 3 is a schematic cross-sectional view of the image sensor in Fig. 2 along the direction A-A'.

FIG. 4 is another schematic cross-sectional view of the image sensor in the direction A-A' of FIG. 2 .

FIG. 5 is a schematic arrangement diagram of filter units in a filter unit array provided by an embodiment of the present application.

6 to 14 are schematic arrangement diagrams of filter units in several filter unit groups provided by the embodiments of the present application.

FIG. 15 is a schematic flowchart of an image processing method provided by an embodiment of the present application.

FIG. 16 is an image schematic diagram of the image processing method in FIG. 15 .

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

It should be understood that the specific examples herein are only for helping those skilled in the art to better understand the embodiments of the present application, rather than limiting the scope of the embodiments of the present application.

It should also be understood that the various implementation manners described in this specification may be implemented individually or in combination, which are not limited by the embodiments of the present application.

The technical solutions of the embodiments of the present application can be applied to various image sensors, such as complementary metal oxide semiconductor (CMOS) image sensors (CMOS image sensors, CIS), or charge coupled devices (CCD) image sensor, but this is not limited in this embodiment of the present application.

As a common application scenario, the image sensor provided in the embodiment of the present application may be applied to a smart phone, a camera, a camera, a tablet computer, and other mobile terminals or other terminal devices with an imaging function.

FIG. 1 shows a schematic structural diagram of an image sensor. As shown in FIG. 1 , the image sensor 100 includes: a pixel array (pixel array) 110, a row selection circuit 120, a column selection circuit 130, a control circuit 140, an analog to digital converter (ADC) circuit 150, a front-end Signal processing circuit 160 and back-end signal processing circuit 170 .

Specifically, as shown in FIG. 1 , a plurality of square pixel units in the pixel array 110 are arranged in M rows×N columns, where M and N are positive integers. Generally speaking, on the plane where the pixel array 110 is located, the row direction of the M rows and the column direction of the N columns are perpendicular to each other. In some cases, for the convenience of description, in a plane, two directions perpendicular to each other, such as row direction and column direction in this application, may be referred to as horizontal direction and vertical direction.

In the pixel unit array 110 shown in FIG. 1 , any side of each square pixel unit is parallel or perpendicular to the row direction or the column direction.

Optionally, the pixel unit may include devices such as a photodiode and a field effect switch, for receiving an optical signal and converting the optical signal into a corresponding electrical signal.

Optionally, if the image sensor needs to collect color images, a color filter array (CFA) may be arranged above the pixel unit array 110, wherein a color filter unit may be correspondingly arranged above each pixel unit. , for the sake of description, hereinafter, the pixel unit provided with the color filter unit above is also referred to as a color pixel unit, for example, the pixel unit provided with the red filter unit above is referred to as a red pixel unit (represented by R in FIG. 1 ) ), the pixel unit provided with the green filter unit above is called the green pixel unit (represented by G in FIG. 1 ), and the pixel unit provided with the blue filter unit above is called the blue pixel unit (represented by B in FIG. 1 ) ).

At present, the CFA of most image sensors adopts the Bayer format based on the three primary colors of RGB. For example, as shown in FIG. 1 , the CFA of the Bayer format is arranged above the pixel unit array 110 , and the pixel unit array 110 has 2×2 pixels. The unit is a basic unit, and each basic unit includes 1 red pixel unit, 1 blue pixel unit, and 2 green pixel units, wherein the 2 green pixel units are arranged adjacent to each other with a common vertex.

The row selection circuit 120 is connected to each row of pixel units in the pixel unit array 110 through M row control lines, and can be used to turn on and off each pixel unit in each row of pixel units. For example, the row selection circuit 120 is connected to the gate of the field effect switch of each pixel unit in the first row of the pixel unit array 110 through a row control line, and controls the working state of the photodiode by turning on or off the field effect switch. Wherein, the M row control lines are all parallel to the horizontal direction.

The column selection circuit 130 is connected to each column of pixel units in the pixel unit array 110 through N column control lines, and can be used to select the signal value output of each pixel unit in each column. For example, the column selection circuit 130 is connected to the source of the field effect switch of each pixel unit in the first column of the pixel unit array 110 through a column control line, and controls the output of the electrical signal converted by the photodiode. Wherein, the N column control lines are all parallel to the vertical direction.

The control circuit 140 is connected to the row selection circuit 120 and the column selection circuit 130, and is used to provide timing to the row selection circuit 120 and the column selection circuit 130, and to control the row selection circuit 120 and the column selection circuit 130 to select the pixel cell array 110. pixel unit, and output the pixel value of the pixel unit.

Optionally, after the row selection circuit 120, the column selection circuit 130 and the control circuit 140 cooperate to output the pixel value generated by the pixel unit array 110, the pixel value of the pixel unit array 110 is transmitted to the ADC circuit 150 for analog-to-digital conversion, and the The analog pixel values are converted into digital pixel values to form a digital image, which is convenient for subsequent image processing by the signal processing circuit 160 to output an optimized color image.

Optionally, the signal processing circuit 160 may include, but is not limited to, an image signal processor (image signal processor, ISP) for linearizing, removing dead pixels, removing noise, color correction, and demosaicing the digital image. ), automatic exposure control (automatic exposure control, AEC), automatic gain control (automatic gain control, AGC), automatic white balance (auto white balance, AWB) and other processing.

For the image sensor 100 using the Bayer format CFA, the red pixel units can only receive red light signals, the green pixel units can only receive green light signals, the blue pixel units can only receive blue light signals, and the light signals received by each pixel unit The smaller intensity results in a larger SNR of the image, which affects the image quality.

And for the image sensor of Bayer format CFA, the high-frequency information of luminance and chromaticity information in the image are likely to overlap, and color aliasing will occur, and color moire is likely to be generated.

Based on the above problems, the present application proposes an image sensor in which a white filter unit is added to the CFA, some pixel units in the pixel unit array receive color light signals, and some pixel units receive white light signals, thereby increasing the light signals received by some pixel units. On this basis, the pixel values of multiple pixel units in the pixel unit array are processed. On the basis of ensuring the color information of the image, the image quality parameters such as SNR and resolution of the image are improved to obtain an optimized color image.

Fig. 2 is a schematic top view of an image sensor 200 provided by an embodiment of the present application, and Fig. 3 is a schematic cross-sectional view of the image sensor 200 along the A-A' direction.

As shown in FIG. 2 and FIG. 3 , the image sensor 200 includes:

The filter unit array 210 includes multiple filter unit groups 211, and each filter unit group in the multiple filter unit groups 211 includes 4×4 filter units;

In the 4×4 filter units, each row, each column and each diagonal line includes 2 white filter units and 2 color filter units.

The pixel unit array 220, located below the filter unit array 210, includes a plurality of pixel units, and the plurality of pixel units in the pixel unit array 220 correspond to the plurality of filter units in the above-mentioned filter unit array 210 one-to-one.

In a possible implementation manner, as shown in FIG. 3 , a plurality of filter elements in the filter element array 210 may be disposed on the upper surface of the plurality of pixel elements in the pixel element array 220; in another possible implementation manner Among them, the plurality of filter units in the filter unit array 210 may also be suspended above the plurality of pixel units in the pixel unit array 220 .

In addition, as shown in FIG. 3 , as an example, each filter unit in the filter unit array 210 is correspondingly disposed directly above each pixel unit in the pixel unit array 220 , in other words, the center of each filter unit corresponds to the The centers of the pixel cells coincide in the vertical direction. In addition to this method, each filter unit in the filter unit array 210 is correspondingly disposed diagonally above each pixel unit in the pixel unit array 220. At this time, each pixel unit in the pixel unit array 220 can receive an oblique direction. The specific position of the filter unit array 210 is not limited in this embodiment of the present application.

The pixel unit in the pixel unit array 220 corresponding to the color filter unit is used for receiving the color light signal passing through the color filter unit, and correspondingly outputs the color pixel value; the pixel unit in the pixel unit array 220 corresponding to the white filter unit It is used for receiving the white light signal passing through the white filter unit, and correspondingly outputting the white pixel value; the color light signal and the white light signal are jointly used to generate the target image of the photographing object. For example, a pixel unit corresponding to a red filter unit receives a red light signal, and the corresponding output pixel value may be called a red pixel value; a pixel unit corresponding to a white filter unit receives a white light signal, and the corresponding output pixel may be called a white pixel value.

Further, as shown in FIG. 4 , FIG. 4 shows a schematic cross-sectional view of another image sensor 200 along the A-A' direction.

As shown in FIG. 4, in addition to the above-mentioned filter unit array 210 and pixel unit array 220, the image sensor 200 further includes:

The microlens array 230 is arranged above the filter unit array 210 and is used for converging the optical signal returned by the photographing object to the filter unit array 210 and reducing the crosstalk of optical signals between adjacent pixel units, wherein the microlens array 230 It includes a plurality of micro-lenses, and the plurality of micro-lenses are in one-to-one correspondence with the plurality of filter units in the above-mentioned filter unit array 210 and the plurality of pixel units in the above-mentioned pixel unit array 220 .

In some embodiments, the pixel structure in the image sensor may be referred to as an on-chip lens (OCL) pixel structure.

Optionally, as shown in FIG. 4 , in the image sensor 200 , a dielectric layer 240 may be further included between the filter unit array 210 and the pixel unit array 220 for connecting the pixel unit array 220 and the filter unit array 210.

In addition, the filter unit array 210 may further include a surrounding dielectric 215 and a reflective grid 216; the reflective grid is used to reflect the light signal incident at a large angle to prevent the loss of the light signal.

The pixel unit array 220 may include a semiconductor substrate 221 and a photosensitive element 222, wherein the photosensitive element 222 is located in the semiconductor substrate 221, and the photosensitive element 222 includes but is not limited to a photodiode (PD). Optionally, the pixel unit array 220 may further include an isolation region 223 between two photosensitive elements 222 to prevent electrical signal interference between two adjacent photosensitive elements.

It can be understood that, in addition to the basic structure shown in FIG. 4 , the image sensor 200 may also include other stacked structures, such as at least one metal interconnection layer, to electrically connect a plurality of pixel units in the pixel unit array, etc. , the structure of the image sensor in the embodiment of the present application is not limited to this.

Optionally, the top view shown in FIG. 2 is actually a schematic diagram of the arrangement of a filter unit array 210 provided in an embodiment of the present application.

As shown in FIG. 2 , in this embodiment of the present application, each filter unit in a filter unit group 211 is a quadrilateral filter unit, for example, all may be square filter units, and 16 square filter units A square filter unit group is formed. In the filter unit group 211, the number of color filter units (shown as shaded blocks in the figure) and white filter units (shown as blank blocks in the figure) are equal, that is, it includes 8 color filter units and 8 a white filter unit. In the plane where the image sensor is located, a filter unit group includes two types of white filter units and color filter units in the horizontal direction (row direction), the vertical direction (column direction) and the 45° direction (diagonal direction). form of filter unit.

It should be noted that in this application, the white filter unit refers to a filter or filter material used to transmit white light. In some embodiments, the white filter unit may also be a transparent material or an air gap. , which is used to transmit all light signals in the environment including white light. Specifically, the white light may be mixed light of various colored lights. For example, the three primary colors of light in the spectrum: blue, red and green, can be mixed in a certain proportion to obtain white light, or the mixture of all visible light in the spectrum is also white light.

Correspondingly, the color filter unit refers to a filter or filter material for transmitting colored light. Specifically, the colored light can be a light signal in any wavelength range in the visible light spectrum. For example, the red filter unit can be used to transmit red light, and the red light can be a wavelength range between 620nm and 750nm in the visible light spectrum. light signal. Similarly, color filter units of other colors are also used to transmit light signals of corresponding colors.

Through the solution of the embodiment of the present application, since each filter unit group includes 2 white filter units and 2 color filter units in each row, each column and each diagonal line, therefore, multiple filter units The filter unit array formed by the light unit group also includes two types of filter units: white filter unit and color filter unit in each row, each column and oblique direction. The white filter unit and the color filter unit are in the filter unit. The uniform distribution in the cell array not only facilitates image processing by subsequent image algorithms, but also facilitates image color recovery, reduces image color loss, and avoids the generation of colored moire fringes. It can also obtain more accurate full-resolution grayscale images and further improve image image quality parameters such as SNR and resolution.

Optionally, in some embodiments, the filter unit group may include filter units of three colors (a first color filter unit, a second color filter unit, and a third color filter unit), for example, you can It is a filter unit of three primary colors, that is, a filter unit of red, green, and blue (red green blue, RGB), or it can also be a filter unit of three complementary colors, that is, cyan, magenta, and yellow (cyan magenta yellow, CMY) filter unit of three colors, or, it can also be a filter unit of one primary color and two complementary colors, or a filter unit of one complementary color and two primary colors.

In the case where the filter unit group includes color filter units of three colors, the color light signals of the three colors passing through the color filter units of the three colors can cover the visible light band. The specific color of the light unit is not limited.

For the convenience of description, in the following, the filter unit group 211 includes color filter units of three colors of red, green and blue as an example for description. When the filter unit group includes color filter units of other three colors , please refer to the technical solution below.

FIG. 5 shows a schematic arrangement diagram of the filter units in the filter unit group 211 .

As shown in FIG. 5 , the filter unit group 211 includes a white filter unit 201 , a red filter unit 202 , a green filter unit 203 and a blue filter unit 204 , optionally, in one filter unit group 211 Among them, the ratio of the number of white filter units 201 , the number of green filter units 203 , the number of red filter units 202 and the number of blue filter units 204 is 4:2:1:1.

Optionally, in the filter unit group 211, the white filter units and the color filter units in each row are arranged at intervals, the 2 white filter units in each column are arranged continuously, and the 2 color filter units are consecutively arranged. set up.

As an example, as shown in FIG. 5 , the two white filter units 201 in the first row and the second row are arranged at intervals in the second and fourth columns, and the two white filter units in the third row and the fourth row are arranged at intervals. The cells 201 are arranged in the first column and the third column at intervals. Correspondingly, the two white filter units 201 in the first column and the third column are continuously arranged in the third row and the fourth row, and the two white filter units 201 in the second column and the fourth column are continuously arranged in the first row. Line 1 and line 2.

Optionally, the colors of the two color filter units in each row and the two color filter units in each column are different.

For example, in FIG. 5, the two color filter units in the first row and the fourth row are the green filter unit 203 and the blue filter unit 204, and the two color filter units in the second row and the third row are Red filter unit 202 and green filter unit 203 . Correspondingly, the two color filter units in the first and second columns are the green filter unit 203 and the red filter unit 202, and the two color filter units in the third and third columns are the blue filter units 204 and the green filter unit 203.

Specifically, in the filter unit group shown in FIG. 5 , the two red filter units 202 may be located in the second row, the first column and the third row, the second column, respectively, and the two red filter units 202 may be represented as R ₂₁ and R ₃₂ ;

The four green filter units 203 may be located in the 1st row and the 1st column, the 2nd row and the 3rd column, the 3rd row and the 4th column, and the 4th row and the 2nd column, and the four green filter units 203 can be represented as G ₁₁ , G ₂₃ , G ₃₄ and G ₄₂ ;

The two blue filter units 204 may be located in the 1st row, 3rd column and the 4th row, 4th column, respectively, and the two blue filter units 204 may be denoted as B ₁₃ and B ₄₄ .

Optionally, based on the unchanged position of the white filter unit 201 in FIG. 5 , according to the requirement that the colors of the 2 color filter units in each row and the 2 color filter units in each column are different, the color filter unit in FIG. The positions of the color filter units of different colors are changed.

For example, based on the filter unit group in FIG. 5 , the positions of the white filter unit and the green filter unit are kept unchanged, and the positions of the red filter unit and the blue filter unit are changed.

Optionally, in some embodiments, the filter unit group includes at least two target filter unit sets, and two color filter units of the same color in each target filter unit set are arranged at a common vertex angle.

Under the condition that the above conditions are satisfied, FIG. 6 shows a schematic arrangement diagram of the filter units in several transformed filter unit groups.

In the various filter unit groups shown in FIG. 5 and FIG. 6 , two target filter unit sets are located in the middle two rows, and one target filter unit set is two green filter units set at a common vertex angle, Another set of target filter units is two blue filter units or red filter units set at a common vertex angle.

For another example, based on the filter unit group in FIG. 5 , only the position of the white filter unit is kept unchanged, and the positions of the green filter unit, the red filter unit and the blue filter unit are changed.

Optionally, in other embodiments, the color filter units of the same color in the filter unit group are arranged in a center-symmetric arrangement.

Under the condition that the above conditions are satisfied, FIG. 7 shows a schematic arrangement diagram of the filter units in several transformed filter unit groups.

In the above-mentioned various filter unit groups shown in FIG. 5 to FIG. 7 , the filter unit group includes at least two target filter unit sets, or the color filter units of the same color are arranged symmetrically in the center, which is convenient for subsequent operations. The image pixel interpolation process can retain more original image color information, which increases the accuracy of color recovery. The specific pixel interpolation process will be described in detail below.

It can be understood that, for each filter unit group in the above-mentioned embodiments of the application, the filter unit group that will undergo geometric transformation, for example, rotated, also falls within the protection scope of the present application.

For example, for the filter unit group shown in FIGS. 5 to 7 , after rotating it 90° clockwise, various filter unit groups as shown in FIG. 8 can be formed.

It can be understood that the filter unit group in FIG. 5 to FIG. 7 and the filter unit group in FIG. 8 can be equivalent to the same filter unit group, and the difference is only the filter unit in the row direction in FIG. 5 to FIG. 7 . The arrangement rule is the same as the arrangement rule of the filter units in the column direction in FIG. 8 . Correspondingly, the arrangement rule of the filter units in the column direction in FIG. 5 to FIG.

As shown in Figure 8, in each column of the filter unit group, two white filter units and two color filter units are arranged at intervals; in each row of the filter unit group, two white filter units Continuously set, and 2 color filter units are set continuously.

It can also be understood that, in addition to the filter unit group shown in FIG. 8 , any filter unit group in the above-mentioned FIGS. 5 to 7 is rotated by other angles, or the filter unit group formed by other geometric transformations is the same. It belongs to the protection scope of the present application and will not be listed one by one here.

The filter unit groups of the above embodiments are all obtained by transformation based on FIG. 5 . In addition to the structures listed above, the filter unit groups of the present application can also be several dotted lines in FIG. 9 to FIG. 11 . The filter unit group 211 shown in the box.

It can be understood that the filter unit group 211 in FIG. 9 to FIG. 11 and the filter unit group 211 in FIG. 2 can form a filter unit array with the same central area, and the difference is only in the outermost circle of the filter unit array 210. Or the arrangement forms of the two circles of filter units are different. Therefore, the filter unit arrays formed by the filter unit groups 211 in FIG. 2 and FIG. 9 to FIG. 11 can be equivalent to the same filter unit array.

Therefore, in the present application, any 4×4 filter unit in the filter unit array 210 in FIG. 2 can be divided into one filter unit group, and the filter unit group in any case of division is in this within the scope of protection of the application.

Similarly, the filter unit array formed by any filter unit group in the present application also falls within the protection scope of the present application.

Referring to the filter unit group shown in FIG. 9, it can be seen by comparing FIG. 9 with FIG. 5 that the positions of the white filter unit and the color filter unit in FIG. 5 are reversed, that is, the filter unit shown in FIG. 9 is formed. Form of light unit group.

Similar to the above-mentioned embodiment, based on the position of the white filter unit in FIG. 9 unchanged, the positions of the color filter units of different colors in FIG. 9 can be changed, and various filter units as shown in FIG. 12 can be obtained. Group.

For example, as shown in (a) to (c) of FIG. 12 , the several filter unit groups include two target filter unit sets. As shown in (d) to (g) of FIG. 12 , in the several filter unit groups, the color filter units of the same color are arranged symmetrically in the center.

Further, after the filter unit group shown in (a) to (g) in FIG. 9 and FIG. 12 is rotated 90° clockwise, the diagrams (a') to (h') in FIG. 12 are formed The filter unit group shown.

Similarly, the structure of the filter unit group obtained by performing other rotation transformations on the various filter unit groups in FIG. 9 and FIG. 12 , or by geometric transformation, also falls within the protection scope of the present application.

Referring to the filter unit groups shown in FIG. 10 and FIG. 11 , in the two embodiments, in each row of the filter unit group, two white filter units and two color filter units are arranged at intervals; In the two columns of the light unit group interval, 2 white filter units are arranged continuously, and the 2 color filter units are arranged separately; in the other two columns of the filter unit group interval, the 2 white filter units are arranged separately, and 2 color filter units are arranged in succession.

Comparing FIG. 10 and FIG. 11 , by exchanging the white filter unit and the color filter unit in FIG. 10 , the filter unit group in FIG. 11 can be obtained.

Based on the position of the white filter unit in FIG. 10 unchanged, the positions of the color filter units of different colors in FIG. 10 can be transformed to obtain various filter unit groups as shown in FIG. 13 .

Similarly, based on the position of the white filter unit in FIG. 11 unchanged, the positions of the color filter units of different colors in FIG. 11 can be transformed to obtain various filter unit groups as shown in FIG. 14 .

As shown in Figures 10, (a) to (c) in Figure 13, and Figures (d) to (g) in Figure 14, in these filter unit groups, the color filter units are arranged as four objects A collection of filter units.

As shown in Figure 11, Figures (d) to (g) in Figure 13 and Figures (a) to (c) in Figure 14, the several filter unit groups include two target filter unit sets, The two target unit sets include a green filter unit set.

Further, after the filter unit group shown in (a) to (g) in FIG. 10 and FIG. 13 is rotated 90° clockwise, the diagrams (a') to (h') in FIG. 13 are formed The filter unit group shown. After the filter unit group shown in (a) to (g) in FIG. 11 and FIG. 14 is rotated 90° clockwise, the filter unit group shown in (a') to (h') in FIG. 14 is formed. Filter unit group.

Similarly, filter unit group structures obtained by performing other rotational transformations or geometric transformations on various filter unit groups in FIGS. 10 , 11 , 13 and 14 also fall within the protection scope of the present application.

Specifically, as shown in (a') to (h') in Fig. 13 and (a') to (h') in Fig. 14 , in each column of the filter unit group, 2 white filter The light unit and the two color filter units are arranged at intervals; correspondingly, in the two rows separated by the filter unit group, the two white filter units are continuously arranged, and the two color filter units are arranged separately; in the filter unit group In the other two rows of the interval, the two white filter units are arranged separately, and the two color filter units are arranged consecutively.

The basic structure of the image sensor 200 in the present application and the schematic diagrams of the arrangement of various filter unit groups 211 are described above with reference to FIGS. 2 to 14 , and the image processing for the image sensor 200 of the present application is described below with reference to FIGS. 15 and 16 . method.

Fig. 15 shows a schematic flow chart of an image processing method. FIG. 16 shows a schematic diagram of an image through the image processing method in FIG. 15 .

As shown in Figure 15, the image processing method 10 includes:

S110: Perform sub-sampling on the image formed by the pixel unit array, and acquire a first sampling image including color pixel values and a second sampling image including white pixel values.

As an example, picture 1# in FIG. 16 is an image generated by a 4×4 pixel unit in the pixel unit array, and the 4×4 pixel units are the pixel units corresponding to the filter unit group shown in FIG. 10 above. .

In Figure 1# in FIG. 16 , the white pixel value 101 is the pixel value generated after the white pixel unit receives the white light signal, the white filter unit 201 is correspondingly set above it, and the red pixel value 102 is the red pixel unit receiving red light. The pixel value generated after the signal, the red filter unit 202 is correspondingly set above it; similarly, the green pixel value 103 is the pixel value generated after the green pixel unit receives the green light signal, and the green filter unit is correspondingly set above it 203, the blue pixel value 104 is the pixel value generated after the blue pixel unit receives the blue light signal, and the blue filter unit 204 is correspondingly disposed above it.

Picture 2# in Fig. 16 is the first sampling picture after sub-sampling the red pixel value, green pixel value and blue pixel value in picture 1#, picture 3# in Fig. The second sampled image after sub-sampling the white pixel values. In the first sampling map and the second sampling map, the relative positional relationship of pixel values is consistent with the relative positional relationship of pixel values in the original image 1#.

S120: Perform interpolation processing on the first sample image and the second sample image to obtain a first image and a second image.

As an example, picture 4# in FIG. 16 is the first image after interpolation processing of the first sample picture (picture 2#), specifically, the pixel value of the i-th row and the j-th column in the 4# picture can be expressed as X _ij , where X represents the color information of the pixel value. For example, the green pixel value in the 1st row and 1st column in the upper left corner can be expressed as G ₁₁ , and G ₁₂ and G ₂₁ can be obtained by interpolation according to G ₁₁ and G ₂₂ . Similarly, R ₁₄ and R ₂₃ can be obtained by interpolation according to R ₁₃ and R ₂₄ ; B ₃₁ and B ₄₂ can be obtained by interpolation according to B ₃₂ and B ₄₁ ; G ₃₃ and G ₄₄ can be obtained by interpolation according to G ₃₄ and G ₄₃ , so that according to the first The first image is obtained by interpolating pixel values in the sample map.

It can be seen that in the first image, half of the pixel values (as shown by the dotted box in the figure) retain the color information in the original image, which is more conducive to color restoration in subsequent image processing and improves image quality. And the interpolation processing process is relatively simple, which can simplify the algorithm in the image processing process and improve the efficiency of the image processing.

Picture 5# in FIG. 16 is the second image after interpolation processing of the second sample picture (picture 3#). The interpolation process may use any interpolation algorithm in the prior art, which is not limited in the embodiments of the present application. . It can be understood that, because the second sampling map and the second image are not used to represent the color information in the image, but are used to improve the brightness of the image. In the embodiment of the present application, white pixel values are evenly distributed in the second sampling image, and white pixel values are retained in the horizontal direction, vertical direction and oblique 45° direction of the second sampling image, so it can be obtained more accurately The full-resolution second image is beneficial to improve the resolution and image quality of the final image.

S130 : Re-mosaic processing is performed on the first image to obtain a third image.

As an example, after image 4# in FIG. 16 undergoes remosaic processing, the third image shown in image 6# is obtained. It can be seen that the third image is a data image in Bayer format. In the embodiment of the present application, the process of rearranging the mosaic may adopt any rearrangement method in the prior art, which is not specifically limited in the embodiment of the present application.

Since the third image after re-mosaic processing is the Bayer format that is commonly used in the field of image processing, the third image can be applied to more types of image signal processors (ISPs), so that the image sensor in this application can Adapt to more ISPs and apply to more application scenarios.

S140: Fusion of the third image and the second image to obtain an optimized color image.

Optionally, the resolution of the third image is the same as the resolution of the second image.

As an example, the optimized color image 7# is obtained by image fusion of 5# and 6# in Figure 16. The color image after image fusion can better maintain the brightness information while ensuring the color information of the image, thereby It can effectively improve the image quality in low light conditions. At the same time, in the frequency domain, the luminance and chrominance information of the color image are not easily overlapped, so moire fringes are not easily generated.

It can be understood that the above-mentioned image processing method 10 may be implemented by a processor or a processing circuit, in other words, optionally, the above-mentioned image sensor may further include a processing unit for implementing the above-mentioned image processing method 10 .

In addition to the image sensor 200 provided in the above-mentioned embodiments of the application, the present application further provides an electronic device, and the electronic device may include the image sensor 200 in any of the above-mentioned embodiments.

The electronic device can be any electronic device with an image acquisition function. For example, it can be a mobile terminal such as a mobile phone and a computer, or a photographing device such as a camera or a video camera, or an automatic teller machine (Automated Teller Machine, ATM), etc. etc., the application does not limit the application scenarios of the electronic device.

Optionally, the above-mentioned processing unit for executing the image processing method 10 may not be located in the image sensor 200, but may be located in the processing unit in the electronic device where the image sensor 200 is located. For example, if the electronic device is a mobile phone, the processing unit It may be an image signal processing unit in a processor in a mobile phone, or the processing unit may also be a separate image signal processing chip in a mobile phone, and the specific hardware form of the processing unit is not limited in this embodiment of the present application.

It should be understood that the processor or processing unit in this embodiment of the present application may be an integrated circuit chip, which has a signal processing capability. In the implementation process, each step of the above method embodiments may be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA) or other programmable Logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

It can be understood that the image sensor of the embodiment of the present application may further include a memory, and the memory may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory. It should be noted that the memory of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that make contributions to the prior art or the parts of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (22)

1. An image sensor, comprising:

a filter cell array including a plurality of filter cell groups, each of the plurality of filter cell groups including 4 × 4 filter cells; wherein, in the 4 × 4 filtering units, each row, each column and each diagonal line include 2 white filtering units and 2 color filtering units;

the pixel unit array comprises a plurality of pixel units, the pixel unit array is positioned below the light filtering unit array, and the plurality of pixel units in the pixel unit array correspond to the plurality of light filtering units in the light filtering unit array one by one.

2. The image sensor according to claim 1, wherein the filter unit group includes first, second, and third color filter units of different colors, and a number of the first color filter units is equal to a sum of numbers of the second and third color filter units.

3. The image sensor according to claim 2, wherein in the filter unit group, the number of the second color filter units is equal to the number of the third color filter units.

4. The image sensor of claim 3, wherein in the set of filter units, 2 color filter units in each row are different in color, and 2 color filter units in each column are different in color.

5. The image sensor according to claim 4, wherein in each row of the filter unit group, 2 white filter units are disposed at intervals from 2 color filter units;

in each column of the filter unit group, 2 white filter units are continuously disposed, and 2 color filter units are continuously disposed.

6. The image sensor as claimed in claim 5, wherein the color filter cells of the same color in the filter cell group are arranged in a central symmetry.

7. The image sensor of claim 5, wherein there are 2 sets of target filter cells in the middle two rows of the set of filter cells, the sets of target filter cells comprising 2 same color filter cells arranged at a common vertex angle.

8. The image sensor of claim 7, wherein in the set of filter units, the white filter units are respectively located in a first row, a second column, a first row, a fourth column, a second row, a second column, a second row, a fourth column, a third upper column, a third row, a third column, a fourth row, a first column, a fourth row, a third column;

the second color filter units are respectively positioned in a first column of a second row and a second column of a third row;

the first color filter units are respectively positioned in a first row and a first column, a second row and a third column, a third row and a fourth column, and a fourth row and a second column;

the third color filter units are respectively positioned in the third column of the first row and the fourth column of the fourth row.

9. The image sensor according to claim 4, wherein in each row of the filter unit group, 2 white filter units are disposed at intervals from 2 color filter units;

in two rows of the filter unit group at intervals, 2 white filter units are continuously arranged, and 2 color filter units are separately arranged; in the other two columns of the filter unit group interval, 2 white filter units are separately arranged, and 2 color filter units are continuously arranged.

10. The image sensor of claim 9, wherein the middle two columns of the set of filter units are provided with 2 sets of target filter units comprising 2 color filter units of the same color arranged at a common vertex angle.

11. The image sensor of claim 9, wherein there are 4 sets of target filter cells arranged in the set of filter cells, and the set of target filter cells comprises 2 color filter cells of the same color arranged at a common vertex angle.

12. The image sensor of claim 11, wherein in the set of filter units, the white filter units are respectively located in a first row, a second column, a first row, a fourth column, a second row, a first column, a second row, a third column, a third row, a third column, a fourth row, a second column, a fourth row, a fourth column, and a fourth column;

the second color filter units are respectively positioned in a third column in the first row and a fourth column in the second row;

the first color filter units are respectively positioned in a first row and a first column, a second row and a second column, a third row and a fourth column, and a fourth row and a third column;

the third color filter units are respectively positioned in the third row and the first column and the fourth row and the first column.

13. The image sensor according to any one of claims 2 to 12, wherein the first color filter unit, the second color filter unit, and the third color filter unit are configured to pass light signals of three colors respectively, and the wavelength bands of the light signals of the three colors cover a visible light wavelength band.

14. The image sensor of claim 13, wherein the first color filter unit, the second color filter unit, and the third color filter unit are respectively three colors of red, green, blue, cyan, magenta, and yellow.

15. The image sensor of claim 14, wherein the first color filter unit is a green filter unit, the second color filter unit and the third color filter unit are a red filter unit and a blue filter unit, respectively.

16. The image sensor of any one of claims 1 to 12, further comprising:

the micro-lens array comprises a plurality of micro-lenses, is positioned above the light filtering unit array and is used for converging light signals returned by a shooting object to the light filtering unit array, wherein the plurality of micro-lenses in the micro-lens array correspond to the plurality of light filtering units in the light filtering unit array one by one.

17. The image sensor according to any one of claims 1 to 12, wherein pixel values of color pixel units in the pixel array are used to generate first image data of the photographic subject, pixel values of white pixel units in the pixel array are used to generate second image data of the photographic subject, and the first image data and the second image data are used to synthesize a target image of the photographic subject;

the white pixel unit is a pixel unit corresponding to the white filter unit, and the color pixel unit is a pixel unit corresponding to the color filter unit.

18. The image sensor according to claim 17, wherein pixel values of color pixel units in the pixel array are used to generate an intermediate image through interpolation processing, the intermediate image being used to generate the first image data in bayer format through demosaic processing.

19. The image sensor according to claim 18, wherein 2 pixel values among the 2 x 2 pixel values of the intermediate image are original pixel values of the color pixel unit, and the other 2 pixel values are pixel values obtained by interpolation processing.

20. The image sensor of claim 17, wherein the first image data and the second image data are of the same resolution.

21. The image sensor of any one of claims 1 to 12, wherein the image sensor is a Complementary Metal Oxide Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

22. An electronic device, comprising:

the image sensor of any one of claims 1 to 21.

Images