+

WO2025046235A1 - Procédé de réduction de variabilité d'image dans des changements d'éclairage globaux et locaux - Google Patents

Procédé de réduction de variabilité d'image dans des changements d'éclairage globaux et locaux Download PDF

Info

Publication number
WO2025046235A1
WO2025046235A1 PCT/GB2024/052253 GB2024052253W WO2025046235A1 WO 2025046235 A1 WO2025046235 A1 WO 2025046235A1 GB 2024052253 W GB2024052253 W GB 2024052253W WO 2025046235 A1 WO2025046235 A1 WO 2025046235A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
regions
colour
shadow
data
Prior art date
Application number
PCT/GB2024/052253
Other languages
English (en)
Inventor
Hadi MABOUDI
Original Assignee
Opteran Technologies Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Opteran Technologies Limited filed Critical Opteran Technologies Limited
Publication of WO2025046235A1 publication Critical patent/WO2025046235A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image

Definitions

  • Shadows stemming from diverse light sources, introduce discrepancies in colour, texture, and shape, leading to inconsistencies in conventional visual features across different scenarios. These inconsistencies hinder the precise matching and recognition of locations, thereby compromising overall navigation effectiveness. It should be acknowledged that improved accuracy in recognition in this case does not require true colours only consistent colours. [0008] To address this concern, a process to detect and remove shadows is also needed in the context of colour constancy in order to mitigate the adverse effects of shadows on visual- based navigation systems. Shadows, cast by objects and structures under varying lighting angles and intensities, profoundly alter scene and object appearances. This alteration introduces disparities in the visual data captured by a robot's sensors, making it difficult to accurately match features across diverse lighting conditions.
  • present disclosure provides a bio-inspired solution to the colour constancy challenge, for example, in the use case of matching the overlap between two cameras in Opteran Development Kit (ODK) by adjusting the colour distribution of these cameras.
  • the solution encompasses, amongst other features described herein, an algorithm/model for stable colour appearance and high colour discrimination that is inspired by the visual systems of humans and mainly insects and is tailored to the functional features of the ODK camera system.
  • the model underlying the solution integrates three mechanisms: Chromatic adaptation model, LMC model (a model of lamina monopolar cells in insect visual lobe), and Colour opponent coding, which are further described in the following sections.
  • the bio-inspired solution to the colour constancy challenge also comprises a process to estimate the local illumination within these regions of a partitioned input image.
  • the approach relies on three key components: Selective attention mechanism, Gray-edge hypothesis, and colour normalisation. These three components are integrated into a single algorithm/model to reduce local illumination.
  • PCT application claiming priority from GB2313382.0 072153.00049
  • present disclosure provides, in conjunction with illumination correction as described herein, another solution for image processing that is configured to detect and accurately remove shadows from the input image that arise from varying lighting conditions. These shadows can introduce inconsistencies in image features and significantly impact accurate perception.
  • Shadow detection and removal thus play a critical role in enhancing the process of visual place recognition and localization for applications such as robotic navigation.
  • the proposed solution integrates two mechanisms: Shadow detection and Shadow removal, as described herein, is purposed to fix true colours and textures obscured by shadows inherent in the input image, making a valuable addition to the Opteran Vision framework.
  • the present disclosure provides a method or a computer- implemented method for processing images based on colour constancy removing illumination from the images, the method comprising: obtaining an image in a first colour space; converting the image to data in a second colour space; transforming the data using chromatic adaptation; performing a first normalisation on the transformed data, wherein the first normalisation comprises applying a dynamic spatial filtering technique to adjust the transformed data based on light intensity; applying a set of filters to the normalised data, wherein the set of filters is convoluted based on the normalised data in relation to the image; performing a second normalisation on the filtered data to obtain an illumination estimation of the image in relation to the filtered data; and outputting the normalised data from the second normalisation, wherein the normalised data maintains colour constancy based on the illumination estimation, removing the illumination from the normalised data.
  • the present disclosure provides a method or computer- implemented method for reducing effect of illumination on images, the method comprising: receiving an input image; partitioning the input image into a plurality of regions; analysing the plurality of regions based on colour information and spatial position of pixels in each region; selecting from the plurality of regions a subset of regions that are influenced by an illuminant based on the analysis; identifying coloured edges for at least said subset of regions; extracting the colour information from the coloured edges; decomposing reflectance and illumination components of the input image using the extracted colour information; correcting illumination of the input image based on the decomposed reflectance and illumination components; outputting an image with illumination corrected.
  • the present disclosure provides a method or computer-implemented method for providing a shadow-free image, the method comprising: receiving an input image PCT application claiming priority from GB2313382.0 072153.00049 in the first colour space, wherein the input image comprises at least one shadow region; generating shadow region masks for said at least one shadow region; removing shadow from shadow regions of the input image based on the shadow region masks; and outputting a shadow-free image.
  • the present disclosure provides an apparatus for processing images to maintain colour constancy of the images, the apparatus comprising: at least one model configured to perform steps according to the first, second, and/or third aspect as well as any of the aspects described herein.
  • the present disclosure provides a system for processing images to establish colour constancy for an image by removing illumination from the image, the system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform the first, second, and/or third aspect as well as any of the aspects described herein.
  • the methods described herein may be performed by software in machine-readable form on a tangible storage medium e.g. in the form of a computer program comprising computer program code means adapted to perform all the steps of any of the methods described herein when the program is run on a computer and where the computer program may be embodied on a computer-readable medium.
  • tangible (or non-transitory) storage media examples include disks, thumb drives, memory cards etc. and do not include propagated signals.
  • the software can be suitable for execution on a parallel processor or a serial processor such that the method steps may be carried out in any suitable order, or simultaneously.
  • the methods or computer-implemented methods herein may be described in terms of one or more models. It is thus appreciated that the term "method” may be interchangeably recited as "model” throughout the disclosure where appropriate, though in some cases, a model may not correspond to a single method necessarily but may instead be incorporated as part of said method. [0022] This application acknowledges that firmware and software can be valuable, separately tradable commodities.
  • Figure 1a is a flow diagram of a model architecture for image processing according to aspects of the disclosure
  • Figure 1b is a flow diagram of the image processing based on the model architecture according to aspects of the disclosure
  • Figure 2 is a pictorial diagram of a model application to the Spyder Checkr (Macbeth ColourChecker chart) under different illumination according to aspects of the disclosure
  • Figure 3 is a pictorial diagram of a comparison of distributions of pixel intensity between the original and corrected images according to aspects of the disclosure
  • Figure 4 is a pictorial diagram of Fano factor (relative variance) of the selected patches according to aspects of the disclosure
  • Figure 5 is a pictorial diagram of colour distance between colour boards in an environment dataset (set 1 in relation to Figure 9) according to aspects of the disclosure
  • Figure 6 is a
  • Figure 19b is a flow diagram of the image processing based on said another model architecture according to aspects of the disclosure.
  • Figure 20 is a pictorial diagram of exemplary input and output of the image processing based on said another model architecture according to aspect of the disclosure; PCT application claiming priority from GB2313382.0 072153.00049 and
  • Figure 21 a pictorial diagram of exemplary input and output of the image processing for a different input image.
  • Common reference numerals are used throughout the figures to indicate similar features. Detailed Description [0050] Embodiments of the present invention are described below by way of example only.
  • Colour constancy is an important feature of colour perception. It refers to the ability to perceive the colour of objects, irrespective of varied light sources. This ability is reported in different species such as humans, fish, and bees. Colour constancy allows different species to perceive the colour of objects relatively constantly under varying illumination conditions or identify objects irrespective of illumination conditions. For example, an object will appear green to us during midday, when the main illumination is white sunlight, as well as at sunset, when the main illumination is red.
  • the evaluation of the present invention revealed that the model(s) proposed herein reduced the intensity PCT application claiming priority from GB2313382.0 072153.00049 variation of colour patches illuminated (both locally and globally) by different natural and artificial light sources.
  • the present invention enhances the camera’s capacity in terms of colour discriminability by increasing the colour distance between objects. It is useful for the challenge of matching the overlap between two cameras in ODK by adjusting the colour distribution of the cameras as shown according to the figures.
  • the present invention exhibits some potential advantages for the Opteran Vision Framework by upgrading the colour coding in the context of image processing.
  • One aspect of the present invention is a biologically plausible algorithm to estimate light illumination and suggest a stable colour encoding mechanism for visual object recognition in autonomous robots. It takes the inspiration for an accurate and fast colour encoding algorithm from the individual neuron and neural network level of human and insect visual systems.
  • a multi-layer neural network incorporating three of the visual mechanisms suggested for colour constancy; retina photo receptor adaptation, lateral normalisation between photo receptors and the center-surround spectral opponency in the early visual system (namely Figure 1a and 1b).
  • An overview of the proposed model follows, serving as an exemplary implementation or aspect of the present invention. It is understood that this aspect may be readily combined with other aspects of the model herein described.
  • the method, algorithms, and/or model described herein may comprise one or more steps for partitioning an input image, raw image, or training data/dataset into a plurality of partitions (this process is also referred to as image segmentation) before to be processed further as described in the present disclosure.
  • the method for image segmentation or partitioning an image may include but is not limited to thresholding, region growing, edge-based segmentation, clustering, histogram-based bundling, k-means clustering, watershed, active contours, ML-based segmentation using Convolutional Neural Networks, graph-based segmentation, and superpixel-based segmentation.
  • colour space refers to an abstract space with a specific organization of reproducible representations of colour. It is understood that colour space may PCT application claiming priority from GB2313382.0 072153.00049 be arbitrary, i.e., with physically realised colours assigned to a set of physical colour swatches with corresponding assigned colour names or structured with mathematical rigor.
  • an image is converted to a Long-Medium-Short ( ⁇ ⁇ ⁇ ) colour space in order to simulate the ⁇ , ⁇ , ⁇ cones in the human eye.
  • the gamma correction applied to the colours (Equation 1) is removed to generate the linear ⁇ ⁇ ⁇ of the image.
  • Gamma correction herein refers to a nonlinear operation used to adjust the brightness and contrast of an image. It involves applying a non-linear mapping to the pixel values of an image to compensate for the inherent non-linear response of display devices, such as monitors and televisions.
  • Gamma correction can be used to control the brightness of the image. It helps ensure that the perceived brightness and contrast of the image remain consistent across different devices and viewing environments.
  • raw image includes gamma correction.
  • the gamma correction of the raw image can be removed, resulting in an image with each ⁇ ⁇ ⁇ value of a processing colour transformed into equivalent linear ⁇ ⁇ ⁇ colour space.
  • Gamma correction transforms colour intensities from the physical world into a more uniform arrangement for humans.
  • Equation 5 we implement the processing of LMC cells by applying a single transformation, ⁇ to the photoreceptors’ response (Equation 5). This transformation normalises the photoreceptor’s output in respect to the population activity of all at the same colour channel photoreceptor.
  • Double Opponent model we finally propose a model of opponent coding according to the single-opponent and double-opponent cells in the human and insect visual system (see Double Opponent model).
  • Double Opponent model we can estimate the global illumination of the input image illuminated by external light sources. This allows us to correct a large range of colours in images with different light conditions due to varying natural light sources and changing artificial light.
  • the corrected image in LMS colour space is transformed into ⁇ ⁇ ⁇ ⁇ space by applying the inverse transformation ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ and Equation 2.
  • Chromatic adaptation in the context of colour constancy is defined as the ability of animal colour perception to adjust retina sensitivity in response to changes in light sources.
  • Chromatic adaptation is a technique for explaining colour constancy based on the animals’ ability, including humans, to adjust the sensitivity of their photoreceptors to changes in light PCT application claiming priority from GB2313382.0 072153.00049 sources.
  • Chromatic adaptation is closely related to the adjustment of cone sensitivity which happens in the retina when the illumination changes.
  • chromatic adaptation technique applies an adapted version of Von Kries’ chromatic adaptation, which is based on the ⁇ ⁇ ⁇ cones’ sensitivity response function ( ⁇ ⁇ ⁇ colour space).
  • Document describing the general approach of Von Kries’ chromatic adaptation (Luo, M. Ronnier. "A review of chromatic adaptation transforms.” Review of Progress in Coloration and Related Topics 30 (2000): 77-92.) is hereby incorporated herein by reference.
  • each cone increases or decreases responsiveness based on the illuminant spectral energy.
  • Each cone cell increases or decreases spectral sensitivity to adapt to new illumination from the previous illumination to perceive objects as constant colours.
  • Von Kries adaptation transforms one illuminant to another to maintain white colour consistency in both systems. Further extensions of the Von Kries’ adaptation have also been introduced. For example, multiple CAT transformation and the Bradford model are popular. Here, we implement some of these models by application of the matrix transformation shown in Implementation of Chromatic adaptation transform.
  • b) A model of lamina monopolar cells ( ⁇ ⁇ ⁇ model): It has been hypothesized that the dendrites of insect lamina monopolar cells integrate visual information from neighbouring photo receptors. Therefore, they contribute to the spatial summation of visual information and normalise the responses of photoreceptors by using temporal coding. This process also enhances a neuron’s information capacity.
  • the outputs of photo receptors are propagated in the way of colour opponency via single-opponent cells in retinal ganglion layers and LGN and double-opponent cells in V1 of the visual cortex.
  • the single-opponent cells process the colour information through the centre-surround structure of their receptive fields (RF).
  • RF receptive fields
  • the RF structure of single-opponent cell type II is shaped as two Gaussian functions with red-on in the centre and green-off in the surround.
  • Double-opponent cells compute both colour opponency and spatial opponency. It has been suggested that double-opponent cells are potentially the basis of illumination encoding. Interestingly, double-opponent cells were also observed in goldfish and honeybees. In the present disclosure we suggest a simple form of S-DO Opponent model for colour constancy that improves the estimation of global illumination combined with the two other mechanisms of colour constancy described above. Documents (Conway, Bevil R., et al. "Advances in color science: from retina to behavior.” Journal of Neuroscience 30.45 (2010): 14955-14963; Shapley, and Conway, Bevil R., David H. Hubel, and Margaret S. Livingstone.
  • a method to evaluate colour constancy models is to use an angular error that measures angular distance between the estimated illumination and the ground-truth illumination. Since it is difficult to collect data with corresponding ground truth illuminants, we decided to use a different approach to quantitatively analyze the model's performance. In this way, pre-selected patches of pixels located within sections on the ColourChecker illuminated under varying light conditions are analysed before and after applying out model. The model was evaluated with three different datasets of the colour constancy task: Spyder Checkr 24 dataset, Environment dataset, and Lab dataset.
  • the Spyder Checkr 24 from datacolour is a standard for colour calibration.
  • This dataset contains 34 images of the colour chart illuminated by different natural and artificial lights.
  • the Environment dataset has raw-RGB pair images which were captured under varying lighting conditions (daily natural lights and lamp artificial lights) (see Data Collection).
  • the Lab dataset includes RGB images of the lab space also captured under varying light conditions and sources, as shown according to the figures. [0067]
  • the Fano factor was PCT application claiming priority from GB2313382.0 072153.00049 calculated for each channel of selected patches. It measures the relative variance of the colour intensity and shows the extent of variability in relation to the mean of population.
  • the distance between two colours allows for a quantified analysis of how far apart two colours are from one other.
  • the Delta E ( ⁇ ⁇ ) metric is used to measure the degree of colour changes over different illumination and verify the improvement of the colour discrimination of our model/algorithm. It evaluates the distance in the CIELab colour space and represents the relative perceived magnitude of colour difference. The larger ⁇ ⁇ , the greater distance between the colours.
  • Step 1) Estimate the global illuminant of an image; and Step 2) Convert the image to destination illuminant using the illuminant obtained in Step 1.
  • Step 1) Grey world model: To estimate global illumination, we used the grey world method. The grey world assumption is a simple method which assumes an image contains objects with different reflective colours, that are uniform from minimum to maximum intensities and therefore averaging all pixels gives a grey colour.
  • Illumination estimation (calculated or derived data representation of illumination present in an image) is calculated by averaging all pixel values for each channel, the illumination estimation would give an average colour which provides an approximation of the illuminant colour. For an image with equal representation of colours, illumination estimation gives an average colour grey.
  • LMC model It has been proposed that neighboring photo receptors in flies are laterally connected by large monopolar cells (LMC) which improves the contrast encoding and contributes to spatial summation of the visual information. In fact, LMC cells cause a uniform distribution of the photoreceptor's response to the visual input by sending feedback to the photoreceptors and changing their temporal responses. It is therefore appreciated that documents (Laughlin, Simon. "A simple coding procedure enhances a neuron's information capacity.” Zeitschrift für Naturforschung c 36.9-10 (1981): 910-912; and Stöckl, Anna Lisa, David Charles O’Carroll, and Eric James Warrant.
  • temporal coding is a phenomenon where the timing of a pulse signal is indicative of a value associated with the pulse signal.
  • the temporal responses matrix is a sparse matrix which works similarly to temporal coding: each position along the first dimension of the matrix corresponds to one pixel, and the value of each pixel is indicated by its position along the second dimension of the matrix.
  • the first dimension of the matrix has a length equal to the product ⁇ ⁇ of the pixel array dimensions ⁇ and ⁇ of the image
  • the second dimension of the matrix has a length ⁇ corresponding to a range of possible pixel values ⁇ ( ⁇ , ⁇ ) per colour channel.
  • each row of the matrix ⁇ represents the activity of each photoreceptor such that the peak of activity moves to the left or right based on PCT application claiming priority from GB2313382.0 072153.00049 the intensity of the pixel.
  • ⁇ ⁇ ⁇ ⁇ [ ⁇ ⁇ ⁇ ⁇ ⁇ ]′ (Equation 5)
  • the superscript ' represents the matrix transpose.
  • ⁇ ⁇ is in dimension mn x 1 and can be reshaped to the image size m x n as the final output in this stage.
  • Dynamic spatial filtering refers to a form of normalisation of data in a colour space, where the data is representative of the receptors’ representation of the visual environment. Dynamic spatial filtering effectively simulates and modifies the response of the receptors that are exposed to high light intensity and dim light intensity, where these receptors respectively are influenced by the lateral inhibition and spatial summation of neighboring receptors.
  • Double Opponent model the first stage of colour-sensitive cells are single-opponent cells in the LGN which encode colour information within their centre-surround receptive fields (RFs) in the way of red-green, blue-yellow and black-white opponency.
  • RFs centre-surround receptive fields
  • ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ represent the red-green, blue-yellow and achromatic single opponent cells, respectively.
  • the sign '+' and '-' denote the excitation and inhibition.
  • ⁇ controls the relative contribution of RF surround.
  • Data colour dataset The small colour chart data was collected on a Raspberry Pi Camera V2 with the same 8 megapixel Sony IMX219 (8 MP Camera Sensor) image sensor as the ODK. This singular camera setup has a fixed focus (not fish eye) lens. This was positioned in front of the Datacolour SpyderCheckr24: 24 Colour Patch and Grey Card for camera calibration against the window, for natural illumination. Some images had the addition PCT application claiming priority from GB2313382.0 072153.00049 of yellow lamp or white LED light to increase the illumination variation.
  • Pi camera A repeat of this image collection was taken in consistent light conditions with the Pi camera.
  • the colours included were; back, white, grey, red, yellow, green, blue and pink (visible according to at least one of the figures).
  • the ODK was configured to save the raw camera images (not 4pi etc) to a rosbag every three minutes. To collect a variation of lighting conditions, the experiment took place next to a window, allowing natural light to illuminate the scene. Additionally, the lights in the office were on/off at random intervals. To add structured light and shadows to the scene, three lamps were placed in the setup. The lamps were smart bulbs, controlled through the app to turn on and off every 3/7/11 minutes, respectively. Two lamps were set to yellow light and one white light to add variation to the scene. The lamps were positioned to create shadows, bright spots and patches of illumination creating a difficult scene.
  • the data colour chart shown in the figure contains a range of spectral reflectance colour patches that represents most possible intensity ranges that are suitable for many uniform illumination conditions. Since the colour checker chart contains uniform colour ranges, we can use this chart to estimate the PCT application claiming priority from GB2313382.0 072153.00049 illumination of an image used at least in part by the present invention.
  • Another aspect of the present invention is a method that functionally utilises the Gray- edge hypothesis, which posits that the average edge difference within a scene window is achromatic, to estimate the local illumination within these regions of a partitioned input image.
  • the resultant estimation of the local illumination can be used to correct the illumination of the input image using retinex-based correction, effectively reducing local illumination, which leads to precise colour correction for the input image.
  • the method tackles the challenges posed by local illumination resulting from multiple light sources.
  • our proposed solution demonstrates promising results in mitigating the impact of local illumination and improving colour constancy across various scenarios as can be seen based on the results shown in the figures.
  • this method can be combined with any of the other models/approaches described herein. In combination, these methods/models/approaches of the present invention are empowered to address a wide range of application requirements and effectively handle the complexities of lighting conditions encountered in real-world environments.
  • this other aspect of the present invention contributes valuable insights into the field of colour constancy and provides a robust methodology for enhancing colour accuracy in images affected by local illumination.
  • Present method works to further refine and may be extendable to handle additional lighting challenges and broaden its applicability across various domains in computer vision and robotics.
  • Local Illumination [0092] Further improving colour constancy of images, the present disclosure provides another method of image processing in conjunction with any and all herein described methods. This method addresses at least some of the aforementioned challenges, such as those posed by multiple light sources, by removing local (colour) illuminations from an input image.
  • this approach is proposed to address the local illumination of an input image and may be used in conjunction with any of the approaches described herein, further improving colour constancy.
  • the output of the local illumination approach can be used as the image in the first colour space used for removing global illuminations from a raw PCT application claiming priority from GB2313382.0 072153.00049 image.
  • the approach relies on three key components: I) Selective attention mechanism, II) Gray-edge hypothesis, and III) colour normalisation. These three components are integrated into a single algorithm/model to reduce local illumination. The combined algorithm represents a significant reform of previous research, offering a new and highly efficient process that yields improved results as shown according to the figures.
  • k-means is an unsupervised machine learning algorithm that aims PCT application claiming priority from GB2313382.0 072153.00049 to partition a dataset into a predetermined number (k) of distinct, nonoverlapping clusters.
  • k-means is used to group similar pixels together based on their colour or intensity values, more likely with the same illumination.
  • a subset of regions is selected whereby considering both the colour information (that is represented in numerical values that describe the intensity or magnitude of different colour channels) and the spatial position of pixels (or their location within an image or a coordinate system). This achieves better segmentation results.
  • the algorithm will assign each pixel to one of the k clusters based on the similarity of their colour value.
  • the number of patches is only a free parameter of the model that can be adjusted based on the size of the image input.
  • this mechanism can be improved by selective attention such as the saliency map as opposed to using a clustering algorithm, i.e. k-means.
  • Integration of saliency-based selective attention mechanisms can be used to refine and improve model performance. Rather than analysing the Gray-Edge Hypothesis in all regions sequentially, the use of saliency maps can guide our attention to focus on the most informative and visually salient regions (here referred to as salient regions) of the image. By leveraging saliency information, we can prioritize regions that are more likely to contain reliable colour information and are less affected by local illumination variations induced by multiple light sources.
  • edges edges
  • Canny edge detection algorithm is a widely used technique in image processing for edge detection. This algorithm comprises several steps to identify colour changes within an image. First, the image is smoothed to reduce noise. Then, the gradient magnitude and orientation are calculated. Non-maximum suppression is applied to thin the edges, and the most robust edges are selected based on thresholding. However, for testing purposes, we can explore an alternative approach using simple gradient operators to estimate image gradients.
  • ⁇ ⁇ ( ⁇ , ⁇ ), ⁇ ⁇ ( ⁇ , ⁇ ) and ⁇ ⁇ ( ⁇ , ⁇ ) are three colour channel of the edges within the attended region.
  • Another aspect of the present invention is a method for removing shadows from an input image.
  • the method comprises two algorithms: shadow detection algorithm and shadow removal algorithm.
  • the two algorithms work in tandem, where the first algorithm detects shadows and creates a shadow mask, and the second algorithm eliminates the detected shadows based on the shadow mask.
  • the final output is a shadow-free image.
  • shadows in the input image can lead to reduced visibility, decreased contrast, and altered colour distribution, which can impact the interpretability and aesthetic quality of the image, causing problems when the image is being used in applications of computer vision and robotics.
  • shadow detection and removal may be used, i.e., medical imaging or remote sensing, removing shadows can be an important step before any accurate analysis or measurements can be performed.
  • Shadow detection and removal may help reduce the number of false positive objects being identified during the above applications.
  • Shadow pixels may in fact disrupt how human colour constancy can be maintained. It is demonstrated that a shadow-free image tends to be robust and suitable for further applications, serving as input for illumination according to any aspect described herein in order to maintain human colour constancy.
  • the shadow detection and removal algorithm may be suitably performed on the output of the image processing process described herein.
  • Shadow-free image [00109] Further improving image quality, the present invention may include a shadow detection and removal process. The result of which is a shadow-free image.
  • One advantage of removing shadows or having a shadow-free image is to enhance the visibility of the input image. Simply put, shadows can obscure details in an image, making it difficult to distinguish PCT application claiming priority from GB2313382.0 072153.00049 objects or features.
  • Shadows By removing shadows, important details of the image become clearer and more visible, improving image quality.
  • removal of shadow from the input image also improves the image contrast. For example, shadow pixels of an image often cause a decrease in contrast between different parts of the image. Removing shadows can help restore a more balanced contrast, making objects stand out against their backgrounds. Shadows can also introduce colour variations and shifts due to changes in lighting conditions. The removal results in a more consistent colour representation across the image for maintaining colour constancy of the image or its natural appearance. For example, removing shadows can result in a more natural and evenly illuminated image that closely resembles how the scene might appear under uniform lighting conditions.
  • a shadow-free image has obvious advantages, making the shadow-free image suitable for applications and can be used in conjunction with other algorithms described herein to establish correct colour constancy, thereby enhancing the quality and utility of the raw images from one or more cameras, eliminating the negative effects of shadows, leading to improved visibility, contrast, colour consistency, and aesthetics.
  • the following steps may be carried out to obtain the shadow-free image. It is understood that the algorithm is not limited to only these steps and may encompass other steps described herein.
  • RGB image is the input to the algorithm. Convert RGB image to Lab colour space, extract L channel. Generate and smooth histogram of L channel. Identify local minima on smoothed histogram.
  • both shadowed and lit regions are segmented into smaller units, each assumed to exhibit a single texture pattern.
  • the similarity assessment among segments incorporates diverse visual features like texture density, frequency, entropy, and inter-segment centroid distance.
  • the objective of histogram matching is to eliminate the shadow's impact on the original underlying texture. Following this procedure for all shadow segments results in a shadow-free image.
  • Integrating shadow detection and removal methods into visual place recognition and localization frameworks holds promise for enhancing robotic navigation. By bolstering visual feature consistency and mitigating shadow effects, these methods may improve navigation system accuracy and dependability.
  • Our proposed algorithm/model comprising at least two algorithms: a) shadow detection algorithm and b) shadow removal algorithm.
  • the two algorithms work in tandem, where the first algorithm (steps 1 to 9) detects shadows from an RBG image serving as input and outputs a shadow mask.
  • the shadow removal algorithm (steps 1 to 11) eliminates the detected shadows based on the shadow mask.
  • the final output from both algorithm is a PCT application claiming priority from GB2313382.0 072153.00049 shadow-free image of the original.
  • Shadow detection algorithm carries out the following steps: 1. Input: RGB image ⁇ 2. Extract the dimensions of the image (h, ⁇ ) 3. Convert ⁇ to the Lab colour space and extract the channels ⁇ , ⁇ and ⁇ 4. Generate a histogram of channel ⁇ 5. Smooth the histogram curve using a Gaussian window ⁇ 6. Identify the local minima points on the smoothed histogram 7.
  • Shadow region mask ⁇ ⁇ [00120] In another aspect of the Shadow Removal Algorithm, II) shadow removal algorithm carries out the following steps: 1. Input: RGB image ⁇ and shadow mask ⁇ ⁇ 2. Convert ⁇ to the ⁇ ⁇ ⁇ colour space and extract channels h, ⁇ and ⁇ 3.
  • Colour distance( ⁇ ⁇ ⁇ , ⁇ ) calculate the mean colour distance of colour channels ⁇ and ⁇ (in ⁇ ⁇ ⁇ colour space) for the ⁇ ⁇ ⁇ h and ⁇ ⁇ ⁇ h segments c.
  • Steps 7 and 8 are iteratively performed on all shadow segments. 10. Merge all colour-corrected segments ⁇ ⁇ , ⁇ and lit segments ⁇ ⁇ , ⁇ , into image ⁇ 11. Output: Shadow-free image ⁇ [00121] Specific steps outlined in the shadow removal and detection algorithm as above, as well as other algorithm(s) described herein, might vary depending on the circumstance or the context of the algorithm being used. Therefore, it is recognized that the various aspects are described herein for the present invention, where some of these aspects may not be restricted to the sequence steps as described, and certain steps may vary based on the specific application.
  • FIG. 1a is a flow diagram illustrating an example process 100 for achieving colour constancy according to the present invention.
  • Process 100 illustrates an algorithm for stable colour appearance and high colour discrimination that is inspired by the visual systems of humans and mainly insects and is tailored to the functional features of the ODK camera.
  • FIG. 1b is a flow diagram illustrating an example process 150 corresponding to the algorithm proposed in Figure 1a.
  • the flow diagram depicts the image processing method based on the model architecture according to aspects of the disclosure, which is based on colour constancy removing illumination from the images.
  • the image processing method may comprise the following steps 151 to 165.
  • step 151 the method obtains an image in a first colour space.
  • the first colour space may be a RGB colour space.
  • step 153 the method converts the image to data in a second colour space.
  • the second colour space is a LSM colour space.
  • step 155 the method transforms the data using chromatic adaptation, where said transforming the data using chromatic adaptation, further comprising: estimating an illuminant of the data using a grey world model; and converting the data to a destination illuminant using the illuminant. This may be done by applying one or more matrix transformations to the destination illuminant in accordance with one or more cameras for obtaining the image.
  • step 157 the method performs a first normalisation on the transformed data, wherein the first normalisation comprises applying a dynamic spatial filtering technique to adjust the transformed data based on light intensity.
  • the method applies a set of filters to the normalised data, wherein the set of filters is convoluted based on the normalised data in relation to the image.
  • the set of filters is representative of two layers of a visual system.
  • the set of filters may comprise at least one centre-surround structure representative of RFs encoding colour opponency.
  • the RFs may be red-green opponency, blue-yellow opponency, and achromatic opponency.
  • the set of filters may also comprise at least two filters positioned in series such that one of said at least two filters receives input from the other filter.
  • step 161 the method performs a second normalisation on the filtered data to obtain an illumination estimation of the image in relation to the filtered data.
  • step 163 the method outputs the normalised data from the second normalisation, wherein the normalised data maintains colour constancy based on the illumination estimation, removing the PCT application claiming priority from GB2313382.0 072153.00049 illumination from the normalised data.
  • the standard method may be used to transfer RGB to LMS colour space, where method converts the normalised data from the second normalisation to an image in the first colour space.
  • the method may further identify a data representation of receptor responses in relation to the transformed data when performing the first normalisation.
  • the data representation would comprise a uniform representation of the receptor responses across the image.
  • the data representation may also be applied to the transformed data, where the method integrates the data representation over a time period/frame and normalises the transformed data based on the integrated data representation using the dynamic spatial filtering technique representative of performing temporal coding adjustment for each receptor in respect of the light intensity.
  • the second normalisation may further comprise the step to correct the filtered data with an illumination vector generated using a pooling function, Equation 7, where the filtered data is divisible by the illumination vector.
  • FIG. 1 is a data neural computation of max over ⁇ ⁇ ⁇ ( ⁇ , ⁇ ) exhibits the output of double-opponent filters in the LMS space.
  • the method may receive a raw image from one or more cameras, by removing gamma correction from the raw image, the method obtains the input to step 151, where the method converts the raw image to said image in the first colour space.
  • Figure 2 is a pictorial diagram of a model application to the Spyder Checkr under different illumination according to aspects of the disclosure. Various examples are included in relation to the model application and the results are shown. These are examples of corrected images obtained from applying the model's components to sample colour charts.
  • the left column 201 of the diagram represents the examples of original images of colour charts under varying illumination, where images of various illumination are shown in the respective rows.
  • the global illuminations of the images were excluded by different components of the model and the mixture/combined model (last column 209).
  • the middle columns are intermediate results for chromatic adaptation (second column 203), LMC normalisation model (third column 205), and double-opponent model (fourth column 207) in accordance with Figures 1a and 1b.
  • Chromatic adaption 203, LMC 205, and opponent model 207 exhibit an synergic effect when combined shown as combined model 209.
  • the sample images (left column 201) were captured using a Raspberry Pi V2 Camera in different light conditions.
  • the final column 209 represents the corrected images after the exclusion of global illumination utilising the combined model.
  • the global illumination of the images was excluded by different components of the model and the mixture model. Because the colour condition of the Spyder Checkr on corrected images on the right side of the figure is almost constant, it is indicative that the variability of colour patches at the Spyder Checkr in the left side is significantly reduced (comparing colour patches of the Spyder Checkr in the left and right columns). Hence, the combined model produced colour charts with almost constant colour patches.
  • FIG. 3 is a pictorial diagram of a comparison of distributions of pixel intensity between the original and corrected images according to aspects of the disclosure.
  • the results of the model application in Figure 2 are quantified and illustrated.
  • the figure compares the PCT application claiming priority from GB2313382.0 072153.00049 distribution of pixel intensity between the original and corrected images.
  • Each row shows the distribution of intensity of red (left 301), green (middle 303) and blue (right 305) colour channels of the same selected patch from the original images (left blue) as labelled and the corrected images (right orange) as labelled.
  • FIG. 4 is a pictorial diagram of fano factors (relative variance) of the selected patches according to aspects of the disclosure.
  • the bar graphs show the Fano factor of selected patches which represents the normalised variability of each colour channel (red: left 401, green: middle 403, and blue: right 405) for 5 selected patches 411- 419 of the original images (blue bars) and corrected images by the model (orange bars). It indicates that the model reduced the variability of most of the colour channels of the selected patched.
  • FIG. 5 is a pictorial diagram of colour distance between colour boards in an environment dataset (set 1 in relation to examples of Figure 9) according to aspects of the disclosure.
  • the first row 501 exhibits an example of the original image 501a captured by the ODK camera and its corrected image 501b after being proceed by the model.
  • the original and corrected image in the second row are their respective distances.
  • the matrices in the second row 503 show the colour distance between 8 colour boards represented in the first row (left: original image 503a, right: corrected image 503b). The colour distance between each pair of colour boards is shown by an element of the matrix.
  • the third 505 and fourth 507 rows display the average colour distance (average of colour distance matrix) of 8 patches for all images in the dataset in accordance to Figure 12.
  • the box plots in the last row 509 i.e., summarise third and fourth rows
  • the colour distances between patches were measured for both original and corrected images using the Delta E ( ⁇ E) as explained in a previous section.
  • the matrices display the distances between all pairs of patches as shown.
  • the average colour distance between selected patches i.e., the average of the matrix of colour distance
  • the average colour distance between patches in the corrected images is nearly 2 times larger than the colour distance in the original images, a result which is explainable by the box plots as shown. Similar results were obtained using the dataset illustrated, for example, in Figure 12. From this, it is revealed that the present invention improves the separation between colours and potentially be able to enhance the colour discriminability of the ODK system.
  • FIG. 6 is a pictorial diagram showing an example of the model application for the image matching between two ODK cameras according to aspects of the disclosure.
  • the top row 601 shows images captured by the front and back IMX219 cameras which is established in the ODK system.
  • the bottom row 603 shows images after being corrected by the present invention.
  • the colour distribution of both images matched the colour distribution of the target image.
  • the present invention is able to undertake the matching of colour distribution between the front and back ODK cameras aside from improving colour discriminability. By doing so (undertake) help solve the challenge of overlapping in making the cylinder image from two ODK cameras by reducing the colour distance between the overlap region of the front and back images.
  • FIG. 7 is a pictorial diagram of Spyder Checkr or Macbeth ColorChecker chart according to aspects of the disclosure.
  • the Spyder Checkr is a datacolour chart that contains a range of spectral reflectance colour patches that represents the (most) possible intensity ranges that are suitable for many uniform illumination conditions.
  • a model application under different illumination is provided in the form of Spyder Checkr chart according to Figure 3.
  • FIG 8 is a pictorial diagram of an exemplary data collection setup scheme for obtaining the model results in relation to Figures 1 to 7.
  • the location of ODK camera 805, coloured objects 801, 807a/b, windows (natural lights) 803a/b and lamps (artificial lights) 809a/b/c are shown in the scheme.
  • the scheme setup comprises a radiator object 801 between two windows 803a/b with natural lights facing a plurality of box objects 807a/b.
  • a plurality of lamps 809a/b/c is provided as artificial lights.
  • Distances between the objects are PCT application claiming priority from GB2313382.0 072153.00049 shown accordingly.
  • the ODK camera is situated at the centre of the scheme.
  • Figures 9 and 10 are pictorial diagrams of example images from an environment dataset (set 1 and 2) according to aspects of the disclosure.
  • five samples of the images 901a, 903a, 1001a, 1003a were captured from the colour boards (Top: front camera 901, 1001, Bottom: back camera 903, 1003), using the ODK camera under different light illuminations (see the design of the Environment data in Spyder Checkr with respect to Figure 2).
  • the second row 901b, 903b, 1001b, 1003b shows the corresponding images corrected by applying the proposed/combined model.
  • Figure 11 is a pictorial diagram of a comparison of distributions of pixel intensity between the original and corrected images, Pi Camera according to aspects of the disclosure.
  • each column shows the distribution of intensity of red (left 1101), green(middle 1103) and blue(right 1105) colour channels of the same selected patch from the original images (blue, captured by the Pi Camera) and the corrected images (orange).
  • the colour distributions of red 1111, green 1113, blue 1115, yellow 1117, and white 1119 patches are ordered from the top to bottom rows.
  • Figure 12 is a pictorial diagram of the colour distance between colour boards in the Environment dataset (set 2) according to aspects of the disclosure and with respect to Figure 5.
  • the first row 1201 exhibits an example of the images captured by the ODK camera and its correction by the model.
  • the matrices in the second row 1203 show the colour distance between 8 colour boards represented in the first row 1201 (left: original image, right: corrected image).
  • the colour distance between each pair of colour boards is shown by an element of the matrix.
  • the third 1205 and fourth rows 1207 display the average colour distance (average of colour distance matrix) of 8 patches for all images in the dataset (set 2).
  • the box plots in the last column i.e., summarise third and fourth columns) reveal that the model increases the colour distance between colour boards.
  • FIG. 13 is a block diagram illustrating an example computing apparatus/system 1300 that may be used to implement one or more aspects of the present invention, apparatus, method(s), and/or process(es) combinations thereof, modifications thereof, and/or as described with reference to figures 1a to 12 and 14 to 21 and/or aspects as described herein.
  • Computing apparatus/system 1300 includes one or more processor unit(s) 1302, an input/output unit 1304, communications unit/interface 1306, a memory unit 1308 in which the one or more processor unit(s) 1302 are connected to the input/output unit 1304, communications unit/interface 1306, and the memory unit 1308.
  • the PCT application claiming priority from GB2313382.0 072153.00049 computing apparatus/system 1300 may be a server, or one or more servers networked together.
  • the computing apparatus/system 1300 may be a computer or supercomputer/processing facility or hardware/software suitable for processing or performing the one or more aspects of the system(s), apparatus, method(s), and/or process(es) combinations thereof, modifications thereof, and/or as described with reference to figures 1a to 12 and 14 to 21 and/or aspects as described herein.
  • the communications interface 1306 may connect the computing apparatus/system 1300, via a communication network, with one or more services, devices, the server system(s), cloud-based platforms, systems for implementing subject-matter databases and/or knowledge graphs for implementing the invention as described herein.
  • the memory unit 1308 may store one or more program instructions, code or components such as, by way of example only but not limited to, an operating system and/or code/component(s) associated with the process(es)/method(s) as described with reference to figures 1a to 12 and 14 to 21, additional data, applications, application firmware/software and/or further program instructions, code and/or components associated with implementing the functionality and/or one or more function(s) or functionality associated with one or more of the method(s) and/or process(es) of the device, service and/or server(s) hosting the process(es)/method(s)/system(s), apparatus, mechanisms and/or system(s)/platforms/architectures for implementing the invention as described herein, combinations thereof, modifications thereof, and/or as described with reference to at least one of the figure(s) 1a to 12 and 14 to 21
  • Figure 14 is a pictorial diagram of model applications to a simulation environment under different illuminations according to aspects of the disclosure and with respect to Figure 2.
  • FIG. 15a is a flow diagram of another model architecture for image processing in respect of colour constancy, especially to maintain the local illumination of an input image.
  • the figure shows the process 1500 of the proposed model.
  • a patch 1503 is selected over a sequential mechanism from the input image 1501.
  • the Canney edge detection 1505 the colour of edges within the patch 1503 is computed.
  • FIG. 15b is a flow diagram of the image processing based on said another model architecture shown in Figure 15a.
  • the figure shows the method 1550 for reducing the effect of illumination on images. The method comprises at least the following steps.
  • the method receives an input image.
  • the input image may be a raw image obtained from one or more cameras.
  • the input image may also be an image in a colour space as described herein.
  • step 1553 the input image is partitioned into a plurality of regions.
  • the input image is segmented using a k-means clustering algorithm based on similarity in colour or intensity values of pixels from the input image, which results in producing the plurality of regions.
  • step 1555 the plurality of regions is analysed based on colour information and spatial position of pixels in each region. The analysis is performed by identifying salient regions from the plurality of regions using a clustering algorithm; generating a clustered map based on the identified salient regions; and analysing the salient regions to select the subset of regions that are influenced by the illuminant.
  • a saliency map covering the plurality of regions can be applied. Salient regions are identified from the plurality of regions using the saliency map, which constrains the regions to a subset of partitions. The salient regions are further analysed to select the subset of regions that are influenced by the illuminant. [00165] In step 1557, from the plurality of regions, a subset of regions are selected. The selection is influenced by an illuminant based on the analysis from the previous step. [00166] In step 1559, coloured edges are identified for at least said subset of regions. This may be accomplished using edge detection, specifically using a canny edge detection algorithm configured to select a plurality of edges based on a threshold intensity.
  • step 1561 the colour information is/are extracted from the coloured edges;
  • step 1563 reflectance and illumination components of the input image are decomposed using the extracted colour information where the reflectance and illumination PCT application claiming priority from GB2313382.0 072153.00049 components are two main factors that contribute to the appearance of an object's colour in an image.
  • the reflectance component also known as surface reflectance or albedo, represents the inherent colour and material properties of an object's surface while the illumination component refers to the lighting conditions under which an object is viewed as described in the previous section. In this step, these components are decomposed.
  • step 1565 correcting illumination of the input image based on the decomposed reflectance and illumination components.
  • step 1567 the method outputs an image with illumination corrected.
  • the method may comprise identifying regions with the coloured edges from said least one subset of regions of the input image; and correcting illumination of the input image based on the identified regions with the coloured edges.
  • Figure 16 is a pictorial diagram of exemplary output of the image processing based on said another model architecture.
  • the top panel 1601 displays the original image, while the middle panel 1603 exhibits the coloured edge extracted during the model’s process.
  • the last panel 1605 represents the final output after applying the Retinex-based Correction process.
  • FIG 17 is a pictorial diagram of violin plots 1700 showing colour distance (represented by Delta E) between the images in each dataset for image processing.
  • Delta E is metric that measures the similarity or dissimilarity between images, before and after the correction process. The larger the Delta E, the greater the distance between the colours.
  • the figure shows 5 different violin plots corresponding to dataset 1 to dataset 5, 1701, 1703, 1705, 1707, and 1709. The performance is evaluated with these 5 datasets shown with respect to the violin plots.
  • Two lab datasets are common to that of the previous global colour constancy model. Three new lab datasets are added covering a more diversity of local colour constancy.
  • Each violin plot depicts the colour distance (Delta E) between the images in each dataset. Left, middle and right panels exhibit the Delta E measurements between original images (left), corrected image by global colour constancy model (middle), and corrected images by both local and global colour constancy model (right), respectively. It is assumed PCT application claiming priority from GB2313382.0 072153.00049 that images of each dataset captured from the same place under different lighting conditions, containing both global and local illuminations. [00176] In sum, the effectiveness of the models was evaluated by measuring the colour distance between images before and after their implementation using the Delta E metric.
  • FIG. 18 is a pictorial diagram of model applications for the environment under different illuminations.
  • On the left side 1801 are example input and outputs of the colour constancy models.
  • the top panel 1801a displays the original image randomly selected from the dataset as input.
  • the middle panel 1801b and bottom panel 1801c depict the corrected images obtained after applying the global colour constancy and local colour constancy models, respectively.
  • On the right side 1803 are example images captured by ODK under different lighting conditions.
  • the example images provide a visual representation of the diverse lighting conditions present in any exemplary dataset, where such dataset may be used for generating the output shown on the left side 1801 of the figure, namely bottom panel 1803c.
  • This dataset thus consists of images captured under a range of local lighting conditions. The dataset was carefully selected to include the environment with multiple light sources, ensuring that our model/algorithm was tested under PCT application claiming priority from GB2313382.0 072153.00049 realistic and challenging lighting scenarios.
  • Figure 19a is a flow diagram of another model architecture for image processing.
  • the segmented image 1911 is split into lit regions (here referred to as light segment regions) and shadow regions (here referred to as shadow segment regions).
  • Target shadow segments 1913b (based on the shadow region mask 1909) from the first step are removed by adjusting colour to match lit regions, considering texture features of the input image as described herein.
  • Yellow pixels represent the pixels assigned to the shadow region by the shadow detection algorithm.
  • a similarity metric D identifies the nearest lit regions 1913a from the target shadow segments 1913b based on texture features/patterns thereof. Histogram matching 1913b aligns colours of shadow segments to corresponding lit ones.
  • the correct shadow segments after histogram matching between the shadow and its nearest lit segments are produced and merged 1917 to form the output, a shadow-free image 1919 of the original RGB image.
  • the shadow-free image as input to or applied to output from local colour constancy and/or global colour constancy models as described herein, in the context of robotic navigation allows the robot to confidently recognize familiar places and precise position estimation, even in challenging lighting conditions.
  • Figure 19b is a flow diagram of the image processing based on said another model architecture according to aspects of the disclosure
  • Figure 20 is a pictorial diagram of exemplary input and output of the image processing based on said another model architecture.
  • the original image 2001 displays an example input image that is transformed 2003 into histogram in "lab” space, where the smoothing takes place.
  • the middle histogram 2005 shows the results of the PCT application claiming priority from GB2313382.0 072153.00049 smoothed diagram L Channel (p ⁇ 10).
  • the smoothed histogram generated by the ‘L’ channel of the converted image into ‘Lab’ colour space.
  • the solid red line exhibits the threshold labeling the pixels with the shadow and lit regions.
  • FIG. 21 a pictorial diagram of exemplary input and output of the image processing for a different input image. The figure shows the original image (top left 2101) and resultant image (bottom left 2103) from steps applied to the original image to detect shadow and then exclude the shadow from the image.
  • the figure shows the ground truth image (top right 2105) and image with shadow mask (bottom right 2107) in yellow.
  • the present invention is further described herein as one or more of the following aspects and options. These aspects and options are disclosed according to any of the figures 1 to 21 as appropriate. These aspects and certain options may be combined with any other aspects and features described herein as would be apparent to a skilled person in the field of robotics and machine vision.
  • a method for processing images based on colour constancy removing illumination from the images comprising: obtaining an image in a first colour space; converting the image to data in a second colour space; transforming the data using chromatic adaptation; performing a first normalisation on the transformed data, wherein the first normalisation comprises applying a dynamic spatial filtering technique to adjust the transformed data based on light intensity; applying a set of filters to the normalised data, wherein the set of filters is convoluted based on the normalised data in relation to the image; performing a second normalisation on the filtered data to obtain an illumination estimation of the image in relation to the filtered data; and outputting the normalised data from the second normalisation, wherein the normalised data maintains colour constancy based on the illumination estimation, removing the illumination from the normalised data.
  • a method for providing a shadow-free image comprising: receiving an input image in the first colour space, wherein the input image comprises at least one shadow region; generating shadow region masks for said at least one shadow region; removing shadow from shadow regions of the input image based on the shadow region masks; and outputting a shadow-free image.
  • an apparatus for processing images to maintain colour constancy of the images comprising: at least one model configured to perform steps according to any of herein described aspects.
  • said performing first normalisation further comprising: identifying a data representation of receptor responses in relation to the transformed data, wherein the data representation comprises a uniform representation of the receptor responses across the image; and applying the data representation to the transformed data.
  • said applying data representation further comprising: integrating the data representation over a time period; and normalising the transformed data based on the integrated data representation using the dynamic spatial filtering technique representative of performing temporal coding adjustment for each receptor in respect of the light intensity.
  • second normalisation comprises correcting the filtered data with an illumination vector generated using a pooling function, wherein the filtered data is divisible by the illumination vector.
  • the first colour space is a Red-Green- Blue colour space.
  • the second colour space is a Long-Medium-Short colour space.
  • said transforming the data using chromatic adaptation further comprising: estimating an illuminant of the data using a grey world model; and converting the data to a destination illuminant using the illuminant.
  • said converting the data to a destination illuminant further comprising: applying one or more matrix transformations to the destination illuminant in accordance with one or more cameras for obtaining the image.
  • the set of filters comprises at least one centre-surround structure representative of receptive fields (RFs) encoding colour opponency.
  • the RFs comprise the red-green opponency, blue-yellow opponency, and achromatic opponency.
  • the set of filters comprises at least two filters positioned in series such that one of said at least two filters receives input from the other filter.
  • the set of filters is representative of two layers of a visual system.
  • said analysing the plurality of regions further comprising: identifying salient regions from the plurality of regions using a clustering algorithm; generating a clustered map based on the identified salient regions; and analysing the salient regions to select the subset of regions that are influenced by the illuminant.
  • the clustered map constrains the plurality of regions into a set number of partitions.
  • the partitioning the input image into a plurality of regions further comprising: segmenting the input image using a k-means clustering algorithm based on similarity in colour or intensity values of pixels from the input image.
  • the identifying coloured edges for at least said subset of regions further comprising: performing edge detection using a canny edge detection algorithm configured to select a plurality of edges based on a threshold intensity.
  • the identifying coloured edges for at least said subset of regions further comprising: performing edge detection using sobel operators.
  • said removing shadow from shadow regions of the input image based on the shadow region masks further comprising: converting the input image to data in a third colour space; partitioning the data of the input image into a plurality of regions; segmenting the plurality of regions into shadow segment regions and light segment regions according to the shadow region masks; determining distances between the shadow segment regions and the light segment regions based on texture features of the input image; pairing the shadow segment regions and the light segment regions based on the determined distance; performing histogram matching on the paired shadow segments and light segments to produce colour-adjusted segments; iteratively performing said pairing and histogram matching until every shadow segment has been matched; merging the colour-adjusted segments to form a shadow
  • said pairing the shadow segment regions and the light segment regions based on the determined distance further comprising: identifying one or more light segment regions of closest distance to each shadow segment region; and pairing said each shadow segment with said one or more identified light segment regions.
  • the texture features comprising: edge distance, colour distance, entropy distance, neighbourhood distance, and a combination thereof.
  • said generating shadow region masks for said at least one shadow region further comprising: identifying said at least one shadow region from the input image; and generating shadow region masks based on said at least one shadow region identified.
  • aspects, examples, of the invention as described above such as algorithm(s), model(s), process(es), method(s), system(s) and/or apparatus may be implemented on and/or comprise one or more cloud platforms, one or more server(s) or computing system(s) or device(s).
  • a server may comprise a single server or network of servers
  • the cloud platform may include a plurality of servers or network of servers.
  • the functionality of the server and/or cloud platform may be provided by a network of servers distributed across a geographical area, such as a worldwide distributed network of PCT application claiming priority from GB2313382.0 072153.00049 servers, and a user may be connected to an appropriate one of the network of servers based upon a user location and the like.
  • a system, process(es), method(s) and/or tool for querying any data structure described thereof and the like according to the invention and/or as herein described may be implemented as any form of a computing and/or electronic device.
  • a device may comprise one or more processors which may be microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to gather and record routing information.
  • the processors may include one or more fixed function blocks (also referred to as accelerators) which implement a part of the process/method in hardware (rather than software or firmware).
  • Platform software comprising an operating system or any other suitable platform software may be provided at the computing-based device to enable application software to be executed on the device.
  • Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium or non-transitory computer-readable medium.
  • Computer-readable media may include, for example, computer- readable storage media.
  • Computer-readable storage media may include volatile or non- volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.
  • a computer-readable storage media can be any available storage media that may be accessed by a computer.
  • Such computer-readable storage media may comprise RAM, ROM, EEPROM, flash memory or other memory devices, CD-ROM or other optical disc storage, magnetic disc storage or other magnetic storage devices, or any other medium that can be used to carry or store desired PCT application claiming priority from GB2313382.0 072153.00049 program code in the form of instructions or data structures and that can be accessed by a computer.
  • Disc and disk as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc (BD).
  • a propagated signal is not included within the scope of computer-readable storage media.
  • Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another.
  • a connection or coupling can be a communication medium.
  • the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media.
  • the functionality described herein can be performed, at least in part, by one or more hardware logic components.
  • hardware logic components that can be used may include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs).
  • FPGAs Field-programmable Gate Arrays
  • ASICs Program-specific Integrated Circuits
  • ASSPs Program-specific Standard Products
  • SOCs System-on-a-chip systems
  • CPLDs Complex Programmable Logic Devices
  • the computing device may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device.
  • the computing device may be located remotely and accessed via a network or other communication link (for example using a communication interface).
  • the term 'computer' is used herein to refer to any device with processing capability such that it can execute instructions.
  • a remote computer may store an example of the process described as software.
  • a local or terminal computer may access the remote computer and download a part or all of the software to run the program.
  • PCT application claiming priority from GB2313382.0 072153.00049
  • the local computer may download pieces of the software as needed, or execute some software instructions at the local terminal and some at the remote computer (or computer network).
  • the term 'comprising' is used herein to mean including the method steps or elements identified, but that such steps or elements do not comprise an exclusive list and a method or apparatus may contain additional steps or elements.
  • the terms “component” and “system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor.
  • the computer- executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices.
  • the term “exemplary”, “example” or “embodiment” is intended to mean “serving as an illustration or example of something”.
  • the acts described herein may comprise computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable PCT application claiming priority from GB2313382.0 072153.00049 medium or media.
  • the computer-executable instructions can include routines, sub-routines, programs, threads of execution, and/or the like.
  • results of acts of the methods can be stored in a computer-readable medium, displayed on a display device, and/or the like.
  • the order of the steps of the methods described herein is exemplary, but the steps may be carried out in any suitable order, or simultaneously where appropriate.
  • a computer-implemented method for processing images based on colour constancy removing illumination from the images comprising: obtaining an image in a first colour space; converting the image to data in a second colour space; transforming the data using chromatic adaptation; performing a first normalisation on the transformed data, wherein the first normalisation comprises applying a dynamic spatial filtering technique to adjust the transformed data based on light intensity; applying a set of filters to the normalised data, wherein the set of filters is convoluted based on the normalised data in relation to the image; performing a second normalisation on the filtered data to obtain an illumination estimation of the image in relation to the filtered data; and outputting the normalised data from the second normalisation, wherein the normalised data maintains colour constancy based on the illumination estimation, removing the illumination from the normalised data.
  • second normalisation comprises correcting the filtered data with an illumination vector generated using a pooling function, wherein the filtered data is divisible by the illumination vector.
  • any preceding clauses further comprising: receiving a raw image from one or more cameras; removing gamma correction from the raw image; and converting the raw image to said image in the first colour space.
  • the first colour space is a Red-Green-Blue colour space.
  • the second colour space is a Long-Medium- Short colour space.
  • said transforming the data using chromatic adaptation further comprising: estimating an illuminant of the data using a grey world model; and converting the data to a destination illuminant using the illuminant. 11.
  • the set of filters comprises at least two filters positioned in series such that one of said at least two filters receives input from the other filter.
  • the set of filters is representative of two layers of a visual system. 16.
  • any preceding clauses further comprising: obtaining an input image; partitioning the input image into a plurality of regions; analysing the plurality of regions based on colour information and spatial position of pixels in each region; selecting from the plurality of regions a subset of regions that are influenced by an illuminant based on the analysis; identifying coloured edges for at least said subset of regions; extracting the colour information from the coloured edges; decomposing reflectance and illumination components of the input image using the extracted colour information; correcting illumination of the input image based on the decomposed reflectance and illumination components; outputting an image with illumination corrected. 17.
  • the input image obtained is a raw image received from one or more cameras, said image in the first colour space obtained prior to the first normalisation, said normalised data from the output of the second normalisation, or said image in the first colour space obtained following to the second normalisation.
  • said analysing the plurality of regions further comprising: identifying salient regions from the plurality of regions using a clustering algorithm; generating a clustered map based on the identified salient regions; and analysing the salient regions to select the subset of regions that are influenced by the illuminant.
  • any preceding clauses further comprising: obtaining an input image in the first colour space; identifying shadow regions from the input image; generating shadow region masks based on the identified shadow regions; removing shadow from shadow regions of the input image based on the shadow region masks; and outputting a shadow-free image.
  • the input image is a raw image or an image in the first colour space.
  • An apparatus for establishing colour constancy of an image comprising: one or more cameras for capturing the image in a first colour space; a processing unit for converting the captured image to corresponding data in a second colour space; a first model, a second model, and a third model configured to process said data sequentially to establish colour constancy, wherein the first model is configured to transform the data using chromatic adaptation, the second model is configured to perform a first normalisation on the transformed data, wherein the first normalisation comprises applying a dynamic spatial filtering technique to adjust the transformed data based on light intensity, and the third model is configured to apply a set of filters to the normalised data, wherein the set of filters is convoluted based on the normalised data in relation to the image, and perform a second normalisation on the filtered data to obtain an illumination estimation of the image
  • a computer-implemented method for reducing effect of illumination on images comprising: receiving an input image; partitioning the input image into a plurality of regions; analysing the plurality of regions based on colour information and spatial position of pixels in each region; selecting from the plurality of regions a subset of regions that are influenced by an illuminant based on the analysis; identifying coloured edges for at least said subset of regions; extracting the colour information from the coloured edges; decomposing reflectance and illumination components of the input image using the extracted colour information; correcting illumination of the input image based on the decomposed reflectance and illumination components; outputting an image with illumination corrected.
  • An apparatus for processing images to maintain colour constancy of the images comprising: at least one model configured to perform steps according to any method of clauses 27 to 34.
  • a system for processing images to establish colour constancy for an image by removing illumination from the image comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform any method of clauses 27 to 34.
  • a computer-implemented method for providing a shadow-free image comprising: receiving an input image in the first colour space, wherein the input image comprises at least one shadow region; generating shadow region masks for said at least one shadow region; removing shadow from shadow regions of the input image based on the shadow region masks; and outputting a shadow-free image.
  • An apparatus for processing images to maintain colour constancy of the images comprising: at least one model configured to perform steps according to any method of clauses 37 to 44.
  • a system for processing images to establish colour constancy for an image by removing illumination from the image comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform any method of clauses 37 to 44.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un système, un appareil et un ou plusieurs procédés de traitement d'images sur la base d'une constance de couleur. Le procédé consiste à obtenir une image dans un premier espace colorimétrique ; convertir l'image en données dans un second espace colorimétrique ; transformer des données par adaptation chromatique ; réaliser une première normalisation sur les données transformées, la première normalisation comprenant l'application d'une technique de filtrage spatial dynamique pour ajuster les données transformées sur la base de l'intensité lumineuse ; appliquer un ensemble de filtres aux données normalisées, l'ensemble de filtres étant convolué sur la base des données normalisées par rapport à l'image ; effectuer une seconde normalisation sur les données filtrées pour obtenir une estimation d'éclairage de l'image par rapport aux données filtrées ; et délivrer les données normalisées à partir de la seconde normalisation, les données normalisées maintenant une constance de couleur sur la base de l'estimation d'éclairage, éliminer l'éclairage des données normalisées.
PCT/GB2024/052253 2023-09-01 2024-08-29 Procédé de réduction de variabilité d'image dans des changements d'éclairage globaux et locaux WO2025046235A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB2313382.0 2023-09-01
GB2313382.0A GB2633100A (en) 2023-09-01 2023-09-01 Method for reducing image variability under global and local illumination changes

Publications (1)

Publication Number Publication Date
WO2025046235A1 true WO2025046235A1 (fr) 2025-03-06

Family

ID=88296816

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2024/052253 WO2025046235A1 (fr) 2023-09-01 2024-08-29 Procédé de réduction de variabilité d'image dans des changements d'éclairage globaux et locaux

Country Status (2)

Country Link
GB (1) GB2633100A (fr)
WO (1) WO2025046235A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1994011987A1 (fr) * 1992-11-16 1994-05-26 Technion Research & Development Foundation Ltd. Appareil et procede d'amelioration d'images en couleurs
US20020167615A1 (en) * 2001-02-07 2002-11-14 Ramot University Authority For Applied Research And Industrial Development Ltd. Method for automatic color and intensity contrast adjustment of still and video images

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5771312A (en) * 1996-07-01 1998-06-23 Ramot University Authority For Applied Research & Industrial Development Ltd. Method for automatic partial white balance correction

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1994011987A1 (fr) * 1992-11-16 1994-05-26 Technion Research & Development Foundation Ltd. Appareil et procede d'amelioration d'images en couleurs
US20020167615A1 (en) * 2001-02-07 2002-11-14 Ramot University Authority For Applied Research And Industrial Development Ltd. Method for automatic color and intensity contrast adjustment of still and video images

Non-Patent Citations (13)

* Cited by examiner, † Cited by third party
Title
ANDERSONMATTHEW ET AL.: "Color and imaging conference", vol. 1996, 1996, SOCIETY FOR IMAGING SCIENCE AND TECHNOLOGY, article "Proposal for a standard default color space for the internet-srgb."
BRAINARDDAVID H.BRIAN A. WANDELL: "''Analysis of the retinex theory of color vision", JOSA A, vol. 3, no. 10, 1986, pages 1651 - 1661
CONWAY, BEVIL R. ET AL.: "Advances in color science: from retina to behavior.", JOURNAL OF NEUROSCIENCE, vol. 30, 2010, pages 14955 - 14963
GAO SHAO-BING ET AL: "Color Constancy Using Double-Opponency", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, IEEE COMPUTER SOCIETY, USA, vol. 37, no. 10, October 2015 (2015-10-01), pages 1973 - 1985, XP011667999, ISSN: 0162-8828, [retrieved on 20150901], DOI: 10.1109/TPAMI.2015.2396053 *
HURVICH, LEO M.DOROTHEA JAMESON: "An opponent-process theory of color vision.", PSYCHOLOGICAL REVIEW, vol. 64, 1957, pages 384
LAND, EDWIN H.JOHN J. MCCANN: "''Lightness and retinex theory.", JOSA, vol. 61, no. 1, 1971, pages 1 - 11, XP055441300, DOI: 10.1364/JOSA.61.000001
LAUGHLINSIMON: "A simple coding procedure enhances a neuron's information capacity.", ZEITSCHRIFT FOR NATURFORSCHUNG, vol. 910, 1981, pages 912
LUO, M. RONNIER: "A review of chromatic adaptation transforms.", REVIEW OF PROGRESS IN COLORATION AND RELATED TOPICS, vol. 30, 2000, pages 77 - 92, XP071658279, DOI: 10.1111/j.1478-4408.2000.tb03784.x
SHAPLEYCONWAY, BEVILR.DAVID H. HUBELMARGARET S. LIVINGSTONE: "Color contrast in macaque V1.", CEREBRAL CORTEX, vol. 12, no. 9, 2002, pages 915 - 925
SHAPLEYROBERTMICHAEL J. HAWKEN.: "Color in the cortex: single-and double-opponent cells.", VISION RESEARCH 5, vol. 1, no. 7, 2011, pages 701 - 717
SOLOMON, SAMUEL G.PETER LENNIE: "The machinery of colour vision.", NATURE REVIEWS NEUROSCIENCE, vol. 8, no. 4, 2007, pages 276 - 286
ST6CKLANNA LISADAVID CHARLES O'CARROLLERIC JAMES WARRANT: "Hawkmoth lamina monopolar cells act as dynamic spatial filters to optimize vision at different light levels.", SCIENCE ADVANCES, vol. 6, no. 16, 2020, pages eaaz8645
STOKESMICHAEL., A STANDARD DEFAULT COLOR SPACE FOR THE INTERNET-SRGB., 1996, Retrieved from the Internet <URL:http://www.w3.org/Graphics/Color/sRGB.html>

Also Published As

Publication number Publication date
GB202313382D0 (en) 2023-10-18
GB2633100A (en) 2025-03-05

Similar Documents

Publication Publication Date Title
Wang et al. Semantic segmentation of crop and weed using an encoder-decoder network and image enhancement method under uncontrolled outdoor illumination
US10977494B2 (en) Recognition of weed in a natural environment
CN108197546B (zh) 人脸识别中光照处理方法、装置、计算机设备及存储介质
Bai et al. Crop segmentation from images by morphology modeling in the CIE L* a* b* color space
US8855412B2 (en) Systems, methods, and apparatus for image processing, for color classification, and for skin color detection
JP2023504624A (ja) 作物の被害を特定するためのシステム及び方法
Liu et al. Detection of citrus fruit and tree trunks in natural environments using a multi-elliptical boundary model
US9053537B2 (en) Classifier for use in generating a diffuse image
JP2014515587A (ja) デジタル画像装置用の画像処理パイプラインの学習
Jenifa et al. Classification of cotton leaf disease using multi-support vector machine
US20130114911A1 (en) Post processing for improved generation of intrinsic images
US8249342B1 (en) Color analytics for a digital image
Biswas et al. Microscopic image contrast and brightness enhancement using multi-scale retinex and cuckoo search algorithm
CN113255434A (zh) 一种融合果实特征与深度卷积神经网络的苹果识别方法
CN113132693B (zh) 一种色彩校正方法
He et al. An image segmentation algorithm based on double-layer pulse-coupled neural network model for kiwifruit detection
US8428352B1 (en) Post processing for improved generation of intrinsic images
WO2012064414A2 (fr) Procédé et système de production d&#39;images intrinsèques au moyen d&#39;une contrainte d&#39;éclairage doux
WO2025046235A1 (fr) Procédé de réduction de variabilité d&#39;image dans des changements d&#39;éclairage globaux et locaux
US8553979B2 (en) Post processing for improved generation of intrinsic images
Gaddam et al. Advanced Image Processing Using Histogram Equalization and Android Application Implementation
Shen et al. A holistic image segmentation framework for cloud detection and extraction
Kusnandar et al. A novel method for optimizing color selection using the hadamard product technique
US8655099B2 (en) Relationship maintenance in an image process
Kaur et al. A comparative review of various illumination estimation based color constancy techniques

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24768665

Country of ref document: EP

Kind code of ref document: A1

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载