WO2016001339A1

WO2016001339A1 - Communicating barcode data

Info

Publication number: WO2016001339A1
Application number: PCT/EP2015/065049
Authority: WO
Inventors: Frederik Jan De Bruijn
Original assignee: Koninklijke Philips N.V.
Priority date: 2014-07-03
Filing date: 2015-07-02
Publication date: 2016-01-07

Abstract

Light-emitting apparatus for emitting light to be detected by a rolling-shutter camera, the rolling-shutter camera having an image capture element divided into a plurality of lines which are exposed in a sequence. The light-emitting apparatus comprises a light source for emitting the light, and a controller for modulating the light emitted by the light source. The controller is configured to encode data into the light using a time- varying binary modulation in the intensity of the light, formatted such that when detected by the rolling- shutter camera the time-varying binary modulation will form a binary pattern of light and dark stripes over the lines of the rolling-shutter camera according to a one-dimensional barcode format, this being pre-existing format that is also used for printed or graphically displayed one-dimensional barcodes.

Description

Communicating barcode data

TECHNICAL FIELD

The present disclosure relates to barcodes, and particularly to an alternative method of communicating barcode data. BACKGROUND

A barcode is a visibly-displayed, binary spatial pattern of light and dark stripes (typically black and white) that conveys data according to one of a number of predetermined barcode formats. A barcode may be physically printed, e.g. on the packaging of a product or sticker to be applied to a product or its packaging, or on a printed document; or alternatively a barcode may be displayed on a graphical display, i.e. a computerized or electronic display, for instance as part of an electronic document. The barcode may for example be used to identify a product to which it is applied, or in a document as proof of purchase, or to identify, verify or authenticate a holder of the document.

A barcode is formatted according to a barcode format, which is one of a number of predetermined, standardized formats for conveying data in the pattern of the barcode. The barcode can read by a barcode reader based on a visible-light based technology, and a suitable algorithm or equivalent circuitry configured to recognize the barcode format. For example the barcode reader may take the form of a dedicated laser-based barcode scanner, or a general purpose camera and a suitable application configured to read the barcode from an image captured by the camera based on image recognition techniques.

According to more recent formats and readers, barcodes can also take the form of a two dimensional pattern, and need not necessarily even be limited to a binary pattern (e.g. the format could be based on three colors). However the present disclosure is primarily concerned with one-dimensional binary barcodes according to conventional, pre-existing formats.

Currently barcodes are fixed to the surface on which they are printed. Printed barcodes are relatively cheap and simple, e.g. being printed directly onto a product or document, or on a sticker applied to the product. However, they are also inflexible. If the barcodes are to be used to provide a temporary or changeable label, an image display is required to create the striped pattern, but this is more complex and costly.

SUMMARY

Coded light is a technology whereby information is embedded into the visible light emitted by light sources such as everyday luminaires, using high-frequency light modulation at frequencies beyond human perception such that the primary function of illumination is not affected.

Coded light can be detected using rolling-shutter cameras such as are typically found in many mobile devices like smartphones and tablets. A rolling-shutter camera is one in which the camera's image capture element is divided into lines (i.e. parallel rows or columns of pixels, typically horizontal rows) which are exposed one-after-another in a sequence. The fast time- varying modulation in the light causes a striped spatial modulation over the lines in the images captured with a rolling-shutter camera. In known systems this phenomenon is used to detect coded light using rolling-shutter cameras, but only based on formats such as line-coding formats (e.g. Manchester coding) that are not also designed for visible display or reading as are barcodes.

The inventor has noticed that the same type of camera used to detect coded light, is also already commonly used to detect printed or graphically displayed barcodes such as appear on the packaging of most retail products, but based on image recognition rather than coded light techniques. The inventor has found that coded light can in fact be used to cause a striped pattern in the lines of a rolling shutter camera that takes substantially the same form as a printed or graphically displayed barcode format. I.e. the modulation in the light can be used to send a "virtual barcode" that will form into a barcode pattern when captured in the lines of the rolling-shutter camera. The data then can be read by inputting the pattern from the rolling shutter image sensor into a conventional barcode-reader algorithm or circuit. This use of coded light thus broadens the applicability of existing barcode formats beyond printed or graphically displayed media; or conversely broadens the applicability of existing barcode- reader algorithms or circuits to the detection of coded-light. For instance, as barcode detection is becoming an embedded application in many smartphone and tablet operating systems, the disclosed idea may extend the use of such embedded barcode 'scanning' applications to the detection of coded light.

Hence according to one aspect disclosed herein, there is provided a light- emitting apparatus for emitting light to be detected by a rolling-shutter camera, the rolling- shutter camera having an image capture element divided into a plurality of lines which are exposed in a sequence. The light-emitting apparatus comprises a light source for emitting the light, and a controller for modulating the light emitted by the light source. The controller is configured to encode data into the light using a time-varying binary modulation in the intensity of the light, formatted such that when detected by the rolling-shutter camera the time-varying binary modulation will form a binary pattern of light and dark stripes over the lines of the rolling-shutter camera according to a one-dimensional barcode format, this being pre-existing format that is also used for printed or graphically displayed one-dimensional barcodes.

In embodiments, the controller is configured to receive an input signal comprising data formulated for display according to said one-dimensional barcode format for printed or graphically displayed one-dimensional barcodes, and to convert to said input signal into said one-dimensional binary modulation of the light from the light source. Thus the controller can take a signal that was intended to generate a printed or graphically displayed barcode, and convert this instead to a coded light signal, but one which can still ultimately be read according to the same format by a conventional or existing barcode-reader algorithm or circuit.

For example the apparatus may comprising a barcode reader, and the controller may be configured to receive said input signal by using the barcode reader to read a printed or graphically displayed barcode comprising said data. The controller may thus convert the printed or graphically displayed barcode to said one-dimensional binary modulation of the light for detection using the rolling-shutter camera according to said same format.

In further embodiments, the apparatus may comprise a diffuser arranged to diffuse said light before the detection by the rolling shutter camera. This can make the light flood a larger area of the camera's image capture element, enabling more reliable detection of the barcode image in the lines of the image capture element.

In yet further embodiments, the controller may be configured to adapt the data such that the levels and/or spacing of the stripes in said binary pattern deviate from the barcode format within a tolerance of said barcode format; the data being adapted to reduce a visible flicker in the light caused by the data, reduced relative to a strict adherence to said format. Existing barcodes will not result in a DC-free signal when converted to light modulation, which may lead to visible flicker in the light. Embodiments of the present disclosure exploit the tolerance of existing barcode formats and their corresponding readers (originally designed to accommodate for factors such as imperfect reflection, faded or imperfect printing, damaged packaging, etc.) to deliberately vary the modulation of the light within the tolerance of the barcode format, to give a modulation that is closer to being DC- free (and therefore flicker free).

According to another aspect disclosed herein there is a provided a corresponding receive-side device. The device comprises a barcode-reader module for reading printed or graphically displayed one-dimensional barcodes according to a one- dimensional barcode format, and a rolling-shutter camera for receiving light emitted by a light source. The rolling-shutter camera is arranged to capture an image over a plurality of lines, each line of the image representing the received light captured at a successive time, and the device further comprises an image processing module for processing said image. Data is encoded into the light using a time- varying binary modulation in an intensity of the light, formatted such that when detected by the rolling-shutter camera the time- varying binary modulation forms a binary pattern of light and dark stripes over the lines of the rolling- shutter camera according to said one-dimensional barcode format; and the image processing module is configured to perform one or more coded-light processing operations on the image in order to prepare said pattern for reading by the barcode-reader module according to the barcode format, and, following said processing, to output the pattern to the barcode-reader module for reading the data from said pattern according to said format.

In embodiments, said one or more coded-light processing operations may comprise: applying a Wiener filter or other filter to mitigate a filtering effect of an exposure time of the camera. Alternatively or additionally, said one or more coded-light processing operations comprise: producing an average background by averaging a plurality of frames captured by the camera at a frame rate, and subtracting the averaged background from said image, wherein the barcode pattern is repeated in said modulation with a repetition rate, and wherein each of said plurality of frames experiences the barcode pattern at a different spatial offset due to the repetition rate not matching the frame rate, said averaging thereby generating the averaged background image with the modulation substantially removed.

In embodiments the barcode-reader module may be a barcode reader application (i.e. software), for example embedded in an operating system of the device, or running on the operating system.

In further embodiments, the image processing module may be configured to de-focus the camera from the light source to obtain a greater coverage of the light over the image than if focused on the light source. This may be used as an alternative to (or even in addition to) the diffuser between the light source and camera.

In further embodiments, the device may be a mobile device (e.g. smartphone or tablet), and may comprises a user interface configured to output directions to a user to hold the device in a suitable orientation for capturing said pattern in the lines of the rolling-shutter camera according to said barcode format.

In yet further embodiments: the image processing module may comprise an image processing application for processing the pattern prior to being output to the barcode- reader module for reading; and the device may be operable to communicate via a first medium, other than time- varying binary modulation of light being used to form a barcode in the lines of the rolling-shutter camera. E.g. the first medium may comprises an RF channel, or the use of said camera or another camera of the device to read a printed or graphically displayed barcode. The device may then be configured to obtain a link for downloading the image processing application or an update thereto via the first medium, to download the image processing application or update based on said link as obtained via the first medium, and to subsequently use the image processing application or update to detect said pattern and output the pattern to the barcode-reader module. For instance if a user has a device not already equipped to obtain the barcode formatted data from the coded light, the user can exploit the existing barcode-reader functionality in the more conventional manner in order to obtain the software for the new way of obtaining barcode data.

In yet further embodiments, the device may comprise a transmitter (e.g. RF) for sending messages to the light source. A controller of the device may then be configured to send a message to the light source via said transmitter requesting that the light source switches to a mode in which it modulates the light with said time- varying binary modulation.

According to a further aspect disclosed herein, there is provided a method comprising: controlling a light source to emit light to be detected by a rolling-shutter camera, the rolling-shutter camera having an image capture element divided into a plurality of lines which are exposed in a sequence; and encoding data into the light using a time- varying binary modulation in an intensity of the light, formatted such that when detected by the rolling-shutter camera the time-varying binary modulation will form a binary pattern of light and dark stripes over the lines of the rolling-shutter camera according to a one-dimensional barcode format, being a same format as used for printed or graphically displayed one- dimensional barcodes. In embodiments, the method may further comprises steps in accordance with any of the emitting or receive side operations disclosed herein.

According to another aspect disclosed herein, there is provided a computer program product comprising code embodied on at least one computer-readable storage medium and configured so as when executed on one or more processors, to perform operations of the emit-side controller, image processing module and/or receive-side controller according to any of the embodiments disclosed herein. The computer-program product may be stored locally at the apparatus or device, and/or on a server comprising one or more server units at one or more sites (being downloadable from the server to the apparatus or device), and/or any other storage medium or media or combination.

BRIEF DESCRIPTION OF THE DRAWINGS

To assist the understanding of the present disclosure and to show how embodiments may be put into effect, reference is made by way of example to the accompanying drawings in which:

Fig. 1 is a schematic block diagram of a system comprising a light emitting apparatus and a device with a camera;

Fig. 2 is another schematic block diagram of a system comprising a light emitting apparatus and device with camera;

Fig. 3 is a schematic block diagram of a mobile device;

Fig. 3a is a schematic block diagram of an emit-side apparatus comprising light source, controller and input barcode scanner;

Fig. 4 schematically illustrates an image capture element of a rolling-shutter camera;

Fig. 5 schematically illustrates a barcode pattern formed in the lines of a rolling-shutter camera;

Fig. 6 is a timing diagram of a rolling-shutter capture process; Fig. 7 is a picture of a product with barcode; and

Figs. 8a and 8b show images captured by a rolling-shutter camera comprising a barcode pattern formed by modulation in the captured light;

Figs. 9a and 9b show an example of a barcode pattern and an example of a modified barcode pattern;

Figs. 10a and 10b show an example of a signal associated with a barcode pattern and an example of a signal associated with a modified barcode pattern; Fig. 11 shows an example of magnitude responses associated with a barcode pattern and a modified barcode pattern; and

Fig. 12 is a schematic block diagram, illustrating an example barcode signal modification method.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following, light is modulated so that when captured by a rolling-shutter camera, it forms a spatial pattern that conforms with one of the pre-existing barcode formats used for printed or graphically displayed barcodes, such that the captured pattern can be read by a conventional barcode reader application or circuit. Thus the modulation in the light is used to send a "virtual barcode" that will form on the image sensor of the rolling-shutter camera.

In embodiments, this means no printed medium or graphical display is required to show the barcode, and even a single LED emitting modulated light can be sufficient. Regardless of the medium, print or light, the detected pattern is the same; and as the pattern is the same, there is in principle no need for additional detectors other than the barcode scanner application or circuit.

Figures 1 and 2 schematically illustrate examples of a system set-up for implementing the encoding of barcodes via the modulation of light (as used by the inventor to demonstrate the concept).

The system comprises a light source 4, for example an LED-based lamp comprising one or more LEDs. The light source 4 is connected to a controller 2 via a driver (not shown) allowing the controller to control the emission of light from the light source 4, at least in that the controller 2 is able to modulate the light 5 emitted by the light source 4 with a time-varying modulation. The system also comprises a device 6 which in embodiments takes the form of a user terminal, preferably a mobile user terminal such as a smartphone or tablet. This device 6 comprises a rolling-shutter camera 8. The light source 4 is arranged so that at least some of the light 5 it emits can be captured by the camera 8. In the arrangement of Figure 1, the light source 4 and camera 8 are disposed such the camera 8 has a direct line-of- sight of the light source 4, and thus captures light 5 from it directly. Figure 2 shows an alternative arrangement in which the camera 8 captures a reflection of the modulated light 5 via an illuminated surface 10.

The controller 2 comprises a function generator that is arranged to repeatedly transmit, by means of this modulation, a binary pulse train that corresponds to the binary pattern of a barcode. I.e. it modulates the light with a signal such that the rolling shutter capture by the camera 8 results in a spatial barcode pattern with the same characteristics as a printed barcode, as will be discussed in more detail below. The controller may be

implemented in software stored on a memory and arranged to be executed on a processor of the light source 4, or connected to the light source 4. Alternatively it is not excluded that some or all of the functionality of the light source's controller 4 is implemented in dedicated hardware circuitry, or in configurable or reconfigurable circuitry such as a PGA or FPGA.

A "rolling shutter" camera is the type of camera usually integrated into a mobile phone or tablet. In a rolling-shutter camera, the camera's image capture element is divided into a plurality of lines (typically horizontal lines, i.e. rows) which are exposed in sequence line-by-line. Typically the sequence "rolls" in order across the frame, e.g. in rows top to bottom, hence the name "rolling shutter".

Figure 4 represents the image capture element 20 of an example rolling-shutter camera, as used in the camera 8. The image capture element 20 comprises an array of pixels for capturing signals representative of light incident on each pixel, e.g. typically a square or rectangular array of square or rectangular pixels. In a rolling-shutter camera, the pixels are arranged into a plurality of lines, e.g. horizontal rows 22. To capture a frame each line is exposed in sequence, each for a successive instance of the camera's exposure time T_exp. In this case the exposure time is the duration of the exposure of an individual line. Note of course that in the context of a digital camera, the terminology "expose" or "exposure" does not refer to a mechanical shuttering or such like (from which the terminology historically originated), but rather the time when the line is being used to sample the light from the environment. Note also that a sequence in the present disclosure means a temporal sequence, i.e. so the exposure of each line starts at a slightly different time. For example first the top row 221 begins to be exposed for duration T_exp, then at a slightly later time the second row down 22₂ begins to exposed for T_exp, then at a slightly later time again the third row down 22₃ begins to be exposed for T_exp, and so forth until the bottom row has been exposed. This process is then repeated to expose a sequence of frames.

Figure 6 shows a timing diagram an example rolling shutter process, each line being exposed in turn for time T_exp, at a line rate fii_ne. Note that in embodiments the exposure times may overlap, i.e. the next line 22₂ starts being exposed after its preceding line22i but not necessarily before that preceding line 221 has finished being exposed, and so forth (also therefore T_exp is not necessarily equal to 1 / fiine) - In embodiments, the process may repeat over multiple frames at a rate f_frame, thus providing continuous video capture. Alternatively a single still image may be used. The above-described nature of a rolling-shutter camera has the effect that a time-varying modulation in the captured light will be translated into a spatial modulation over the lines of the camera (if the modulation is not too slow or indeed too fast relative to the exposure time T_exp). This effect is already exploited in existing systems to detect coded light signals using rolling-shutter cameras. However, existing coded light signals are coded using schemes such as line-coding schemes (e.g. Manchester coding), and are not encoded with any particular consideration of the "appearance" of the spatial pattern on the image capture element 20, nor that this could be made to correspond to any known visual, spatial encoding format.

As illustrated in Figure 5, in the present disclosure the modulation in the light

5 is encoded not according to any particular coded light format to be detected by

conventional coded light decoding techniques, but rather so that when captured it forms a spatial pattern over the lines 22 of the image capture element 20 corresponding to a barcode format - i.e. a format designed for printed or graphically displayed barcodes. This means the captured image can be output to be read by a barcode reader application or circuit rather than a coded light decoder.

To achieve this, the light 5 is coded with a binary modulation (on-off or high- low intensity), with the timing of the modulation being tuned relative to the line rate j_ine of the target camera 8 (or a range of target cameras). If the momentary modulation level of the light 5 is on or high when a given line 22 is being exposed, then then line will record a light stripe in the captured image, corresponding to a "white" stripe of the barcode format. If on the other hand the momentary modulation level of the light 5 is off or low when a given line 22 is being exposed, then that line will record a dark stripe in the captured image, corresponding to a "black" stripe of the barcode format.

The spatial extent of this "virtual barcode" depends on the symbol rate of the time signal and the line rate of the rolling-shutter sensor. To ensure the pattern is adequately communicated, the timing of the modulation in the light 5 is arranged such that one "on" or "off (or high or low) period (i.e. the duration r_symbol [s] of a single channel symbol) in the time-varying modulation is greater than or equal to the twice the time interval r_line = l/ une [^s] between consecutive line captures (Nyquist criterion), but not so large that transmitting the data of a whole barcode lasts longer than the time to expose the whole active area of the image capture element 20 once (one frame). I.e. each barcode stripe covers one or a few lines 22 of the image capture element 20, and the pattern of at least one whole barcode fits into the image capture element 20. N.B. the condition is "at least one", as the barcode pattern is preferably transmitted from the light source 4 in a repetitive fashion (repeated instances of the pattern are concatenated, with a small space between them). The detection by a conventional barcode scanner application or circuit will stop after one detection, which is standard behavior of barcode scanners to avoid bursts of multiple detections from the same barcode. The condition "at least one" is added here to ensure that each image of video stream always contains at least one single, not-truncated barcode. If two or more instances of the barcode pattern are transmitted in the space of one frame, this means one barcode including idle interval will occupy less than half an image height.

The simplest way to assure detection of at least one single barcode repetition within the height of each captured image is to choose the symbol period r_symbol = l/ _symboi satisfies:

1

^ ^X ^symbol + ^idle < ₂ ^X ^ ^X ^line >

with L the number of channel symbols of a single barcode including possible leading and or trailing guarding symbols, r_idle [s] the time interval between consecutive barcode repetitions to create the required spatial separation within a captured image (a "quiet zone"), and H the number of lines in a captured image.

As a further consideration, it is known that the shutter time (i.e. line exposure time) r_exp of a rolling shutter camera produces a filtering effect which can cause inter- symbol interference in coded light detection. In the present case, this filtering effect manifests itself in lack of "sharpness" in the captured barcode pattern. In embodiments herein, preferably the exposure time of the camera 8 is locked to a very small value in order to cause a sufficiently "sharp" appearance of the modulated light signal. The exposure time is the temporal equivalent of the spatial "measuring aperture" for printed barcodes, which according to ISO/IEC 15416 is chosen to be smaller than 80% of the spatial symbol size. Therefore by analogy, the condition: can for example be used as condition for the exposure time. Alternatively longer exposure times can be accommodated by applying a suitable filter such as Wiener filter to mitigate the filtering effect of the camera's exposure time (to at least some extent).

Referring again to Figure 1 , in embodiments a diffuser screen or layer 9 may be placed between the light source 4 and the camera 6. This helps ensure the light 5 more fully floods the whole area of the image capture element 20, as the aim is to capture the time- varying modulation in the light rather than to capture a picture of the light source (i.e. there is no need for any details of the light source to be identifiable in the image). Alternatively the reflecting surface 10 may act as a diffuser if it predominantly causes diffusive reflection rather than mirror reflection. In other alternative arrangements, a diffuser 9 or 10 may not be required at all. Instead, the controller 17 may be configured 17 to defocus the camera 8, so that it is not focused on the light source 4 and hence the image capture element 20 is flooded with light from the light source. Another possibility is that no diffuser or defocussing is used, and instead the device 6 is held close to the camera 8, or the light source 4 has a large emitting area.

Figure 7 shows an example of a barcode printed on the packaging of a product, and Figures 8a and 8b show a detail of the corresponding pattern (from an actual capture) seen by a rolling-shutter camera 8 in an example experiment performed by the inventor using the disclosed techniques (in this example with use of a diffuser 9). Figure 8a shows a detail of the captured pattern under relatively non-uniform illumination of the sensor and using an exposure time T_exp that is larger than the duration of a single channel symbol, such that the pattern appears smeared along the vertical direction. Figure 8b shows a detail of the captured pattern under relatively uniform illumination of the sensor and using an exposure time that is shorter than the duration of a single channel symbol, such that the pattern appears sharp along the vertical direction.

Figure 3 shows more detail of the receiving device 6. The receiving device 6 comprises: the rolling-shutter camera 8, optionally an image processing module 14 coupled to the camera 8, and a barcode reader module 16 coupled to the image processing module 14 or directly to the camera 8. The device 6 also comprises a controller 17 and a transmitter 18 coupled to the controller 17. In embodiments, each of the image processing module 14, barcode reader module 16 and controller 17 is implemented as a portion of code stored on a memory of the device 6 and arranged for execution on a processor of the device 6.

Alternatively, it is not excluded that one or more of these modules 14, 16, 17 is implemented wholly or partially in dedicated hardware circuitry, or configurable or reconfigurable circuitry such as a PGA or FPGA.

The barcode reader module 16 is configured to read barcodes according to one or more conventional barcode formats, i.e. the same format as already used for printed and/or graphically displayed barcodes. Preferably the barcode reader module 16 takes the form of a barcode reader application (i.e. software), either included as part of the device's operating system or as a separate application running on the operating system. It may have been originally included on the device 6 (e.g. phone or tablet) for the purpose of reading barcodes from images of visibly-displayed barcodes captured by the camera 8.

In some embodiments, no special coded-light processing is required, and image formed directly by the modulation on the image sensor 20 can be output straight to a conventional barcode reader module 16. Provided that the captured radiant or illuminated area is sufficiently uniform, the proper combination of exposure time (T_exp) and sensitivity (ISO) will lead to a normal detection by a detection algorithm (as the inventor has found by the experiment).

However, in other situations the fact that the pattern has been formed from time-varying modulation in the light falling on the rolling-shutter image sensor 20 may introduce some artefacts that need to be mitigated before the image can be supplied to the barcode reader 16, and/or this fact may provide opportunities for further enhancing the image before being read by the barcode reader 16 . In these cases some extra signal processing may be required or desired in order to prepare the pattern for input to the barcode reader module 16. Hence in accordance with embodiments of the present disclosure, the receiving device 6 is provided with an image processing module 14 configured to receive the image formed in the lines of the image capture element 20 of the camera 8 due to the invisible modulation of the light 5, and to process the image based on one or more coded- light image processing techniques in order to enhance the barcode pattern into a form suitable for reading by the barcode reader module 16 based on the same, conventional barcode format as used for reading printed or graphically displayed barcodes. The image processing module 14 then outputs at least a portion of the processed image comprising the pattern (or even the pattern represented as a ID digital signal) to the barcode reader 16 to be read according to aid barcode format.

One artefact that may require processing before the pattern is supplied to the barcode reader 16 is the filtering effect caused by the exposure time T_exp of the rolling shutter camera 8. As will be appreciated by a person skilled in the art, the fact that each line 22 of the rolling shutter sensor 20 is exposed for a time Texp effectively forms a rectangular filter (box filter) applied to the captured samples. This can cause a "smearing out" of the pattern in the image. For short exposure times (preferably T_exp < 0.8T_symb_oi) this is not a problem for the present purposes, as the image remains sharp enough for the barcode pattern to be read by a conventional barcode reader 16.

Further, if the capture area is not sufficiently uniform, image processing may be required to remove the non-uniformity before the barcode pattern can be detected by the barcode reader module 16. This can be performed by taking an average image resulting from multiple captured images, on the basis of the repeated barcode pattern being repeated with a timing such that the light modulation has a different spatial vertical offset in each image.

In such cases, the image processing module 14 is arranged to receive a plurality of frames captured by the rolling-shutter camera, e.g. a sequence of contiguous frames, thus forming a "stack" of images. Further, the modulation encoding the barcode pattern is repeated multiple times. Each repetition can be thought of as a packet having a duration T_pack_et. E.g. if the barcode pattern is modulated into m symbols of duration T_symb₀i, and packets are separated by an inter-packet idle time Tidie, then T_pack_et = niT_symb₀i + Tidie- As long as 1/ T_pack_et is not equal to an integer multiple of the frame rate ff_rame (and preferably not a simple ratio such as a half integer multiple) then each of the frames in the stack will experience the barcode pattern at a different respective spatial offset within the respective frame (e.g. different vertical offset in the case of horizontal rows 22). I.e. the pattern "rolls" across the image over multiple frames. By averaging a sufficient number of such frames (e.g. > 5, or > 10) all with the pattern at slightly different offsets, the image processing module 14 obtains an averaged background image with the effect of the modulation substantially removed. The image processing module then subtracts the averaged background image from the image that is to be supplied to the barcode reader module 16 ("substantially removed" meaning to a sufficient degree to allow reading of the barcode pattern by the barcode reader module).

The skilled person may also be aware of other coded light processing techniques which may be used to improve a pattern formed by light modulation in a rolling shutter-camera.

Note that in embodiments, the processing by the image processing module 14 may be performed with the images in two-dimensional form or one-dimensional form. In the latter case the individual pixel samples of each line 22 are first averaged over the respective line to produce an average sample representative of the line, such that the image is represented by a ID set samples with one sample per line 22, e.g. per row. After the processing, the image processing module 14 may supply the output pattern to the barcode reader module 16 in 2D form or ID form. In the case where the image processing is performed in one dimension, the image processing module 14 may output the pattern to the barcode reader module 16 also in ID form (if the barcode reader is arranged to accept a digital ID signal representation of a barcode as an input). Alternatively (e.g. if the barcode reader module 16 will not accept a ID signal), the image processing module 14 may use the resulting ID form of the pattern to artificially re-generate a 2D image of the barcode pattern (with each ID sample simply being extended uniformly across the respective line it represents), and supply this re-generated 2D pattern to the barcode reader 16 (which will never need to know this was not a directly captured image).

In addition to coded-light specific operations (or even of these are not involved), the image processing module 14 may perform one or more general image processing operations such as:

i) the image processing module 14 detects the presence of the barcode pattern in the image and then, in response to the detection, decides to forward the image to the reader 16 unmodified (possibly with some processing such as contrast adjustment and/or rotation rotation); or

ii) the image processing module 14 detects where the barcode pattern is located in the image, and extracts a portion of the image comprising the barcode pattern from the image (optionally with extra processing such as contrast adjustment and/or rotation) then forwards the extracted barcode pattern to the reader 16.

The barcode reader module 16 then reads and decodes the barcode from the image or pattern it receives from the image processing module 14, based on conventional barcode reading techniques. Alternatively any one both of (i) and (ii) may be intrinsic features of the barcode reader module 16 (designed to be used on an image of a printed or graphically displayed barcode captured in the conventional fashion, but which may equally work on the virtual barcode formed from the time- vary light modulation).

To generate the barcode at the emitting side 2, 4, there are a number of possibilities. In one embodiment, a user or application at the emitting side may simply provide the unencoded data that is desired to be communicated (e.g. a message or product identifier) to the controller 2 as its input. The controller 2 then automatically formulates this into a digital barcode signal suitable for representation as a barcode (i.e. a sequence of Is and 0s coded to digitally represent the pictorial pattern of light and dark stripes according to the relevant barcode format), then modulates the light based on this signal. For example the data may be generated by a device (e.g. mobile device at the emit-side) on the basis of a source of identification, validation or authorization information (e.g. for identifying validating or authenticating a user of the device). This identification, validation or authorization data may be part of a database that is stored remotely or on the device 6.

Alternatively, the controller 2 may receive, as its input, data that has already been formulated into a digital barcode signal (i.e. that already digitally represents the pattern of light and dark stripes of a barcode). In this case, the barcode signal may have been designed or intended by a user or application for printing on a physical surface, or for generating a barcode to be visibly shown on a graphical display. However, the controller 2 of the lighting apparatus converts this into the light-encoded form disclosed herein. Thus advantageously, in embodiments the disclosed system is able to take barcode data originally intended or designed to generate a visible barcode and make this available in an alternative or additional form with little or no extra effort by the user.

In one particular instance of this shown in Figure 3 a, the emit-side apparatus further comprises a barcode-scanner 1 (camera or laser based) for generating the input signal by reading an actual printed or graphically displayed barcode. I.e. the barcode scanner 1 reads the actual physical barcode in order to determine the digital barcode signal represented by that barcode and then supplies this signal as an input to the controller 2 where it is converted into a modulated light signal 5 that will mimic the same spatial pattern when captured with a rolling-shutter camera 8.

In another alternative, the barcode reader 1 may be operable to read a nonlinear or 2D printed or graphical barcode (e.g. a QR-code), and the controller 2 may be configured to convert this into a modulated light signal that will mimic another, ID spatial barcode pattern representing the same data when captured with a rolling-shutter camera 8 (note that the use of modulated light with rolling-shutter capture cannot mimic 2D barcodes).

In further embodiments, the controller 2 may be configured to modulate the emitted light 5 so that the pattern it forms on the image capture element 20 of the rolling- shutter camera 8 will in fact deviate slightly from the barcode format, but nonetheless within a tolerance of that format (and therefore the tolerance of the barcode reader module 16 designed to read that format). Particularly, this deviation is controlled by the controller 2 in order to reduce a visible flicker caused by the coding in the emitted light 5.

Some dedicated coded light formats have the property that the signal average over the duration of a packet always adds up to zero, so each packet is DC-free. This property is desirable in many scenarios as it ensures the coding does not cause a visible flicker in the light, or at least substantially reduces visible flicker compared to other schemes (in fact complete imperceptibility of coded light follows from sufficient absence of low frequency contributions in the perceptible range from DC to about 200 Hz).

However, the property that the data is DC free (or free of low frequency components) is not required for printed bar codes. Yet formats for printed bar codes are typically subject to one or more other constraints, such as the property that long runs of the same symbol, '0' or Ί ', cannot exist.

If the light source 4 is not intended for illumination of a space, but e.g. merely used as an indicator light or as a backlight of a small information display, then the barcode patterns could be encoded in the light 5 without concern for the non-DC-free property. Light flicker perception thresholds are higher in cases where the luminous modulated structures are smaller, particularly where the light does not fill the periphery of the user's vision. However, where the light source has a primary function of illumination, flicker will be much more noticeable, and so the encoding of traditional non-DC-free barcode formats in the light may not be so desirable without further measures being taken.

Therefore in embodiments, the controller 2 is configured to adapt the modulation in such a way that the packet containing the barcode is substantially DC-free (or at least has a lower DC content that it would otherwise), but nonetheless will still form a pattern in the captured image that will be recognized by the barcode reader module 16 at the receive side.

The DC-free property can be achieved by adaptation of symbol amplitudes (representing the printed '0' or Ί '). Here the controller 2 is configured to exploit the fact that for printed (or graphically displayed) barcodes a certified detector has to be able to deal with variations in printed reflectivity (e.g. there are ISO guidelines on this). Alternatively or additionally, the DC-free property may also be achieved by manipulation of the leading and trailing space around the code. Thus the system exploits the fact that the tolerances for detection of printed codes are broader than the tolerances for visible imperceptibility of light- embedded codes. An advantage therefore is that existing camera-based barcode readers require little or no adaptation to the system.

To illustrate this idea further, Figures 9a and 9b show examples of the barcode associated with the product barcode in Figure 7. Figure 9a shows the original bi-level (black- and-white) pattern of the original barcode including the quiet zones left and right of the pattern, following the minimal requirements for robust detection as prescribed by ISO/IEC 15416. Figure 9b shows an example of a modified barcode, which is based on the barcode in Figure 9a. The modified barcode is not bi-level, but rather is modified into a multi-level code, such that intermediate grey-levels appear when printed. In Figure 9b these grey levels are depicted in a dithered fashion for reproducibility. Note that the modified barcode in Figure 9b is still compliant to ISO/IEC 15416 and it allows proper detection by a regular barcode scanner. Figures 10a and 10b show examples of modulation signals associated with the product barcode in Figure 7. Figure 11 shows an example of a magnitude response associated with the signals in Figures 10a and 10b and in this example based on a symbol-rate /_symboi of 4 kHz. The modulation amplitude relative to the average light level can be expressed as a percentage of the average light level and can in principle be chosen at any value up to 100%, causing the light output to switch between 0 and a maximum level that is twice the average level. The choice of a relative modulation depth smaller than 100% can be a compromise between visibility of light-flicker and detectability of the data embedded in the modulation.

Figure 10a shows an example of a signal associated with the original bi-level barcode of Figure 9a. The channel- symbol value Ί ' is associated with the white spaces of the printed barcode and the channel- symbol value '-1 ' is associated with the black bars.

The signal in Figure 10a is not DC-free such that its application as a light- modulation signal can give rise to visible light flicker each time that a single packet (i.e. barcode) is transmitted in the light. In case of a repetitive transmission, the lowest light flicker frequency will be equal to the packet repetition frequency. The magnitude response of a single packet is depicted in Figure 11 and indicated by the solid curve. The solid curve shows the appearance of relatively strong signal magnitudes for the lower frequency range, part of which is in the range of perceptual flicker frequencies. This, in contrast to the dashed curve which is associated with the modified modulation signal depicted in Figure 10b.

Figure 10b shows an example of a modified modulation signal, and associated with the modified barcode in Figure 9b. Note that in this example this has been achieved by changing the signal levels associated with a bar and a space, "-1" and "1". In addition, each run of three or more equal channel symbols, have been replaced by sequence containing a variation in signal levels. Compared to the original signal in Figure 10a, the modified signal is DC-free and has also been modified to exhibit a suppressed signal magnitude in the same lower- frequency range such that requirements for absence of visible light flicker are more easily met. Note that the modulation signal in Figure 10b exhibits jumps between a limited set of possible signal levels, which has a benefit in that the modulation-signal can be readily implemented through generation by an LED driver. When captured with a rolling shutter camera, repetitive transmissions of this modified modulation signal will appear as repetitive realizations of the barcode pattern depicted in Figure 9b, but appearing as horizontal lines in the camera frame along the rows of the image sensor. As indicated before, while exhibiting suppressed light flicker visibility once captured with a rolling shutter camera, the resulting spatial pattern remains detectable as a ISO/IEC 15416-compliant barcode. Figure 12 shows a block diagram, schematically illustrating an example barcode signal modification algorithm for implementation in software and/or hardware of the emit-side controller 2, for generating a DC-free or substantially DC-free version of the barcode signal for modulation into the light 5 emitted by the light source.

The algorithm comprises: an input stage 24 for receiving an input signal in the form of an unmodified barcode signal, a convolution stage 26, a kernel (h) 28, a weighting (a) 30, a multiplication stage 32, a first subtraction stage 34, a signal averaging stage 36, a second subtraction stage 38, and an output stage for outputting the modified version of the barcode signal. The convolution stage 26 receives an instance of the input signal from the input stage 24 and convolves it with the kernel (h) 28. The result of this is output to the multiplication stage 32 where it is multiplied with the weighting (a) 30. The result of this is output to the first subtraction stage 34 where it is subtracted from an instance of the input signal from the input stage 24. The resulting output of the first addition stage 34 is supplied to the signal averaging stage 36 where it is averaged of the duration of a single packet. The second subtraction stage 38 subtracts this averaged signal with the instantaneous version of the output of the first subtraction stage 34. The result of this second subtraction forms the output signal which is output via the output stage 40.

The input signal of the algorithm is the original bi-level barcode signal composed of a sequence of channel symbols valued or "1". The output signal of the algorithm is a modified, multi-level barcode signal that is DC-free and exhibits less low- frequency signal energy relative to the input barcode signal. In this example, the input signal is convolved with kernel and after multiplication with a weighting factor subtracted from the input signal to create an intermediate multi-level signal that exhibits suppressed low- frequency signal energy. Subsequently, after subtraction of the signal average, an output signal is composed that is also DC-free. In this example the signal average is a fixed value equal to the average of the intermediate barcode signal calculated over one single packet, i.e. one single barcode duration including the quiet zone of ten symbols between packets. The

[1 1 1!

- - - I and a weighting factor a = 0.8.

However, in other embodiments other algorithms may be used to reduce the

DC component. For instance in another example, the symbol values may be varied randomly by a small amount within the tolerance of the relevant barcode standard. This will not result in a perfect DC-free signal, but the DC component can nonetheless be reduced relative to strict values of "-1" and "1". In another example, the modifications to the symbol values may be determined empirically by testing trial modifications and observing the resulting DC component.

With regard to the repetition of the barcode in the coded light signal, note that in embodiments the coded light transmitter 2, 4 is broadcasting the barcode pattern

repetitively, but also asynchronously with the receiver 6. This means, the top of the scan may correspond to the tail of the barcode pattern. However, the detection of a barcode start need not be a problem, and in embodiments fractional barcode patterns are not misinterpreted.

Firstly, barcodes can carry redundancy; for example in the EAN-13 barcode format, the last digit is in fact a decimal checksum that needs to match the checksum of the twelve preceding digits. The barcode reader 16 can use this to check that a full barcode pattern (corresponding to one full instance of a barcode) has been successfully received.

Further, in the repeated transmission, the controller 2 at the transmit side obeys the minimal requirements for a quiet zone following the ISO/IEC15416 standard (or other barcode standard). Note that in embodiments which apply a modification in order to reduce DC or low components, the quiet zone may also in fact be modulated, but this will still be within the standard tolerance of the barcode format. Moreover, by choosing the symbol clock such that at least two full barcode patterns fit inside in one image height (one frame), then at least one complete barcode pattern will always be visible (especially if the entire image or most of the image is filled with coded light). Thus a normal image-based barcode detection algorithm can detect the coded light without any need to stitch fractional messages before input to the barcode algorithm. Another alternative would be that less than two full barcode patterns, but still more than one, fit within one image height (one frame), and the frame rate is not equal to nor an integer multiple of the repetition rate of the barcode pattern; such that the pattern drifts across the frame area over multiple frames, and eventually a full barcode pattern will happen to fall within one frame. As another alternative, the image processing module 14 could use a coded- light stitching algorithm to stitch together partial barcode patterns from different frames.

Further, a convenient typical behavior of barcode readers is that they briefly pause after a detection to avoid repeated detections of the same barcode. As a result, once a complete barcode has been seen and detected from the light encoded barcode, detection stops. Further, mobile barcode apps can deal with all kinds of barcode appearances (orientations, perspective deformations). The appearance of a light encoded barcode will appear to the barcode reader 16 as just another variation, and the inventor has found mobile barcode detector apps to be quite robust when detecting barcode patterns formed via coded light.

Therefore the robustness of barcode readers is sufficient to robustly detect light encoded barcodes as well.

There are a number of other optional features that may be implemented at the receive side 6, 8 to enhance the user experience.

In embodiments, if the device 6 is a mobile device such as a smartphone or tablet, its controller 17 may be configured to output instructions to the user to hold the device 6 in an orientation that is most beneficial for capturing the pattern, e.g. so that the camera 8 faces the light source 4, and/or the light from the light source 4 more fully fills the view of the camera 8 in the case where no diffuser 9 is used. For example the instructions may be output via a display (not shown) of the device 6, and/or audibly.

In further embodiments, the device 6 may be equipped with a mechanism for bootstrapping itself, in case the device 6 comes pre-equipped with a barcode-reader application 16 but not the necessary image processing software 14 for obtaining the light- encoded barcodes from the light 5. To accommodate this, the device 6 may comprise an alternative communication means by which it can obtain a link (e.g. Internet link) from which it can download the image processing software 14 (or an update to existing image processing software). For example, the controller 17 may use the camera 8 and barcode reader application 16 to read a printed or graphically displayed barcode in the conventional fashion, and this barcode may communicate the link allowing the controller 17 to download the software 14 for adapting the device to be able to read light-encoded barcodes. From that point onwards, the device 6 does not have to rely on printed barcodes and can instead use their light-encoded equivalents. Alternatively the separate communication means could be an RF channel communicating to the device 6, e.g. an RF tag based on a near- field communication (NFC) technology, or a channel based on a short-range (unlicensed) RF transmission technology such as Wi-Fi, ZigBee or Bluetooth.

For instance, customers in a supermarket may be presented with a printed or graphically displayed barcode, an RF tag, or a Wi-Fi, ZigBee or Bluetooth signal; informing them via their devices 6 that they can scan LEDs on shelves to obtain more information.

In yet further embodiments, the controller 17 of the device 6 is configured to use its transmitter 18 to communicate a trigger to the light emitting apparatus 2,4, requesting the controller 2 to switch to barcode output. For example the trigger may be transmitted over a visible light communication link (in the opposite direction to the barcode data) or over an RF channel (with possible bidirectional communication).

An example use case could be where a user wants to obtain barcode data, but can't get physical access to the product. For instance, consider a scenario where the user wishes to check what type of light bulb is in a socket (e.g. which of multiple different types of LED-based bulb). During commissioning and/or assembly this is quite often done using barcodes, which works well if the bulb is in the user's hand where it can be easily scanned. If the bulb is an existing light in a ceiling and not readily accessible, another possibility would be to use a coded light ID, but that works only if a database or document is available linking the coded light ID to the barcode. Using embodiments of the present disclosure however, if this is not available, the device 6 could instead ask the bulb to output its barcode via the light it emits.

Another option would be to obtain barcode data from a product that is on display in a retail environment but not easily accessible, e.g. if the product is in a display case. If the product is illuminated with a light that can emit the barcode in accordance with embodiments of the present disclosure, the user may use his or her device 6 to ask the light to transmit the product's barcode in light-encoded form in order to see details of the product.

Note that it is not necessary for the device 6 to request the light-encoded barcode in all embodiments. For example, a light bulb or product may instead continuously emit a light-encoded version of its own barcode, or a product on display may be continuously be illuminated by or associated with light encoding the product's barcode. Thus the same barcode pattern is simultaneously made available in two different alternative forms, both the printed or graphically displayed form of an actual barcode, and a light-encoded form in accordance with the techniques disclosed herein. The user may then detect the barcode pattern however he or she finds most convenient or feasible depending on factors such as the capability of his or her device 6 and/or the accessibility of the bulb or product.

It will be appreciated that the above embodiments have been described only by way of example. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS:

1. Light-emitting apparatus for emitting light (5) to be detected by a rolling- shutter camera (8), the rolling-shutter camera having an image capture element (20) divided into a plurality of lines (22) which are exposed in a sequence, and the light-emitting apparatus comprising:

a light source (4) for emitting the light; and

a controller (2) for modulating the light emitted by the light source, wherein the controller is configured to encode data into the light using a time- varying binary modulation in an intensity of the light, formatted such that when detected by the rolling- shutter camera the time-varying binary modulation will form a binary pattern of light and dark stripes over the lines of the rolling-shutter camera according to a one-dimensional barcode format, being a same format as used for printed or graphically displayed one- dimensional barcodes.

2. The apparatus of claim 1, wherein the controller (2) is configured to receive an input signal comprising data formulated for display according to said one-dimensional barcode format for printed or graphically displayed one-dimensional barcodes, and to convert to said input signal into said one-dimensional binary modulation of the light (5) from the light source (4).

3. The apparatus of claim 2, comprising a barcode reader, wherein the controller

(2) is configured to receive said input signal by using the barcode reader to read a printed or graphically displayed barcode comprising said data, thereby converting the printed or graphically displayed barcode to said one-dimensional binary modulation of the light (5) for detection using the rolling-shutter camera (8) according to said same format.

4. The apparatus of any preceding claim, wherein the controller (2) is configured to adapt the data such that the levels and/or spacing of the stripes in said binary pattern deviate from the barcode format within a tolerance of said barcode format; the data being adapted to reduce a visible flicker in the light caused by the data, reduced relative to a strict adherence to said format.

5. A system comprising :

the light-emitting apparatus of any one of claims 1 to 4,

the rolling shutter camera, and

a barcode-reader module for reading the data from the pattern according to said format.

6. A device (6) comprising:

a barcode-reader module (16) for reading printed or graphically displayed one- dimensional barcodes according to a one-dimensional barcode format;

a rolling-shutter camera (8) for receiving light (5) emitted by a light source (4), the rolling-shutter camera being arranged to capture an image over a plurality of lines (22), each line of the image representing the received light captured at a successive time; and

an image processing module (14) for processing said image;

wherein data is encoded into the light using a time- varying binary modulation in an intensity of the light, formatted such that when detected by the rolling-shutter camera the time- varying binary modulation forms a binary pattern of light and dark stripes over the lines of the rolling-shutter camera according to said one-dimensional barcode format; and the image processing module is configured to perform one or more coded-light processing operations on the image in order to prepare said pattern for reading by the barcode-reader module according to the barcode format, and, following said processing, to output the pattern to the barcode-reader module for reading the data from said pattern according to said format.

7. The device of claim 6, wherein said one or more coded-light processing operations comprise:

applying a Wiener filter or other filter to mitigate a filtering effect of an exposure time of the camera; and/or

producing an average background by averaging a plurality of frames captured by the camera (8) at a frame rate, and subtracting the averaged background from said image, wherein the barcode pattern is repeated in said modulation with a repetition rate, and wherein each of said plurality of frames experiences the barcode pattern at a different spatial offset due to the repetition rate not matching the frame rate, said averaging thereby generating the averaged background image with the modulation substantially removed.

8. The device of claim 6 or 7, wherein the barcode-reader module (16) is a barcode reader application, either embedded in an operating system of the device (6) or running on the operating system.

9. The device of claim 6, 7 or 8, wherein the image processing module (14) is configured to de-focus the camera (8) from the light source (4) to obtain a greater coverage of the light (5) over the image than if focused on the light source.

10. The device of any of claims 6 to 9, wherein the device (6) is a mobile device, and comprises a user interface configured to output directions to a user to hold the device in a suitable orientation for capturing said pattern.

11. The device of any of claims 6 to 10, wherein:

the image processing module (14) comprises an image processing application for processing the pattern prior to being output to the barcode-reader module (16) for reading;

the device (6) is operable to communicate via a first medium, other than time- varying binary modulation of light being used to form a barcode in the lines (22) of the rolling-shutter camera (8); and

the device is configured to obtain a link for downloading the image processing application or an update thereto via the first medium, to download the image processing application or update based on said link as obtained via the first medium, and to subsequently use the image processing application or update to detect said pattern and output the pattern to the barcode-reader module.

12. The device of claim 11, wherein the first medium comprises:

an RF channel, or

using said camera or another camera of the device to read a printed or graphically displayed barcode.

The device of any of claim 6 to 13, comprising a transmitter (18) for sending to the light source (4), and a controller (17) configured to send a message to the light source via said transmitter requesting that the light source switches to a mode in which it modulates the light (5) with said time- varying binary modulation in order to communicate the barcode pattern. 14. A method comprising:

controlling a light source (4) to emit light (5) to be detected by a rolling- shutter camera (8), the rolling-shutter camera having an image capture element (20) divided into a plurality of lines (22) which are exposed in a sequence; and

encoding data into the light using a time- varying binary modulation in an intensity of the light, formatted such that when detected by the rolling-shutter camera the time-varying binary modulation will form a binary pattern of light and dark stripes over the lines of the rolling-shutter camera according to a one-dimensional barcode format, being a same format as used for printed or graphically displayed one-dimensional barcodes. 15. A computer program product comprising code embodied on at least one computer-readable storage medium and configured so as when retrieved from the storage medium and/or downloaded via communication network, and executed on one or more processors, to perform operations of:

encoding data into the light using a time- varying binary modulation in an intensity of the light, formatted such that when detected by the rolling-shutter camera the time-varying binary modulation will form a binary pattern of light and dark stripes over the lines of the rolling-shutter camera according to a one-dimensional barcode format, being a same format as used for printed or graphically displayed one-dimensional barcodes.