CN104969264A - Method and apparatus for adding annotations to plenoptic light fields - Google Patents

Abstract

A method, comprising the steps of: retrieving data (100) representing a light field with a plenoptic capture device (4); executing program code for matching (101) the retrieved data with corresponding reference data; executing program code for retrieving (102) at least one annotation (61, 63, 64) in plenoptic format associated with an element of the reference data; executing program code for generating annotated data in plenoptic format from the retrieved data and the annotation (103).

Description

Method and apparatus for adding annotations to plenoptic light fields

Background technique

The present invention relates to augmented reality methods and devices, and in particular, to methods and various devices for adding annotations to data corresponding to scenes.

Rapid progress in the development of hand-portable devices such as smartphones, palmtop computers, portable media players, personal digital assistant (PDA) devices, etc. has resulted in the inclusion of novel features and applications involving image processing. For example augmented reality applications are known in which a user points a portable device at a scene such as a landscape, a building, a poster, or a painting in a museum, and the display shows the image with superimposed information relating to the scene. Such information can include: names, for example, of mountains and residences; names of people; historical information for buildings; and commercial information, such as advertisements, eg restaurant menus. Examples of such systems are described in EP1246080 and in EP2207113.

It is known to provide annotation information to a portable device by a server in a wireless communication network. Annotation systems involving communication networks with servers and portable devices, as well as annotation methods, are also known.

Many annotation methods involve the step of comparing an image, such as a 2D image produced by a standard pinhole camera with a standard CCD or CMOS sensor, or a computer-generated image, with a set of reference images stored in a database. Since actual viewing angles and lighting conditions can vary with respect to images stored in the database, the comparison algorithm aims to remove the influence of these parameters.

For example, WO2008134901 describes a method in which a first image is captured using a digital camera associated with a communication terminal. Query data related to the first image is transmitted via the communication network to a remote identification server where a matching reference image is identified. An enhanced image is generated and displayed at the communication terminal by replacing at least a portion of the annotated image for a portion of the first image. The augmentation of the first image taken with the camera takes place in planar space and only deals with two-dimensional images and objects.

Ray information, such as the direction of the ray in each point in space, is discarded in traditional image annotation systems. Annotations without ray information make a realistic view of the annotated scene more difficult. For example, capturing or displaying textures on the surface of an object requires lighting information. Although each object has different textures on its surface, it is impossible to add texture information in current annotation systems. This results in attached annotations not actually being integrated in the scene.

In addition, the rapid growth of augmented reality applications may cause a flood of annotations in the future. Some scenes such as in a city contain many elements associated with different annotations, resulting in an annotated image with a very large number of annotations covering a large part of the background image. In many cases, the user is only interested in a limited number of those annotations, and the others are only distracting. Therefore, it will often be desirable to limit the number of annotations and provide a way of selecting which annotations should be displayed.

Furthermore, computational overhead is an important issue for annotated scene viewing. Computational expenses will need to be reduced.

It is therefore an object of the present invention to solve or at least alleviate the above mentioned problems of existing augmented reality systems.

Contents of the invention

According to the invention, these objects are achieved by a method comprising the following steps:

Retrieving data representing the light field with a plenoptic capture device;

executing program code for matching captured data to corresponding reference data;

executing program code for retrieving annotations in plenoptic format associated with elements of said reference data;

Program code is executed for generating annotated data from said captured data and said annotations in a plenoptic format.

The invention is also realized by an apparatus for capturing and annotating data corresponding to a scene, said apparatus comprising:

a plenoptic capture device for capturing data representing a light field;

processor;

monitor;

program code for, when executed, causing the processor to retrieve at least one annotation in a plenoptic format associated with an element of data captured with the plenoptic capture device, and for The view generated from the captured data and including the at least one annotation is rendered on the display.

The invention also provides apparatus for determining annotations, said apparatus comprising:

processor;

memory;

program code for, when executed, causing the processor to receive data representative of a light field, match the data to a reference data, determine from the memory the current the annotation in the plenoptic format and sending the annotation in the plenoptic format or data corresponding to the image of the annotation in the plenoptic format to the remote device.

The claimed addition of annotations in plenoptic formats allows for a more realistic integration of annotations in images in plenoptic formats; the annotations appear to be elements of the captured scene, not just text superimposed on top of the image. Annotations in the plenoptic format (also called "plenoptic annotations" in this application) contain a more complete description of the light field than conventional annotations, including information on how to modify the light.

Annotation provision in plenoptic format also allows selection of annotations that should be displayed, depending on the focal length and/or viewpoint selected by the user during the reproduction of the image or eg automatically based on his interests.

Since the annotations are in the same space (ie plenoptic space) as compared to the captured data, the computational overhead for the annotation process is reduced.

In particular, the computational expense for rendering plenoptic data in a human understandable format is reduced. In fact, since the image in the plenoptic format and the plenoptic annotation are in the same space, the rendering process is equivalent for both. In one embodiment, a single rendering process can be used to render images and associated annotations. In this case, the projection parameters selected for the plenoptic reconstruction process (such as change of viewpoint, depth, choice of focus...) are also applied on the plenoptic annotation. For example, the same transformation can be used to display plenoptic annotations at various distances when changing the focus or viewpoint of the plenoptic image. In another embodiment, the effect of the annotation is applied to the captured plenoptic image and a rendering of the modified plenoptic image is performed.

Thus, plenoptic annotations, ie annotations in a plenoptic format, provide a realistic way of displaying annotations, allow more types of annotations including annotations with textures, and increase computational efficiency.

Unlike traditional annotations, plenoptic annotations can contain as much information about rays as images captured by plenoptic capture devices. Thus, it is possible to synthesize annotations directly in the captured light field without losing the ray information caused by the projection onto the 2D image. For example, annotations can preserve the characteristics of light reflections on the surfaces of the annotated objects, which is not possible with traditional annotation systems. In this sense, the annotated view seems more realistic.

Direct modification of rays can facilitate computations such as the simultaneous generation of annotated scenes from multiple viewpoints. In the example of annotated scene generation, annotation processing and other additional processing such as blurring or sharpening to the scene is applied directly once in plenoptic format, instead of attaching annotations and applying additional processing on the generated 2D image for each viewpoint. Therefore, direct synthesis of plenoptic images and plenoptic annotations in a plenoptic format can lead to reduced computational overhead.

The invention also relates to a method for attaching annotations to a reference image in a plenoptic format, the method comprising:

presenting said reference image in a plenoptic format with a viewer;

select comment;

selecting with the viewer a location for the annotation and one or more directions from which the annotation can be seen;

The position and the orientation are associated in memory with the reference image and the annotation in a plenoptic format.

This method can be performed with a suitable authoring system such as a suitable software application or website.

Description of drawings

The invention will be better understood with the aid of the description of an embodiment given as an example and illustrated by the accompanying drawings, in which:

Figure 1 schematically illustrates a plenoptic capture device for capturing data representing a light field of a scene with an object at a first distance.

Figure 2 schematically illustrates a plenoptic capture device for capturing data representing a light field of a scene with an object at a second distance.

Fig. 3 schematically illustrates a plenoptic capture device for capturing data representing a light field of a scene with an object at a third distance.

Figure 4 schematically illustrates a system comprising various device elements together embodying the invention.

Figures 5A-5B show annotated views rendered from the same plenoptic data, where the viewpoint selected by the user during the rendering has changed between the two views, resulting in the same annotation rendered differently.

Figures 6A-6B show views of annotations rendered from the same plenoptic data, where during the rendering the viewpoint selected by the user has changed between the two views, resulting in making the first annotation visible on the first view And make the second annotation visible on the second view.

Figures 7A-7B show views of annotations rendered from the same plenoptic data, where during the rendering the focal length selected by the user has changed between the two views, resulting in making the first annotation visible on the first view And make the second annotation visible on the second view.

8 is a block diagram of a method for generating and rendering views with annotations in a plenoptic format.

9 is a block diagram of a method for modifying the rendering of annotations when a viewer selects a different viewing direction and/or a different focal length on a view.

10 is a block diagram of a method for associating annotations in a plenoptic format with reference data.

11 is a block diagram of a method for sequential annotation of a series of plenoptic images, such as video plenoptic images or plenoptic images captured by a user on the move.

Detailed ways

Conventional cameras capture a 2D projection of a scene on a sensor and generate data indicating the intensity of light on each pixel with or without color. On the other hand, plenoptic capture devices known as such capture data representing the light field, ie not only the intensity of the light but also a matrix containing more complete information about the light field including the direction of the light.

A complete light field can include up to 7 parameters to describe each ray (or to describe a ray at a given position): 3 for position, 2 for direction, 1 for wavelength and (in the video case) 1 for time. Some current plenoptic cameras deliver plenoptic data comprising: 2 parameters for position, 2 parameters for direction, and 1 parameter for wavelength. The sensor generates plenoptic data representing a so-called plenoptic light field, ie a matrix indicating at least the position and direction of light rays. It means that plenoptic data generated by a plenoptic capture device contains more information about the light field than conventional 2D image data generated by a conventional 2D camera.

So far, at least two companies, Lytro and Raytrix, have proposed plenoptic sensors capable of recording such plenoptic light fields. Their two cameras are slightly different in design, but the main idea is to break down the different directions of light that is thought to fall on a single photosite (or pixel) in a standard camera sensor. For that purpose, as illustrated on Fig. 1, an array of microlenses 20 is placed behind the main lens 1, replacing the sensor of a conventional camera.

In that way, the microlens 21 redirects the light according to the incident angle of the light, and the redirected light reaches different pixels 210 of the sensor 21 . The amount of light measured by each of the NxM pixels 210 forming a sub-image depends on the direction of the light beam hitting the microlens 20 in front of that sub-image.

1 to 3 illustrate a simple one-dimensional sensor comprising n=9 sub-images each having a row of N×M pixels (or light bits) 210, in this example N equals 3 and M is equal to 1. Many plenoptic sensors have a higher number of sub-images and a higher number of pixels for each sub-image, say 9×9 pixels, allowing between N×M=81 different orientations of light on the microlens 20 Make a distinction. Assuming all objects of the scene are in focus, each sub-image thus contains patches of luminance values indicating the amount of light onto that sub-image from various directions.

In this construction, the array of microlenses 20 is located on the image plane formed by the main lens 1 of the plenoptic capture device and the sensor 21 is located at a distance f from the microlenses, where f is the focal length of the microlenses. This design allows high angular resolution but suffers from relatively poor spatial resolution (the effective number of pixels per rendered image is equal to the number of microlenses). This problem is addressed by other plenoptic capture devices where the microlenses are focused on the image plane of the main lens, thus creating a gap between the microlenses and the image plane. The price to pay in such a design is poor angular resolution.

As can be observed on FIGS. 1 to 3 , the plenoptic light field corresponding to a scene with a single point 3 in this example depends on the distance from the point 3 to the main lens 1 . In Fig. 1, all beams from this object reach the same microlens 20, thus resulting in a plenoptic light field in which all pixels in the sub-image corresponding to this microlens record a first positive light intensity, while those corresponding to other lenses All other pixels record a different, null light intensity. On Fig. 2 where object 3 is closer to lens 1, some beams originating from point 3 reach pixels of other sub-images (ie sub-images associated with the two microlenses adjacent to the previously hit microlens). On FIG. 3 , where object 3 is at a greater distance from lens 1 , some beams originating from point 3 reach different pixels associated with the two microlenses adjacent to the previously hit microlens. Thus, the digital data 22 delivered by the sensor 21 depends on the distance to the object 3 .

The plenoptic sensor 21 thus delivers plenoptic data 22 containing for each sub-image corresponding to the microlens 20 (N×M) values indicating the amount of light from various directions on the lens above this sub-image collection. For a given focused object point, each pixel of the sub-image corresponds to an intensity measurement of a ray hitting the sensor at a particular angle of incidence φ (in the plane of the page) and θ (perpendicular to the plane of the page).

Figure 4 schematically illustrates a block diagram of an annotation system embodying the invention. The system includes user devices 4 such as handheld devices, smartphones, tablets, cameras, glasses, goggles, contact lenses and the like. The device 4 comprises: plenoptic capture means 41, such as the camera illustrated in Figures 1 to 3, for capturing data representing the light field on the scene 3; a processor, such as a microprocessor 400 with suitable program code ; and a communication module 401 , such as WIFI and/or a cellular interface, for connecting the device 4 to a remote server 5 such as a cloud server through a network such as the Internet 6 . The server 5 comprises: a memory 50 with a database such as a SQL database, a collection of XML documents, a collection of images in a plenoptic format, etc., for storing reference plenograms representing images and/or one or more global models collection of data; and a processor 51 comprising a microprocessor with computer code for causing the microprocessor to perform the operations required in the annotation method. Annotations and corresponding positions can also be stored in memory 50 together with reference plenoptic data.

The program code executed by the user device 4 can include, for example, application software or application programs (apps) that can be downloaded and installed in the user device 4 by the user. The program code can also contain part of the operating code of the user device 4 . Program codes can also include codes embedded in webpages or executed in browsers, such codes include Java, Javascript, HTML5 codes, etc. The program code may be stored as a computer program product in a tangible device readable medium such as flash memory, hard disk, or any type of permanent or semi-permanent memory.

Execution of program code by the microprocessor 400 in the user device 4 is used to cause this microprocessor to send at least some of the captured data sets corresponding to the light field, or characteristics of those data sets, to the remote server 5 . The program code is arranged to send the data in a "plenoptic format", ie without losing information about the direction of the light rays. The program code can also cause the microprocessor 400 to receive from the server 5 annotated data in plenoptic format, or annotated images, or annotations associated with previously transmitted plenoptic data, and to render images corresponding to the annotated images. A view of the captured data.

Plenoptic annotation methods can include two parts: an offline process and an online process. Typically, the main purpose of the offline process is to associate annotations with reference images in plenoptic format, or with other 2D, stereoscopic, or 3D reference images.

offline stage

In the case of reference images in plenoptic format, the offline process may include steps such as the following:

1. receiving from the device 4 reference data in plenoptic format and representing the light field;

2. Presenting a reconstructed view of the plenoptic reference image, such as with a plenoptic viewer;

3. Select Plenoptic Annotation,

4. Select the position and orientation for the annotation in the rendered view,

5. Select one or more light field parameters of the annotation,

6. (Optionally) attribute actions to annotations,

7. Based on its position and orientation, associate the reference image ray with the annotation ray in memory.

This offline process can be performed on the server 5, in the user device 4, or in yet another device such as a personal computer, tablet or the like. Typically, this offline process is performed only once for each annotation associated with the reference image. If the selected annotation is not originally available in the plenoptic format, it can be converted into the plenoptic format.

The main purpose of the offline process is to add plenoptic annotations to plenoptic images. The offline process can include two phases. The first stage may be performed by program code executed by a microprocessor in the server 5, which may contain an executable or other code for causing the server 5 to perform at least some of the following tasks:

1. receiving data from the device 4 in a plenoptic format and representing a light field;

2. Retrieving a previously stored model (reference image) and/or a plurality of reference data from the database 50;

3. matching data received from the user device to a portion of the reference image, respectively to one of the plurality of reference images,

4. Determining annotations associated with matching reference images;

5. Send the annotation in plenoptic format or the image of the annotation in plenoptic format to the device 4 .

In various embodiments, instead of sending the captured data to the remote server 5 for matching with a reference image in the server, this matching can be performed at the user using a set of reference images stored locally or using a model stored locally in the device. done locally on the device. In this embodiment, the server 5 is hosted on the user device 4 . The online process can be performed several times according to the user's requirements.

The second stage of the on-line process may be performed by program code executed by a microprocessor in the device 4, which may contain an executable or other code for causing the device 4 to perform at least some of the following tasks :

1. Receiving annotation data in plenoptic format from the server 5, possibly together with associated actions;

2. Apply the received annotation data to the captured plenoptic light field;

3. Reproduce the annotated light field into a user-viewable view;

4. Interpret user interactions and perform associated annotation actions.

In various embodiments, instead of applying the received annotations to the captured plenoptic light field on the device 4, this step can be done on the server 5 side. In this case the final rendered view is transmitted back to device 4 or the entire annotated light field is transmitted back to device 4 .

Thus, the user is able to associate the annotation with a particular position and orientation with respect to the rendered view of the plenoptic reference image, and indicate which light field parameter or parameters the annotation should use in this particular view. Depending on the viewpoint selected by the viewer during rendering of the view, the same annotation may be rendered differently. Because the light field parameters of annotations can change, if the viewer chooses a different viewpoint, the first annotation can be replaced by a second annotation at the same position.

An example of a flowchart for an offline process is illustrated on FIG. 10 . This flowchart illustrates the method that allows the user to select the annotation that must be associated with the reference image, as well as the position, orientation and light field parameters for this annotation, so that this annotation will be applied to a captured image that matches this plenoptic reference image. plenoptic image.

This method may use an annotation authoring system that may run locally in the user's device 4 . The annotation authoring system can also be hosted on a server 5 where the web platform presents some tools to manage annotations and relate them to plenoptic reference images. Services such as augmented reality usage statistics may also be available from the web platform. The annotation authoring system can also run on different servers or devices (including users' personal computers, tablets, etc.).

In step 150, the user selects a reference image, such as an image in a plenoptic format. This image was uploaded on the plenoptic authoring system and used as a supporting image for annotation.

As part of the plenoptic authoring system, the viewer renders the uploaded data to the user in a manner that enables the user to visualize the uploaded data. If the data is in a plenoptic format that cannot be easily interpreted by a human as such, this may involve using a plenoptic rendering module to render the plenoptic model in a space intelligible to the user. The viewer constitutes the tool that manipulates the plenoptic data and places annotations at desired positions and orientations for a given view, but all processing and combination with plenoptic annotations is done directly in plenoptic space.

In one embodiment, the plenoptic model can be rendered as a 2D view so that the user can visualize the 2D view from one viewpoint at a time and at one focal length at a time, allowing him to understand and edit the plenoptic model. For navigating from one 2D view to the other, controls are available such that another 2D view can be displayed on request.

In another embodiment, the plenoptic model can be rendered as a partial 3D scene, where different directions of light rays can be visualized. The main difference from a standard full 3D scene is that the 3D scene detection is limited when it is reproduced from the plenoptic model. For example, the view direction as well as the view position are restricted to those already captured by the plenoptic capture device.

In step 151 the user selects the plenoptic annotation that he wants to associate with a specific element or position of the plenoptic model. As already mentioned, plenoptic annotations are defined in plenoptic space and are thus described in terms of rays. Those rays can describe elements such as text, images, video, or other elements acting directly on the plenoptic image rays. Plenoptic annotations can be retrieved, for example, from a library of plenoptic annotations in a database or in a file browser. Plenoptic annotations can also be created on the fly, for example by capturing it with a plenoptic capture device, entering text with a text editor, drawing images and/or recording sound or video.

In one embodiment, plenoptic annotations can be presented as previews on the authoring system in a library or list. The plenoptic annotation preview corresponds to the rendition of the annotation used for the default view. This default view can be randomized or in a preferred embodiment taken as an intermediate view corresponding to the plenoptic annotation extent with respect to position and orientation. The preview allows the user to get a quick and clear idea of what the plenoptic annotations correspond to. For the general type of annotations that do not act on the model wavelength, ie these annotations are not visualized as such, the preview plot is applied to the central annotation of the current model view rendered by the authoring system. Therefore, if this type of annotation has only the effect of rotating all model rays by 10°, the preview will consist of the central part of the view rendered by the current model, where each ray has been rotated by 10°.

In step 152, the user selects with the plenoptic annotation authoring system a location in the coordinate system of the rendered view of the selected reference model at which he wants to add a plenoptic annotation. This can be done, for example, by dragging the annotation at the desired location from the annotation preview list on top of the displayed view, and possibly translating, rotating, resizing, cropping and/or otherwise editing the annotation. Alternatively, the user can also enter coordinates as values in the control plane.

In step 152', the user can adjust the parameters of the annotation ray to generate another view of the annotation. When the user changes the parameters of the annotation using, for example, a computer mouse pointer to change the orientation of the annotation, the rays of the annotation are combined with the rays of the plenoptic model, and for each new position or new orientation a new 2D view. This is made possible when the user mouse pointer and its movement are projected into plenoptic space. The movement of the pointer is then applied to the annotation in a plane parallel to the virtual plane corresponding to the 2D rendered view.

Once the plenoptic model and the annotated rays are combined, the effects of the annotations are applied to the rays of the reference image. The process of overlaying plenoptic annotations can be viewed as the process of modifying the light. The captured plenoptic data can contain information about the direction of the rays, the wavelength (ie color) for each ray, thus annotations can be considered as modifications of those parameters. For example, attaching text on the surface of an object can be seen as a modification of the wavelength of light at specific regions on the surface.

The type of effect produced by an annotation is determined by the annotation itself. In one embodiment, plenoptic annotations consist, for example, of opaque text only. In this case, the model ray wavelengths are completely replaced by the annotated ray wavelengths for the mapped rays. For other annotations, by taking into account the annotation changing the texture of the model, the model's rays can have their directions changed by the annotation to reflect the new texture. In yet another example, model ray positions may be altered by annotations.

Plenoptic annotations can be viewed as filters that modify light. This provides more possibilities for displaying annotated scenes. A further example of this process is changing the direction of a ray of light. As an example, a glow effect can be applied to rays incident from a specific object in a captured plenoptic image by adding randomness to the direction of the rays. Ability to make annotated objects reflective. Another example is modification of properties of a surface, such as modification of texture information. Since plenoptic annotations allow modification of variables of light rays, such as direction and wavelength, it is possible to modify the surface of an object as if textures were added to it by combining modifications of variables. For example, plenoptic annotations can change a flat surface with a red color to a rough surface with a yellow color by modifying the direction and wavelength.

Information describing the effects of annotations on model rays may be stored in a plenoptic annotation array, as will be described in step 154 .

In step 153, the user selects one or more annotated light field parameters. This could for example be the wavelength of an annotation to change its color. The user may also define different appearances for the same annotation viewed from different directions, or even different annotations associated with the same element viewed from different directions.

Alternatively, the user can choose to navigate to another view of the plenoptic viewer once successfully adjusted on the rendered plenoptic model. Plenoptic annotations are automatically reported on new views of the plenoptic model. The user can then decide to edit the annotation, changing its light field parameters or appearance for this particular view. He can proceed in the same way for all available views of the plenoptic model.

An interpolation process may occur between the first and second views of the plenoptic annotation to prevent the user from having to navigate through all views of the plenoptic model. These two views of the plenoptic annotation do not have to be contiguous. The user must specify the appearance of the annotation in both views and the plenoptic authoring system will automatically generate an intermediate view of the plenoptic annotation. Other views of the plenoptic model that have not been associated with the annotation will not display it, leading to the possibility that the annotation is not reproduced for a particular viewpoint or focal plane of the scene.

Plenoptic annotations may include data corresponding to rays and described in a set of parameters. When rendering plenoptic annotations for a first specific view, the viewer sets some parameters and allows the user to modify other parameters. Navigating from this view to a second view, the user changes parameters that must be fixed by the viewer while being able to modify other parameters. The interpolation process automatically calculates the ray parameters of the plenoptic annotation between these two views.

In one embodiment, the parameters of each plenoptic annotation can be as follows: 3 (or possibly 2) parameters for the position of rays in space, 2 parameters for their direction, 1 parameter for their wavelength and possibly 1 parameter for time. For a particular view rendered by a plenoptic viewer, the parameters of position, orientation and time can be set eg by the viewer. The user can then change a parameter that is not fixed by the viewer, which in this example corresponds to the wavelength of the light. Let us assume that the user sets it to the first value v1. Now for another view of the annotation, ie for different values of the position, direction and time parameters, let us assume that the user changes the wavelength value for the second view and sets it to say v2. The interpolation process aims at computing annotation values between v1 and v2 for views intermediate the position, orientation and time parameters associated with the first and second views. In other embodiments, interpolation can also be considered to calculate values for other parameters of the plenoptic data including position, direction, wavelength and/or time.

Specific examples of interpolation include, for example: a change in the color of a plenoptic annotation, such as from an orange color to a more reddish color; a change in the visibility of an annotation, where for a particular view an annotation is visible, but for another view, Annotations are hidden.

Different methods of interpolation are possible, including eg linear, quadratic or higher order interpolation between two views of annotations. Also, more advanced interpolation methods can take into account other characteristics of the scene or the annotation itself to generate new rays for the annotation.

Actions can also be associated to all or some of the annotations when they are displayed on the captured image in step 153'. These actions can be triggered by the user or performed automatically using, for example, a timer. Actions include: launching a web browser with a specific URL; making annotations active, such as making one move, appear or disappear; playing a video; launching a menu presenting further possible actions; starting a slideshow or playing an audio file. Actions that allow modification of the view of plenoptic data presented to the user are also possible, such as actions allowing the view of plenoptic data to be focused at a given focal length.

In step 154 the plenoptic annotation is stored and associated in a memory with the corresponding position, orientation and with the selected reference plenoptic model, such as in the database 51 or in the user's device. Knowing the required annotations, it is possible to store annotations attached to each reference plenoptic model in plenoptic format. Each annotation is stored as a separate plenoptic file.

The plenoptic annotation reference data is generated from the plenoptic reference data and the corresponding one or more plenoptic annotations. This augmented reality model takes the form of a file containing all the information required to render back to the plenoptic model with its associated annotations. It thus describes the relationship between plenoptic reference data and its annotations. The plenoptic annotation's reference data can be rendered directly on the plenoptic annotation authoring system to pre-visualize the results, and directly on the client side to render a certain plenoptic augmented reality.

Information describing the effects of annotations on model lighting is stored in the plenoptic annotation data. Modifications defined by annotations act on model light parameters. Thus, annotations can describe modifications such as model ray direction, position, time or wavelength. In other words, this information describes the functionality of the model ray.

Each ray of the annotation is assigned a unique identifier when the annotation is created. When applying annotations on the authoring system, annotation ray unique identifiers are matched to their model's corresponding rays. So, each ray of the model is assigned an annotation ray identifier, which is then used by the system when it has to apply annotations on the model ray by ray, as is generally the case eg in the online phase.

Annotation information can be stored in a 2-dimensional array, where each ray contains information about its effect on the model for each parameter. The unique identifier of the annotation ray is then used to define the corresponding ray effect in the array for each parameter. In other words, the first dimension of the array corresponds to the rays, which are referred to by their identifiers, and the second dimension corresponds to their parameters, ie the light field parameters. Any annotation can be fully represented using this format, since any modification of a model ray for any parameter can be represented in an array.

In one embodiment, annotations can modify all model ray directions by say 10° for an angle. As illustrated in Table 1 below, the 2-dimensional array then contains 10° in the column corresponding to the parameter of the orientation angle. The column reads 10° for all rays, since they are assumed to all act the same way. When it is desired to apply the effect of an annotation on its corresponding model ray, the system will first identify the annotation and model ray pair, extract the unique identifier corresponding to the annotation ray, query the annotation table to see what effect this annotation ray has in order to finally apply the Changes are applied to the model light. In this example, the angles of all model rays affected by the annotation will be rotated by 10°.

Table 1. Annotation array.

As an example of the offline phase, a user might want to add text annotations to a scene containing buildings. Also, the text annotation color will need to vary from one viewpoint to another. The following steps will then be completed by the user:

1. The plenoptic capture of the building is uploaded to the plenoptic annotation creation system

2. Rendering a 2D view from the captured plenoptic image and presenting the 2D view to the user

3. The user selects a text annotation type from the annotation type list, enters his text and drags the text annotation onto the rendered 2D view

4. The user is able to move the viewpoint or annotation position and orientation of the rendered 2D view so that the annotation appears exactly as the user wants

5. The user sets the text color for the currently rendered viewpoint

6. The user moves the viewpoint of the reproduced plenoptic image to another location

7. The user sets the text color to another value for this other viewpoint

8. The plenoptic annotation model is then saved and ready for the online phase of the annotation process.

The plenoptic annotation authoring system performs the following tasks to generate a suitable annotation model based on the previously described user action steps for text annotation:

1. The 2D view is rendered to the user based on the viewpoint settings initially set to default values

2. Plenoptic versions of text annotations are generated by tracing rays from text objects to virtual viewpoints. This creates a collection of rays, each described by a unique identifier. This collection of rays describes the text. These rays are represented in memory by arrays corresponding to the modifications that must be applied to the reference plenoptic image. In this case, the array will contain values for the wavelengths that rays must take to match the annotation ray

3. Annotations are initially located at default positions predefined in the authoring tool. Annotation rays are combined with reference plenoptic image rays. These relationships between annotations and rays of the reference image are stored for future use using ray unique identifiers

4. When the user moves/changes the orientation of the annotation using eg a computer mouse pointer, the different rays of the annotation are combined with other rays of the captured plenoptic image and a new 2D view is generated for each position or orientation modification. This is made possible when the user mouse pointer is projected in plenoptic space. The translation of the pointer is then applied to the annotation in a plane parallel to the virtual plane corresponding to the 2D rendered view. When the annotation is moved, the light relationship between the annotation and the reference image is changed and updated according to the annotation position or orientation change

5. When the user selects a color to use for text for the current viewpoint, the wavelength value of the annotation array is changed to match the selected color

6. When a new viewpoint is selected and a new text color is selected, the wavelength values of the annotation array corresponding to the rays used to generate this new rendered view are changed. Wavelength values intermediate between the first and second viewpoints are interpolated using standard or point-to-point interpolation methods

7. When the user saves the model, the plenoptic annotation array is saved with the uploaded plenoptic reference model so that it can be used in the online phase.

online phase

As explained before, the online phase of the overall annotation process occurs when a user capturing a plenoptic image wants that image to be annotated.

The online stage of the annotation process is applied to the input plenoptic image to get the final plenoptic annotated image. This consists of matching the input image to some reference model, retrieving annotations for the matched reference model, combining the annotations with the input plenoptic image, rendering the annotated view to the user in an intelligible form, and possibly processing the user Interact to generate different actions defined on annotations.

Since the annotation content composed of rays is in plenoptic format and the captured image is also in plenoptic format, those two data sets are in the same space. Annotations can thus be applied directly to the plenoptic image without further projections. The modified plenoptic space to which the annotations have been applied can then be projected into eg a 2D view. This also means that the projection parameters chosen for the plenoptic reconstruction process (such as change of viewpoint, depth, choice of focal point, ...) are also implicitly applied on the plenoptic annotation. For example, annotations will have effects applied to them when changing the focus or viewpoint of the rendering process.

The online plenoptic annotation process as illustrated on Fig. 8 comprises a first step 100 during which data representing a light field in a plenoptic format (plenoptic data) is retrieved. The plenoptic data may be retrieved by a device 4 capturing the data with a plenoptic capture device, or by a device such as a server 5 receiving the plenoptic data from the device 4 via a communication link.

In step 101, the retrieved data is matched with reference data. This step can be performed in the device 4 and/or in the server 5 . This step may involve determining the set of features in the captured data, finding matching reference data representing a reference image with matching features, and registering the captured data with the reference data, as eg described in US13645762. The reference data may represent images in plenoptic format or other images, and may be stored in a memory 51, such as a database, accessible from a plurality of devices. The identification of matching reference data may be based on the user's location, time, hour, signals received from elements of the scene, indications given by the user and/or image similarity. The registration process aims to find the geometric relationship between the user position and the reference data, so that the transformation between the rays of the captured plenoptic image and the rays from the matched plenoptic reference image can be inferred.

In step 102, plenoptic annotations associated with matching reference data are retrieved, eg from memory 51 . This annotation is in plenoptic format, i.e. described in terms of rays. Those annotation rays may represent, for example, text, still images, video images, logos, and/or other elements acting directly on the plenoptic image rays.

Annotations can contain sounds in the plenoptic space, e.g. sounds for a specific group of rays attached to the plenoptic reference image, whereby the sound will only be for which selected rays are also visible in the plenoptic image and/or in focus Some directions to play.

In step 103, the retrieved annotations in plenoptic format are combined with the captured plenoptic data to generate annotated data representing the annotated image in plenoptic format. This combination can take place in the server 5 or in the device 4 . In the latter case, the server 5 can send the annotated data to the device 4, which then assembles it. This combination of annotations is made possible because the transformation that projects the ray projections of the reference image to the captured plenoptic image is known from the matching step (step 101 ). Annotations can thus also be applied to captured plenoptic images.

Plenoptic annotations can be applied to captured plenoptic images using the following methods:

1. Find the transformation used to project the reference plenoptic image ray onto the online plenoptic image ray retrieved in step 100 of FIG. 8 ;

2. Annotations for each retrieval of the reference plenoptic images defined in the offline phase:

1. Identify and select which rays of the reference plenoptic image must be modified according to the annotations by reading the annotation array defined in the offline phase

2. Project the rays identified in point (1) onto the online plenoptic image. This creates a correspondence between selected rays of the reference plenoptic image and rays from the captured plenoptic image

3. For each ray of the captured plenoptic image that has been selected at point (2), apply the transformation to the ray as defined in the plenoptic annotation array. The array is used as a lookup table, where the parameters of the rays and transformations (such as wavelength, direction...) that can be identified due to the selection process of steps (1) and (2) are used as lookup keys.

As an example, if the annotation ray represents text, the annotation array will contain a single non-null light field parameter which is the wavelength corresponding to the color of the text. The captured plenoptic image rays will thus be modified by increasing/decreasing the wavelength of the rays by the factor stored in the annotation array. This factor is looked up in the array using the transformation between rays computed during registration.

In step 104 a view, such as a 2D or stereoscopic view, is rendered from the annotated data and displayed to the user/viewer, such as on a display 40 or with another device. This view rendering process is described in more detail below in conjunction with FIG. 9 .

In step 105, interaction with annotations is enabled. The system can react to different events to perform specific actions previously defined in the offline part of the annotation process. Such an event could be a user interaction with an annotation. A user can point to and interact with a given annotation by means of a touch screen, hand tracking sensor, or any other input means. This interaction will generate interaction events capable of triggering specific actions defined in the offline phase of the annotation process.

Another possible type of event is an event that fires when a particular change in the scene is detected. As explained later in this section, occlusions of objects by reference models in captured plenoptic images can be detected. This occlusion event can trigger actions previously defined in the offline phase of the annotation process. As another example of possible events that trigger annotation actions, a sound recognition module could be used to trigger certain actions based on a certain type of sound detected.

Figure 9 illustrates the rendering of a view and the various possibilities for the viewer to subsequently modify the rendering. As previously indicated, the augmented reality view is rendered in step 104 from the annotated data generated from the captured view and the annotation data in plenoptic format as previously described with FIG. 8 . The rendered view can be a standard 2D view as produced by a pinhole camera, a stereoscopic view, a video, a holographic projection of plenoptic data, or preferably a dynamic image that presents the image with some commands for refocusing and/or changing viewpoint module. The dynamic image module can be a function capable of rendering a plenoptic image as a command value or an HTML5/Javascript webpage as a Flash object, or any other technology allowing the dynamic presentation of several images. Examples of views that may be rendered during step 104 are shown on Figures 5A, 6A, and 7A. The view on FIGS. 5A and 6A contains object 60 with annotation 61 . An additional object 62 at a different depth and thus out of focus is also seen on the view of Fig. 7A. Refocusing or changing viewpoint can be triggered manually by the user (such as by selecting an object or location on or around the image) or automatically (such as when the user moves).

In step 105, the user inputs a command for modifying the viewpoint to generate a novel view from the same plenoptic data during step 107, the novel view corresponding to the same scene viewed from a different viewpoint. Algorithms for generating from plenoptic data various 2D images of a scene as seen from different viewpoints or viewing directions are known as such and are described eg in US6222937. An example of a modified 2D image resulting from this command and executed by the viewpoint selection module 403 is illustrated on FIG. 5B . As can be seen, not only the perspective of the object 60 but also the perspective of the annotation 61 has been modified by this command. In fact, since the annotations are applied directly on the plenoptic space represented by the input plenoptic data, when the view is generated from the plenoptic space, the annotations appear to be transformed in the same way as the plenoptic images. This produces more realistic annotations.

Some annotations may only be visible from the first set of viewing directions, but not from other directions. Thus, as illustrated in Figure 6B, a change of viewpoint during step 105 may result in a new view in which one annotation 61 is made invisible but a new annotation 64 associated with the same object is revealed. Multiple annotations may be associated with a single location of the reference image, but with different viewing directions. Due to different annotation light field parameters set in the offline phase of the annotation process, the annotation itself may also look different when rendered from a first viewing direction compared to a second different viewing direction. The change in appearance can be defined by the annotation itself, but it can also be a function of the input plenoptic image.

In step 106 of Figure 9, the user enters commands for refocusing the image and for generating a new image focused at a different distance from the data in the plenoptic format. This command can be executed by the refocusing module 402 . As can be seen on FIGS. 7A and 7B , this can cause the first annotation 61 visible at the first focal length to disappear or become less sharp at the second focal length shown on FIG. 7B , whereas the second annotation 63 only occurs at this second focal length.

The different commands to change the rendered view in steps 105 and 106 can also be issued automatically with respect to the user movement. In one embodiment, user movement is tracked with an inertial measurement unit (IMU) embedded in the plenoptic capture device. By using this module, the rendered view is automatically updated as the user moves. For example, when the user moves to the left, the viewing direction pans slightly to the left. The same principle applies when the user moves forward, where the focus range also moves forward, resulting in sharper objects in the background plane and softer objects in the foreground plane compared to the previously rendered view. The invention is not limited to the use of IMUs to track user movement. Other means such as directly using the plenoptic image content to track user movement can also be used.

In another embodiment, an online plenoptic annotation process is continuously applied to the plenoptic image stream generated by the user's plenoptic capture device on the move. This continuous process allows the user to continuously move or move his plenoptic capture device and have the plenoptic annotations updated in real time. The plenoptic image stream and rendering of the view (step 104 of Fig. 8) must be processed in real-time so that the user perceives the annotations as if they were part of the scene. In this embodiment, the fact that the viewing direction can then be modified without needing to have another plenoptic capture allows to achieve the same effect with a much lower number of plenoptic images that need to be processed from the stream. In fact, if we assume that a single plenoptic capture allows the rendering of views within a certain range of viewing directions, and as long as the user does not move out of this range, then the plenoptic image from the stream does not need to be processed and only step 104 of FIG. is executed again. This opens new possibilities for more computationally efficient real-time tracking by processing new plenoptic image frames asynchronously as the user is approaching the boundaries of the field of view, so that the user is not aware when a new frame should be processed Delay.

An example of a method for annotating an active plenoptic image is illustrated on Figure 11:

Steps 200 , 201 , 202 and 203 in FIG. 20 are similar or equivalent to steps 100 , 101 , 102 and 103 in FIG. 8 .

In step 204 , viewing direction parameters are calculated as a result of the registration process of step 201 .

In step 205, the view is rendered based on the viewing direction calculated in the previous step.

In step 206 an inertial measurement unit (IMU) is used to determine user movement with respect to the time that step 200 has been calculated. A decision is then taken to either go back to step 200 for processing a new plenoptic image, or go directly to step 204 to update the view direction parameter based on the IMU movement evaluation. The amount of movement is used to determine whether previously captured plenoptic data can be used to generate novel views. This typically depends on the field of view of the plenoptic capture device.

Reproduction of plenoptic annotations can account for possible occlusions. The plenoptic annotation may be occluded if the target element to be annotated is hidden from the capture device view by another object located in the input plenoptic image.

In one embodiment, the rendering module utilizes the plenoptic format of the captured data to visually hide annotations behind irrelevant objects. The rendering module knows from the plenoptic reference data the properties of the captured rays that should come from each element of the captured plenoptic image. If the captured ray has different properties than the element's expected ray, it can indicate that the occluding object is in front of the element, and thus the annotation does not have to be displayed for this element.

In a similar manner, if the light rays corresponding to an element in the captured image have a different direction than the corresponding direction in the reference image, this can indicate that the element is at a different depth. The rendering module can use this information to detect occlusions. Additionally, the color information of the light can also be used to determine whether the captured feature is occluded. However, color information is not sufficient since occluded objects may have the same color as the target element.

application

The process of annotating plenoptic images in the same space as annotations and the provision of annotations in a plenoptic format brings new applications for augmented reality.

A first example of application is the use of a plenoptic annotation system in a social context. In fact, plenoptic images of objects/scenes can be captured by users with their plenoptic capture devices. The captured plenoptic images can then be annotated by the user using various annotations, including plenoptic images that were previously captured and used as annotations. Their annotated scene can then be shared to the user's friends using social networking, so that those friends can experience the annotated scene when they capture it with their own plenoptic capture device. The advantage of using the plenoptic annotation process in this case is leveraged by the fact that since the annotation is a plenoptic image, the annotation is already in plenoptic space. Doing the annotation process in the same plenoptic space is therefore more computationally efficient and yields more realistically annotated scenes.

A second example of an application utilizing different information of plenoptic spaces is the use of specially designed plenoptic annotations in the field of architectural design. As described in the previous section of the invention, the plenoptic annotation consists of rays combined with the plenoptic image rays in the online phase. The way this ray is combined is defined in the offline part of the annotation process. This combination enables the rays from the plenoptic image not to be replaced by other rays from the annotations, but instead just change their direction, for example. By defining annotations, which modify not only the wavelength of the rays of the plenoptic image but also eg their direction, it is possible to simulate changes in the material or texture of the captured scene. In this case of architectural design, plenoptic annotations can advantageously be used to simulate, for example, how a particular room or a particular building would look if different materials were applied to the walls. In another embodiment, a simulation of weather conditions can be applied to the captured plenoptic images. Annotations that simulate rain can be applied to the scene. This will produce an annotated image with a rain effect applied to it so that the user can visually see how the scene will look like in the event of rain or other different weather conditions, where due to the plenoptic information, Different light reflections and refractions are properly handled and calculated in a realistic manner.

As another example, treasure hunting is a popular application in traditional 2D augmented reality solutions. It consists of attaching annotations to physical objects and having friends or others search for these annotations (called treasures) by giving clues to them. In other words, when someone approaches a hidden object, he can use his plenoptic capture device to scan surrounding objects to determine if they are associated with annotations. By using plenoptic annotations, treasure hunting becomes more exciting as we are able to limit annotation visibility to a few viewing directions or focal lengths. For example, a user could attach an annotation to a statue and decide to only make this annotation visible if in the future the seeker is placed in front of the statue and therefore he sees the statue from that angle. Similarly, we can use the refocusing properties of plenoptic space to ensure that the seeker is focused on the statue itself and thus only display the annotation in this case. It makes treasure hunting more engaging, because it prevents the user from randomly scanning around to find treasure and instead forces him to actually solve the puzzle.

Another application involves city guides in urban environments. For example, let us consider that a user is in a city he is visiting and looking for tourist spots such as historical monuments, sightseeing spots, statues, museums, local restaurants... . With his augmented reality system, the user certainly doesn't want to have all the information appearing on his screen at once: he'll just be confused by all this stuff visually superimposed on the screen. Instead, plenoptic annotations can be made dependent on user viewpoint and focus. For example, elements of an image captured by a user at a particular viewing angle (or at a particular range of viewing angles) can be displayed with lower importance than elements facing the user. In one embodiment, low-importance annotations can only be displayed on the screen as titles or points (when the user clicks on the title or point, they can be expanded), while more important points of interest are presented more Details may have greater size or emphasis on the image.

The ability to choose the viewing direction from which annotations are not visible is attractive to vehicle drivers who might want, for example, augmented reality images on a navigator display, but do not want to be attached to elements Annotations such as distractions from traffic-irrelevant advertisements, shops, etc. In this case, those distracting annotations may be associated with a range of orientations selected so that they will not be displayed on images captured from the road.

Terms and Definitions

Various operations of the methods described above may be performed by any suitable means capable of performing operations, such as various hardware and/or software component(s), circuits, and/or module(s). In general, any operations described in the application may be performed by corresponding functional means capable of performing the operations. The various means, logic blocks, and modules may comprise various hardware and/or software component(s) and/or module(s), including but not limited to: circuits, application specific integrated circuits ( ASIC), or general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors together with a DSP core, or any other such configuration. A server can be implemented as a single machine, a collection of machines, a virtual server, or a cloud server.

As used herein, the expression "plenoptic data" designates any data that is generated with a plenoptic capture device, or calculated from other types of data, and that describes a light field image of a scene in which not only light The brightness and color of this light and the direction of the image. Due to the loss of this direction of light, 2D or stereoscopic projections rendered from such plenoptic data are not considered plenoptic images.

As used herein, the expression "plenoptic space" may designate a multidimensional space in which a light field, ie a function describing the amount of light in each direction in space, can be described. The plenoptic space can be described by at least two parameters for the position of a ray, two parameters for its orientation and one parameter for its wavelength and (in the case of video) possibly one parameter for time.

As used herein, the term "annotation" encompasses a wide variety of possible elements including, for example, text, still images, video images, logos, sounds, and/or to other elements in the plenoptic space represented by the plenoptic data. More generally, the term annotation covers different ways of altering different parameters of plenoptic spatial rays represented by plenoptic data. Annotations can be dynamic and change their position and/or appearance over time. Additionally, annotations can be user-interactive and react to user actions (eg, move or transform upon user interaction).

As used herein, the term "pixel" may designate a single single-color light bit, or multiple adjacent light-bits for detecting light in different colors. For example, three adjacent light bits for detecting red, green and blue light can form a single pixel.

As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include deriving, calculating, processing, deriving, investigating, looking up (eg, in a table, database, or another data structure), ascertaining, evaluating, and the like. Also, "determining" may encompass receiving (eg, receiving information), accessing (eg, accessing data in memory), and the like. Also, "determining" may include resolving, selecting, selecting, establishing, and the like.

Capturing an image of a scene involves using a digital pinhole camera to measure the brightness of light reaching the camera's image sensor. Capturing plenoptic data may involve using a plenoptic capture device, or may involve generating light field data from a virtual 3D model or other description of the scene and light sources. Retrieving an image may involve capturing the image, or retrieving the image from a different device over a communication link.

The expression "reconstructing a view", such as "reproducing a 2D view from plenoptic data", covers the action of computing or generating an image, such as computing a 2D image or a holographic image from information contained in the plenoptic data.

The steps of a method or algorithm described in connection with this disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. A software module may reside in any form of storage medium known in the art. Some examples of storage media that may be used include: random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, and the like. A software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media. A software module may consist of an executable program, parts or routines or libraries used in a complete program, multiple interconnected programs, an "application" executed by many smartphones, pc or computers, widgets Widgets, Flash applications, sections of HTML code, etc. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. A database may be implemented as any structured collection of data, including a SQL database, a collection of XML documents, a semantic database, or a collection of information available over an IP network, or any other suitable structure.

Accordingly, certain aspects may include a computer program product for performing the operations presented herein. For example, such a computer program product may include a computer-readable medium having stored thereon (and/or encoded) instructions executable by one or more processors to perform the operations described in . For certain aspects, a computer program product may include packaging materials.

It is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims (28)

1. A method comprising the steps of: Retrieving data (100) representing the light field with the plenoptic capture device (4); executing program code for matching the retrieved data with corresponding reference data (101); executing program code for retrieving at least one annotation (61, 63, 64) in a plenoptic format associated with an element of said reference data (102); Program code is executed for generating annotated data in a plenoptic format from said retrieved data and said annotations (103). 2. The method of claim 1, further comprising: select viewing direction (105); rendering a view (107) of data corresponding to said annotation from said viewing direction, Wherein the representation of the annotation (61) depends on the viewing direction. 3. The method of claim 1, further comprising: rendering (104) a first view of data corresponding to the annotation from a first viewing direction; Select Second Viewing Direction (105); rendering a second view (107) of data corresponding to said annotation from said second viewing direction; Wherein the representation of said annotation (61, 61') is changed between said first view and said second view. 4. The method of one of claims 1 to 3, further comprising: associating a first annotation (61) with a first position and a first orientation; associating (64) a second annotation with the first location and with a second direction; renders a view corresponding to the annotated data, select (105) between a first or a second viewing direction; A view containing the first annotation but not the second annotation is rendered if a first viewing direction is selected, or a view containing the second annotation but not the first annotation is rendered if a second viewing direction is selected. 5. The method of one of claims 1 to 4, further comprising: rendering a first view corresponding to the reference data in the plenoptic format and corresponding to the first viewing direction; associating annotations to elements in said first view; rendering a second view corresponding to said reference data in plenoptic format and corresponding to a second viewing direction; associating an annotation to said element in said second view; An annotation for the element is interpolated in an intermediate view between the first viewing direction and the second viewing direction. 6. The method of claim 5, further comprising the step of computing annotations in a plenoptic format from said first view, second view and intervening views. 7. The method of one of claims 1 to 6, further comprising: rendering a first view (104), the first view corresponding to the annotated data and corresponding to a first focal length; ModifyFocalLength(106); rendering a second view (107), said second view corresponding to said annotated data and corresponding to a modified focal length; wherein the representation of said annotation (61) is changed between said first view and said second view. 8. The method of claim 7, further comprising: associating a first annotation (61) with a first position and a first depth; associating a second annotation (63) with said first location and second depth; rendering a first view (104), the first view corresponding to the annotated data; choose between a first focal length or a second focal length (106); Render a second view (107) containing the first annotation (61) but not the second annotation (63) if the first focal length is selected, or render the second annotation but not the second annotation if the second focal length is selected The view of the first annotation. 9. The method of one of claims 1 to 8, at least one of the annotations being a sound attached to a coordinate and associated with a specific direction. 10. The method of one of claims 1 to 9, at least one of the annotations being a video. 11. The method of one of claims 1 to 10, at least one of the annotations acting as a filter for altering the direction of a ray at a particular location in plenoptic space. 12. The method of claim 11, wherein one of said annotations modifies the direction of light rays. 13. The method of claim 12, wherein one of said annotations modifies a property of a texture or surface of an object. 14. The method of one of claims 1 to 13, wherein the annotations are defined by arrays defining the directions of rays or modifications of directions of rays at different points in plenoptic space. 15. The method of one of claims 2 to 14, wherein rendering comprises: determining when an annotation is retrieved for an element of the light field depending on the depth of the element determined from the direction of the light corresponding to the element Occlusion, or Annotation when to occlude elements of the retrieved light field. 16. The method of one of claims 2 to 15, wherein rendering comprises: retrieving an annotation in the plenoptic format and applying this annotation to a plurality of consecutive retrieved light fields in the retrieved light field stream . 17. The method of one of claims 2 to 16, wherein rendering comprises merging the annotated rays with rays corresponding to the retrieved data. 18. An apparatus (4) for capturing and annotating data corresponding to a scene, comprising: A plenoptic capture device (41) for capturing data representing a light field; processor(400); monitor(40); program code for, when executed, causing the processor to retrieve at least one annotation (61, 63, 64), and for rendering on said display (40) a view generated from the captured data and containing said at least one annotation. 19. The device of claim 18, said program code further comprising a refocus module (402) allowing a user to refocus said view and for changing said annotation depending on a selected focal length presentation. 20. The device of one of claims 18 or 19, said program code further comprising a viewpoint selection module (403) allowing a user to change a viewpoint, said viewpoint selection module (403) being used for said rendering and for determining The presentation of the annotation is changed at a selected viewpoint. 21. A device (5) for determining annotations, comprising: processor (51); storage(50); program code for, when executing said degree code, for causing said processor to receive data representing a light field, match said data to a reference data, determine from said memory an in Annotations (61, 63, 64) in plenoptic format and sending said annotations in plenoptic format or an image of the annotations in plenoptic format to a remote device (4). 22. The device of claim 21 , said program code further comprising a module (510) for adding annotations in plenoptic format and linking them with positions and viewing angles in said reference data Associated. 23. The apparatus of claim 22, further comprising a memory storing the annotation as an array of ray directions or modifications of ray directions in different points in plenoptic space. 24. A method for attaching annotations to a reference image in a plenoptic format, comprising: presenting said reference image in a plenoptic format with a viewer (150); select Annotations(151); selecting with said viewer a location (152) for said annotation and one or more directions (153) from which said annotation can be seen; The position and the orientation are associated in memory with the annotation and the reference image in a plenoptic format (154). 25. The method of claim 24, comprising associating multiple annotations with a single location but with multiple different directions. 26. The method of one of claims 24 to 25, further comprising: rendering a first view corresponding to the reference data in the plenoptic format and corresponding to the first viewing direction; associating a first annotation to an element in said first view; rendering a second view corresponding to said reference data in plenoptic format and corresponding to a second viewing direction; A second annotation different from the first annotation is associated to the element in the second view. 27. The method of one of claims 24 to 26, further comprising: rendering a first view corresponding to the reference data in the plenoptic format and corresponding to the first viewing direction; associating annotations to elements in said first view; rendering a second view corresponding to said reference data in plenoptic format and corresponding to a second viewing direction; associating an annotation to said element in said second view; An annotation for the element is interpolated in an intermediate view between the first viewing direction and the second viewing direction. 28. An apparatus for attaching annotations to a reference image in a plenoptic format, comprising: processor; program code for causing said processor to: present said reference image (150) in a plenoptic format with a viewer; allow a user to select an annotation (151) and a location for said annotation (152) and one or more directions (153) from which the annotation can be seen; A memory stores the annotation, the position and the direction.