WO2024260550A1

WO2024260550A1 - Optical data processing using structural non-linearity

Info

Publication number: WO2024260550A1
Application number: PCT/EP2023/066812
Authority: WO
Inventors: Mustafa Yildirim; Niyazi Ulas DINC; Ilker Oguz; Demetri Psaltis; Christophe Moser
Original assignee: Ecole Polytechnique Federale De Lausanne (Epfl)
Priority date: 2023-06-21
Filing date: 2023-06-21
Publication date: 2024-12-26

Abstract

A device (1) and method for processing data comprises at least one input layer (2), at least one modulation layer (3), and at least one output layer (4). The input layer (2) is configured to receive at least one carrier signal (Sc). The modulation layer (3) comprises input data. The modulation layer (3) is configured to modulate the carrier signal (Sc) such, that a modulated carrier signal (Smc) comprising the input data being linearly transformed is generated. The device (1) is configured such, that the modulated carrier signal (Smc) is repeatedly modulated with the input data and linearly transformed, wherein an output signal (So) comprising output data is produced at the output layer (4), and wherein the output data comprises or consists of the input data being nonlinearly processed.

Description

TITLE

OPTICAL DATA PROCESSING USING STRUCTURAL NON-LINEARITY

TECHNICAL FIELD

The present invention relates to a device for processing data according to claim 1 , to a method of manufacturing such a device according to claim 17, and to a method of processing data with such a device according to claim 18.

PRIOR ART

Data processing such as optical computing utilizes light to perform computations and process information for applications such as artificial intelligence (Al) and optical processing for data communication. Optical computing has the potential to be much faster and more energy-efficient than traditional electronic computing. However, developing a fully optical computing technology has proven to be challenging, as it requires the implementation of both linear and nonlinear computations in the optical domain. Nonlinear optical effects occur when light interacts with a material and causes a polarization of the material that is nonlinear with respect to the electric field of the light. While nonlinear interaction using high peak power laser pulses in multimode/single mode fibers and integrated waveguides has been developed for nonlinear information processing, these methods require bulky and costly mode-locked lasers as light sources. On the other hand, low power light sources have been used for linear computations on various optical platforms such as integrated meshes of Mach Zehnder Interferometer, micro-ring resonators, and free space vector matrix multipliers. In order to generate nonlinearities in optical computing systems, high power laser sources must be used through electronics using Optical-Electrical-Optical (OEO) conversions. This hinders the scalability of optical computing hardware, making it difficult to integrate into practical devices, especially for optical computing applications where power consumption is critical. SUMMARY OF THE INVENTION

It is an object of the present invention to provide a device for processing data that is capable of processing the data at high processing speed and in an energy efficient manner.

This object is achieved with a device according to claim 1. That is, a device for processing data is provided, wherein said device comprises at least one input layer, at least one modulation layer, and at least one output layer. The input layer is configured to receive at least one carrier signal. The modulation layer comprises input data. The modulation layer is configured to modulate the carrier signal such, that a modulated carrier signal comprising the input data being linearly transformed is generated. The device is configured such, that the modulated carrier signal is repeatedly modulated with the input data and linearly transformed, wherein an output signal comprising output data is produced at the output layer, and wherein the output data comprises or consists of the input data being nonlinearly processed.

That is, the device is configured such that a nonlinearity is generated by repetitive linear transformations of the input data. In other words, the device is configured such, that the input data is repeatedly linearly transformed. As a result, an output signal comprising output data is generated, wherein said output data is nonlinear with respect to the input data and is generated as a result of the repetition of the input data in the modulation layer.

The expression "being linearly transformed" preferably means that the carrier signal is modulated by the modulation layer while maintaining linearity. In more mathematical terms, the linear transformation can be understood as a mapping or function between two vector spaces that preserves certain algebraic properties. More specifically, a linear transformation T between vector spaces V and W satisfies two fundamental properties:

1) Preservation of Addition: For any two vectors u and v in V, the transformation of their sum T(u + v) is equal to the sum of their individual transformations, T(u) + T(v).

2) Preservation of Scalar Multiplication: For any vector u in V and scalar c, the transformation of the scalar multiple c * u is equal to the scalar multiple of the transformation, c * T(u). For example, in terms of a simple linear transformation, such as y=ax+b, one can ascribe x as the input and y as the output. If this modulation is repeated, one obtains y=a(ax+b)+b=a²x+2b, wherein this equation is still a linear equation considering x as the input and y as the output. Applied to the present invention, x can be understood as the carrier signal and a as the input data. Then the relation between y and a becomes nonlinear (since there is the square of a). Hence, in the event that the device repeatedly performs a y=ax+b kind of transform, these repetitions establish a nonlinear transform between a and y-

Hence, the interaction of the carrier signal and the modulation layer that comprises input data is governed by a linear transformation. These interactions preferably vary depending on the physical medium. For instance, and as will be explained in greater detail below, if the carrier signal is in the optical domain, it can interact with a modulation layer being provided by a modulator such as an opto-electronic modulator device such as (but not limited to) a liquid crystal Spatial Light Modulator (SLM). Input data can be feed to the liquid crystal SLM as an electrical signal that applies different voltage values to different cells containing liquid crystal yielding a proportional change in the crystal orientation. This change results in an optical path change in different cells containing liquid crystal. The carrier signal accumulates different phase delays because of different optical paths while passing through these cells. This process transfers the input data onto the carrier signal where the interaction is governed by a linear transformation.

The expression "modulation layer comprising input data" preferably means that the modulation layer is configured to modulate the carrier signal or the modulated carrier signal according to a scheme or pattern being associated with the input data. For instance, and as will be explained in greater detail further below, the modulation layer can be (part of) a spatial light modulator. A spatial light modulator is an electrically programmable device that modulates light according to a fixed spatial scheme or pattern. In this case, the modulation layer comprising input data means that the spatial light modulator is electrically controlled by an input signal, i.e. a control signal, being based on or associated with the input data. As a consequence, a modulated carrier signal comprising the input data being linearly transformed is generated. That is, it is preferred that the input data is presented to the modulation layer in the form of an input signal, preferably an electrical input signal and/or an optical input signal, and wherein the input signal is configured to control a modulation scheme or modulation pattern of the carrier signal and/or the modulated carrier signal, respectively. In other words, the modulation layer is preferably controlled by a signal that is based on or associated to the input data.

The input data preferably is information or observations on which a model performs inference for classification and/or regression, see further below. The processing of the data by the device according to the invention preferably corresponds to the performance of a computational task. The device for processing data according to the invention preferably is an optical device for processing optical data, for instance for performing optical computing. In this case it is preferred that the signals, in particular the carrier signal, the modulated carrier signal and the output signal, are optical signals. It is furthermore preferred that the device comprises at least one light source or is in connection with at least one light source being configured to emit at least one light beam, and wherein said light beam provides the carrier signal. However, the device according to the invention can likewise be an electrical device or a mechanical device such as an analogue circuit where the carrier signal being an electrical signal being generated by a voltage source or a current source. Likewise, the device can be a mechanical device such as an acoustic wave reservoir where the carrier signal being a surface acoustic wave being generated by an interdigital transducer.

The modulation layer can be configured to repeatedly modulate the modulated carrier signal and preferably upon receipt of a feedback signal, whereby the input data is repeatedly linearly transformed.

For instance, the device can comprise a single modulation layer, and wherein said single modulation layer is configured to modulate the carrier signal and thus the modulated carrier signal repeatedly. Thereby, the input data is repeatedly linearly transformed and an output signal is produced at the output layer that comprises or consists of the input data being nonlinearly processed.

However, it is likewise preferred that the device comprises two or more modulation layers. The device can comprise at least two modulation layers being arranged in a cascading manner, whereby the modulated carrier signal is repeatedly modulated in a cascading manner and the input data is repeatedly linearly transformed. That is, the device can comprise two or more modulation layers, and wherein the modulation layers modulate the carrier signal and thus the modulated carrier signal in a cascading manner, i.e. the carrier signal is modulated at least once by each modulation layer and, in the event of at least two modulation layers, at least twice. This multiple modulation of the input data produces output data being nonlinear with respect to the input data.

Hence, whereas in the former case the device can be seen as a device comprising or consisting of a single linear system that is used repeatedly and preferably while further using a feedback signal such as an optical feedback signal and/or an electronic feedback signal (see further below), in the latter case the device can be seen as comprising or consisting of multiple linear systems using at least partially the same input data so that the multiple linear transformations result in an overall nonlinear processing.

It should be noted that the device can be a combination of the former case and the latter case, i.e. the device can comprise at least two modulation layers, wherein each modulation layer repeatedly modulates the carrier signal and thus the modulated carrier signal and wherein the carrier signal and thus the modulated carrier signal is additionally also modulated by the modulation layers in a cascading manner.

Any statements made herein regarding a single modulation layer preferably likewise apply to two or more modulation layers and vice versa.

Said linear system or systems can be an optical, an electrical or mechanical system, for instance. That is, the present invention is based on the insight that a nonlinear output signal can be generated with a linear optical device, a linear electrical device, or a linear mechanical device.

Examples of a linear optical system or a liner optical device, respectively, are a holographic display system, a fiber optic communication system or a beam splitter system, for example. A conceivable holographic display system could comprise or consist of a coherent light source, a Spatial Light Modulator comprising or consisting of a coherent light source, a modulator such as a Spatial Light Modulator (SLM), lenses for imaging and a camera. In a holographic display system, an SLM could be used to modulate the phase or intensity of light waves to create a holographic image. The SLM acts as a dynamic optical element that can modify the wavefront of the incident light. By controlling the modulation of the SLM, it is possible to display different holographic images. The SLM's modulation corresponds to the linear manipulation of the phase or intensity of the incident light, preserving the linearity of the overall optical system. It's important to note that while the SLM itself can be a key component in this example of a linear optical system, the entire holographic display system, including other optical elements like lenses and laser sources, can be used to collectively form the linear optical system as a whole. In a conceivable fiber optic communication system, the transmission of data can occur through optical fibers. Such a system involves linear components such as optical fibers, connectors, splitters, and amplifiers that transmit and manipulate light signals while preserving linearity. A device according to the invention in the form of a beam splitter system can be an optical device that divides an incoming beam of light into two or more beams. The behavior of a beam splitter can be modeled as a linear optical system, as it operates based on the principles of reflection, transmission, and interference. It should be noted that these are examples of devices according to the invention in the form of linear optical systems. Other linear optical systems than the just described ones are likewise conceivable. Examples of a device according to the invention in the form of a linear electrical system are an RC Circuit or a Transmission Line. For instance, the device according to the invention can be an RC circuit comprising a resistor (R) and capacitor (C) connected in series or parallel. The voltage-current relationship in the RC circuit follows linear differential equations, allowing for linear transformation. A conceivable device in the form of a transmission line could comprise a coaxial cable or a microstrip line which can transmit electrical signals from one point to another while maintaining linearity, with the behavior described by linear wave equations. Also in this case it is noted that these are examples of conceivable devices only. Many other linear electrical systems are likewise conceivable. Examples of a device according to the invention in the form of a linear mechanical system are a Mass-Spring System or a Linear Vibrating System. For instance, the mass-spring system could comprise a mass connected to a spring, wherein the relationship between the displacement of the mass and the applied force follows Hooke's law, making the system linear. A conceivable vibrating system is preferably configured to exhibit a linear behavior, such as a simple pendulum or a mass attached to a linear oscillator, wherein the motion of these systems can be described by linear differential equations. Also in this case it is noted that many other examples are likewise conceivable.

In each of these examples, the system refers to the overall arrangement or configuration of components that collectively exhibit linear behaviour and thus provide a device according to the invention. The individual devices or components within the system, such as lenses, resistors, or springs, can also be considered linear devices or elements within the broader system.

The input layer and the modulation layer are preferably arranged such, that the carrier signal is allowed to propagate in at least one propagation step between the input layer and the modulation layer, whereby different dimensions of input data are coupled.

Additionally or alternatively, the modulation layer and the output layer are preferably arranged such, that the modulated carrier signal is allowed to propagate in at least one propagation step between the modulation layer and the output layer, whereby different dimensions of input data are coupled.

For example, different dimensions of input data being coupled can be different channels of the input data being coupled, for instance in the event of the input data being an image comprising a certain number or pixels. In this example, each pixel can contain partial information about the whole image. T o make an inference for a certain task, one might need to consider a specific subgroup of pixels collectively. Hence, a computing device should be able to couple the information on different pixels with each other. The propagation step can provide this coupling. Information from a subgroup of pixels for example can diffract (propagate forward, while this propagation disperses the light in space) to a plane that is a certain distance away from the display. On that plane, a specific location receives light from different pixels of the image. Hence, data or information on different pixels are combined or coupled to each other.

Within the context of the present invention, "pixel" preferably refers to discrete units of the modulation layer(s) and the output layer such as pixel of the camera for the detection of the output signal and pixel of the SLM (for modulation layers) for injecting input data and preferably also trainable parameters (see below) in the system or device, respectively.

The device is preferably configured such that the carrier signal and/or the modulated carrier signal is propagating via a free space diffraction. Additionally or alternatively, the device can be configured such that the carrier signal and/or the modulated carrier signal are allowed to linearly or nonlinearly propagate through a medium. Additionally or alternatively, the device can be configured such that the carrier signal and/or the modulated carrier signal are allowed to linearly or nonlinearly propagate through one or more preferably coupled waveguides. Additionally or alternatively, the device can be configured such that the carrier signal and/or the modulated carrier signal are allowed to linearly or nonlinearly propagate through linear optical elements or linear electrical elements or linear mechanical elements. For example, for an electrical implementation in a device being a linear electrical device, the propagation step can be realized by a group of resistors, capacitors connected to each other in a mesh. In the event of the device being a linear mechanical device, it can be strings and masses connected to each other or a tank filled with liquid.

In the event of a free space diffraction, the diffraction can couple the data or information that were from different pixels (of an image, for example) or different elements (of a vector/matrix). How diffraction couples the information can be modelled by a diffraction matrix, which can also be called a propagation matrix. Elements of said matrix can be called propagation parameters. These parameters are preferably fixed and calculated by a physical distances, pixel sizes, wavelength of light, refractive index of the medium where light travels, etc. For a waveguide mesh that consists of MZIs, see further below, there is preferably no free-space diffraction as the light can be guided by the waveguides. In the event of input data in the form of pixels of an image being introduced to different waveguides (each waveguide receives light whose phase or intensity is proportional to corresponding pixel value of an image), one can set how much of light would be coupled to another waveguide along the mesh via modulators on MZIs. Hence for waveguide-MZI mesh, the propagation parameters are not fixed but rather programmable. Hence, said propagation parameters preferably are examples of trainable parameters, see further below.

Moreover, the propagation step is preferably not only configured to couple the data or information from different pixels within a modulation step, but preferably also enables the carrier signal to reach to a successive modulation layer. In the event that the propagation step disperses the carrier signal (e.g. diffraction in the case of optics), it also enables the coupling of information or data encoded onto the carries signal in the previous modulation layer to spatially different positions in the successive modulation layer. Hence, the propagation step preferably enables an inter-layer and intra-layer coupling of information or data, respectively.

Two or more modulation layers are preferably arranged so as to extend in at least one common plane. At least one reflective element is preferably facing towards said modulation layers such, that the carrier signal and/or the modulated carrier signal is reflected at said reflective element and allowed to propagate with respect to said modulation layers.

That is, two or more modulation layers can be arranged within a common plane. At least one reflective element is preferably facing towards said modulation layers, i.e. being arranged above and/or below said modulation layers with respect to a spatial direction. The reflective element can be configured at least partially reflective or entirely reflective for the carrier signal and/or the modulated carrier signal.

In this way the carrier signal and/or the modulated carrier signal is at least partially or entirely reflected at the reflective element and can propagate with respect to the modulation layers.

In other words, at least two modulation layers can be said to be unfolded, i.e. arranged side by side, where a reflective surface of the reflective element facing across the modulation layers enables the repeated modulation of the carrier signal and the modulated carrier signal, respectively.

The reflective surface of the reflective element can be flat or curved to modify a free space propagation such as providing a focusing effect for the carrier signal and/or the modulated carrier signal.

A propagation direction of the carrier signal and/or the modulated carrier signal thus preferably extends along the common plane of the modulation layers.

In the event of two or more modulation layers said two or more modulation layers can be arranged at a spatial distance from one another. However, it is likewise conceivable that said two or more modulation layers are arranged next to one another.

The modulation layer is preferably at least partially transmissive for the carrier signal and/or the modulated carrier signal. Additionally or alternatively, the modulation layer is preferably at least partially reflective for the carrier signal and/or the modulated carrier signal.

That is, the modulation layer or modulation layers can be at least partially transmissive for the carrier signal and/or the modulated carrier signal. In this case it is preferred that the device comprises at least two reflective elements facing towards said modulation layer(s), and wherein the modulation layer(s) are arranged between said at least two reflective elements. In other words, said at least two reflective elements are preferably arranged opposite each other with respect to the modulation layer(s). Again in other words, the modulation layer(s) are preferably arranged between the reflective elements. In this case, the carrier signal and/or the modulated carrier signal can propagate with respect to the modulation layer(s) by being reflected at the reflective elements. The carrier signal and/or the modulated carrier signal being reflected at the reflective elements preferably follows a zig-zag path.

However, it is likewise conceivable that the modulation layer(s) are at least partially reflective for the carrier signal and/or the modulated carrier signal. In this case, the carrier signal and/or the modulated carrier signal can propagate with respect to the modulation layer(s) by being reflected at the reflective element and the modulation layer(s). The carrier signal and/or the modulated carrier signal being reflected at the reflective element and the modulation layer(s) preferably follows a zig-zag path as well.

The carrier signal can be an optical signal, preferably laser light, LED light or ambient light.

The modulation layer can comprise or consist of at least one spatial light modulator (SLM) and/or electro-optical modulator and/or thermal modulator and/or Mach-Zehnder Interferometer (MZI) unit and/or nonlinear optical material. Additionally or alternatively, the modulation layer can be configured to modulate an intensity and/or a phase and/or an amplitude and/or a frequency and/or a polarization of the carrier signal and/or of the modulated carrier signal. For instance, the modulation layer can be provided by a spatial light modulator such as a Liquid Crystal Display (LCD), a Digital Micromirror Device (DMD), a photonic crystal spatial waveform modulator or a grating light valve. Other modulation layers are likewise conceivable. For instance, the modulation layer can be provided by a nonlinear optical material such as a Phase Change Material (PCM), an electro-optical modulator, a thermal modulator, or a Mach-Zehnder Interferometer unit, etc. Such modulators are well-known in the art. A Mach-Zehnder Interferometer (MZI) unit preferably comprises at least one input, at least one modulation layer in the form of modulators (electro-optical, thermal, etc.), at least one coupler region, and at least one output. The input of a MZI unit is preferably configured to receive the carrier signal, the modulator of a MZI unit is preferably configured to modulate in particular complexly modulate (e.g. in amplitude and phase) the received carrier signal, the coupler region of a MZI unit is preferably configured to perform an arbitrary linear transform between the MZI input and the MZI output in the complex domain, and the output of a MZI unit is preferably configured to output the thus modulated carrier signal. That is, the modulation layer preferably comprises or consists of a modulator as it is well-known in the art and/or commercially available.

To this end it is particularly preferred that a surface of the modulator provides the modulation layer. For instance, a surface of a spatial light modulator can provide the modulation layer, or the modulation layer can be provided by a slab of a phase change material, etc.

In the event that the device comprises two or more modulation layers said two or more modulation layers can be the same. For instance, two or more modulation layers can be provided by (the surfaces of) two or more spatial light modulators. It is however likewise conceivable that said two or more modulation layers are different from one another. For instance, one or more modulation layers can be provided by (the surface of) a spatial light modulator and one or more modulation layers can be provided by (the surface of) a slab of a phase change material. That is, the modulation layer can be configured to electrically modulate the carrier signal and/or the modulated carrier signal. The modulation layer can be configured to optically modulate the carrier signal and/or the modulated carrier signal.

It is even conceivable that the modulation layer is configured to optically modulate the carrier signal and/or the modulated carrier signal in addition to an electrical modulation. For instance, the modulation layer can comprise or consist of a Phase Change Material that is configured to optically modulate a carrier signal and modulated carrier signal in the form of light in addition to an electrical modulation. To this end the device can comprise two light sources, wherein the light beam of one of the light sources can be used as an inference beam (carrier signal) and the light beam of the other light source can be used as a data writing beam for writing, i.e. combining, the input data on the Phase Change Material. As will be explained in greater detail below, said data writing is preferably related to one or more trainable parameters. For example, the PCM can be the medium of a modulation layer and thus preferably comprises input data and trainable parameters. For simplicity, one can think of the input data as being an image comprising or consisting of pixels and the trainable parameters can be for example multiplicative factors, multiplying the value in each pixel. Then the multiplied (or combined with trainable parameters) input data has to be presented on PCM. One way to do this is optically. When a light is focused on a specific region on PCM, the refractive index of this region is changed because of the alteration of the crystalline structure of PCM. This yields a refractive index change (for simplicity, the change in absorption is ignored). Hence the input data is translated into refractive index variance on PCM. When the carrier signal passes through the PCM, it accumulates different phase delays depending on the refractive index of the local regions. Hence the input data is encoded on the carrier signal. To this end the carrier signal and the writing beam are preferably different optical beams. Moreover, the writing beam can be seen as an equivalent to the electrical input to an opto-electronic spatial light modulator, see above. Hence, the phase change of the Phase Change Material can modulate both the absorption and the refractive index of the modulation layer in the form of the Phase Change Material, yielding a modulation layer being configured for complex modulation. Alternatively, a modulation layer in the form of a Phase Change Material can be used that relies on a refractive index change, yielding different reflection coefficients in reflection because of Fresnel reflection laws. In any case, the data writing beam can modulate the Phase Change Material either by scanning a focal spot or by displaying a target modulation such as displaying a 2D light pattern from a 2D display device. Here, target modulation means that the input data that is combined with trainable parameters. The device preferably comprises a mesh of two or more MZI units, wherein two or more MZI units of the mesh of MZI units are grouped together into one or more groups of MZI units. In each group of MZI units, the modulation layers of the MZI units comprise the input data such that the modulated carrier signal is modulated in a cascading manner by each group of MZI units having input data modulating the carrier signal and/or the modulated carrier signal. That is, groups of MZI units are conceivable, wherein one group acts as a modulation layer, and wherein it preferably presents the input data combined with the trainable parameters. That is, the device according to the invention allows a nonlinear processing of the input data by introducing the input data optionally modulated by trainable parameters (see further below) into a modulation layer in the form of an MZI modulator as complex transmittance. Recall that in the free space based implementation using 2D spatial light modulators, propagation of light with diffraction provided coupling of different dimensions of input data. Here, in the MZI mesh, such coupling can be achieved by programming MZI units where the parameters that program MZI units are trainable parameters. The modulation layer(s) or modulator(s), respectively, are preferably provided in accordance with a desired acquisition method of the output signal and/or in accordance with a further processing of the output signal.

The device preferably comprises at least one detection device being configured to detect the output signal. Said detection device can comprise or consist of the output layer. Additionally or alternatively, the detection device can comprise or consist of at least one optical detector and/or can be configured for detecting an electric field and/or for holographic recording. That is, the output layer is preferably provided by a detection device. Said detection device is preferably configured to detect the output signal. The detection device preferably comprises or consists of at least one optical detector such as a complementary metal-oxide-semiconductor (CMOS) or a charge-couple device (CCD). It is likewise conceivable that the detection device corresponds to a photodetector array or a camera, respectively. Furthermore, the detection device can be configured for detecting an electric field or for performing a holographic recording, respectively. For example, the output signal can be detected by a camera, and wherein an intensity distribution or a complex electric field distribution is detected in case of a holographic recording, such as (but not limited to) off-axis digital holographic interferometry. In case of a holographic recording, it is preferred that the device is configured to detect or record an interference pattern on the detection device such as a camera by combining the carrier signal with an unmodulated separate reference signal. This enables to retrieve a complex electric field instead of just performing an intensity detection. That is, the device can further comprise at least one reference optical source for emitting at least one reference optical beam. In any case it is preferred that the detection device is well-known in the art and/or commercially available.

The input data is preferably combined with at least one trainable parameter. The device is preferably configured to determine at least one feedback signal, for example based on a comparison of the output signal and a target output signal. However, it is likewise conceivable that the device is configured to train the trainable parameter without a feedback signal being determined from a comparison of the output signal and a target output signal. Instead, the trainable parameters could be trained on a computer by simulating the light propagation on the device. To this end the device is particularly preferably configured to determine at least one feedback signal using the output signal to minimize a cost function as it is known in the art. In any case it is preferred that the device is preferably further configured to transmit the feedback signal to the modulation layer in order to train the trainable parameter based on the feedback signal.

As mentioned earlier, the input data can be combined with the modulation layer via at least one input signal or control signal being sent to the modulation layer. Likewise, the modulation layer can be combined with the input data and the trainable parameter. Upon receipt of the input data being combined with the parameter and/or upon receipt of the feedback signal the modulation layer preferably changes its modulation scheme or modulation pattern accordingly. For example, in the event of the modulation layer being realized by an optically configurable material, a complex transmittance of the optically reconfigurable material can be changed. In the event of the modulation layer being a spatial light modulator, being realized by an optically configurable material, a reflectance of the optically reconfigurable material can be changed.

As mentioned earlier, the input data is preferably combined with the modulation layer in the form of an input signal, and wherein said input signal preferably is an optical signal and/or an electronic signal. The carrier signal preferably is an optical signal and/or an electrical signal as well. Consequently, the modulated carrier signal and the output signal preferably are in each case an optical signal and/or an electrical signal as well. As such, the target output signal and the feedback signal are in each case preferably an optical signal and/or an electrical signal as well.

To this end it is conceivable that the detection device is configured to generate the at least one feedback signal, for instance based on a comparison of the detected output signal and the target output signal, and wherein the detection device is further configured to transmit the feedback signal to the modulation layer so as to train the trainable parameter. Additionally or alternatively, the feedback signal can likewise be generated numerically. In fact, the device can be configured to implement an error backpropagation, wherein at least one error signal being generated in response to the output signal is minimized. Said error signal preferably constitutes the feedback signal mentioned earlier. To this end, the device can be configured to determine discrepancies between the detected output signal and the target output signal, and wherein said discrepancies are fed back to the modulation layer(s) in order to improve the trainable parameters so as to minimize the discrepancies. In addition, a digital twin of the device can be implemented to find optimum trainable parameters as it is known in the art. The target output signal can correspond to a simulated signal or an arbitrarily chosen signal. In this case, the device could be configured to train the trainable parameter such that a discrepancy between the simulated signal and the detected output signal, e.g. an experimental signal, is minimized.

The trainable parameter is preferably trained to minimize the error signal.

The trainable parameter is preferably trained to maximize a data processing performance of the device such as classification accuracy for a classification task or maximize the matching rate of a regression tasks by minimizing the difference between predicted values and ground-truth values.

In other words, the trainable parameter is preferably optimized, for instance by having a configuration realized in a physical differentiable simulation model such as the digital twin, where error backpropagation can be employed.

The device can furthermore be configured to calibrate the trainable parameter, for instance by determining experimental imperfections of the device, and wherein these imperfections can be included in the digital twin. Alternatively, a machine learning approach can be employed to alter a forward pass of the digital twin to collectively account for experimental imperfections by having a loss function that compares an output signal of the forward pass of the digital twin with the output signal numerically evaluated for error backpropagation, all implemented digitally.

The input data being combined with the trainable parameter preferably means that the modulation layer is combined with the input data and the trainable parameter. In particular, the modulation layer preferably comprises the input data being transformed by the trainable parameter. To this end any transform known in the art is applicable.

The modulation layer can comprise one or more trainable parameters. If two or more modulation layers are present, said two or more modulation layers can comprise the same trainable parameters or trainable parameters being from one another.

The trainable parameter preferably is at least one of a capacitance, an inductance, or a resistance in the event that the device comprises or consists of an electrical system. In the event of an optical system, the trainable parameters preferably is at least one of phase of liquid crystal, state of digital micromirror device, temperature of thermal MZI modulator, crystalline state of phase change material.

In order to combine or otherwise implement electrical and optical signals, data or parameters, it is preferred that the device comprises at least one electronic-to-optical interface and/or at least one optical-to- electronic interface. For instance, in the event of the input data and/or the trainable parameter being electrical data and parameters, respectively, said input data and/or trainable parameter can be combined on the modulation layer through at least one electronic-to-optical interface. Such interfaces are well-known in the art.

The output signal being detected by the detection device preferably constitutes a final output signal comprising or consisting of finally processed data. Alternatively, the device preferably further comprises at least one layer of a digital network and is configured to propagate the output signal being detected by the detection device at least once through the layer of the digital network, whereby at least one final output signal comprising or consisting of finally processed data is generated. That is, the device can be configured to generate final output data comprising or consisting of the processed data while directly using the output signal being detected by the detection device. For example, in the event of the signals such as the carrier signal, the modulated carrier signal and the output signal being optical signals, a data processing such as the performance of a computational task can be realized all optically, wherein the output signal being detected by the detection device can be used directly. However, it is likewise conceivable that the device comprises at least one layer of a digital network, i.e. that the device is used in combination with a digital network, and wherein the output signal being detected by the detection device is fed at least once to said digital network. In this case, it is preferred that final output data comprising or consisting of the processed data is generated only after the output signal has been propagated through the digital network. That is, in both cases, the final output signal comprising the final output data, i.e. the processed data, is preferably detected by the detection device. In the former case, the final output signal corresponds to the output signal without that the output signal has been propagated through the digital network. In the latter case, the final output signal corresponds to the output signal that has been propagated at least once through the layer of the digital network. The digital network preferably corresponds to a neural network. That is, the layer of the digital network preferably is a layer of a neural network.

The device can comprise multi-layer networks and can be configured to propagate the output signal at least once through the multi-layer networks. That is, the device is preferably configured to process data, in particular to perform computational tasks, via multi-layer networks. Again in other words, the device can be implemented with multi-layer networks. As such, the device allows an implementation, e.g. an optical implementation, of networks such as neural networks for any application where neural networks are used.

The device is preferably configured to produce output data comprising or consisting of a reduced representation of the input data via optical nonlinear processing. Said reduced representation of the nonlinearly processed input data preferably corresponds to nonlinearly processed input data of less size. It is furthermore preferred that said reduced representation is produced on the detection device, in particular on a digital camera. It is furthermore preferred that said output data comprising or consisting of the reduced representation of the nonlinearly processed input data is further processed by at least one layer of a digital network such as a neural network. That is, the device is preferably configured as a frontend nonlinear processing device.

The input layer, the modulation layer and the output layer are preferably monolithically integrated. For instance, the device can comprise or consist of a chip that comprises a light source, modulation layers and the detection device such as a camera where in the third dimension a transparent medium is present for propagation between modulation layers.

In another aspect, a method of manufacturing a device for processing data, preferably a device as described above, is provided. The method comprises the steps of i) providing at least one input layer, ii) providing at least one modulation layer, and iii) providing at least one output layer. The input layer is configured to receive at least one carrier signal. The modulation layer comprises input data. The modulation layer is configured to modulate the carrier signal such, that a modulated carrier signal comprising the input data being linearly transformed is generated. The device is configured such, that the modulated carrier signal is repeatedly modulated with the input data, wherein an output signal comprising output data is produced at the output layer, and wherein the output data comprises or consists of the input data being nonlinearly processed.

Any statements regarding the device per se preferably likewise apply to the method of manufacturing the device and vice versa.

In another aspect, a method of processing data is provided, the method comprising the steps of i) providing at least one device for processing data as described above, and ii) providing at least one carrier signal. The input layer receives the carrier signal. The modulation layer modulates the carrier signal such, that a modulated carrier signal comprising the input data being linearly transformed is generated. The device repeatedly modulates and linearly transforms the carrier signal, wherein an output signal comprising output data is produced at the output layer, and wherein the output data comprises or consists of the input data being nonlinearly processed.

Any statements regarding the method of processing data preferably likewise apply to the device per se and to the method of manufacturing the device and vice versa.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the invention are described in the following with reference to the drawings, which are for the purpose of illustrating the present preferred embodiments of the invention and not for the purpose of limiting the same. In the drawings,

Fig. 1 shows a conceptual schematic of a device according to the invention comprising an input layer, several modulation layers and an output layer;

Fig. 2 shows another conceptual schematic of a device according to figure 1 , wherein each layer comprises two pixels;

Fig. 3 shows a schematics of a device according to the invention comprising modulation layers and a reflective element, which enables to unfold the modulation layers on a single plane;

Fig. 4 shows a schematics of a calculation loop of trainable parameters in the device according to the invention with a digital twin approach. A) Digital twin implementation for error backpropagation. B) schematics of experimental setup where the trainable parameters are tested and discrepancies between simulation and experiments are fed back to the digital twin to match the results; Fig. 5 shows an experimental sample result of a device according to the invention

(panel A) and a corresponding schematics (panel B);

Fig. 6 shows photographs of a device according to the invention;

Fig. 7 shows another schematics of a device according to the invention comprising modulation layers and two reflective elements, which enable to unfold the modulation layers on a single plane;

Fig. 8 shows another schematics of a device according to the invention comprising modulation layers and a reflective element, which enables to unfold the modulation layers on a single plane;

Fig. 9 shows another schematics of a device according to the invention being monolithically implemented;

Fig. 10 shows another schematics of a device according to the invention comprising modulation layers facing across each other and a transparent material in between for a monolithic configuration or free space;

Fig. 11 shows a schematics of a compact monolithic implementation of a device according to the invention;

Fig. 12 shows a schematics of a compact monolithic implementation of another device according to the invention;

Fig. 13 shows a schematics of a device according to the invention comprising modulation layers in the form of a Mach-Zehnder Interferometer (MZI) mesh;

Fig. 14 shows a schematics of an implementation of a generic data processing framework by means of a device according to the invention.

DESCRIPTION OF PREFERRED EMBODIMENTS

Various aspects of the present invention shall now be illustrated with respect to the figures.

In particular, the invention presents a novel configurable data processing framework, in particular computing framework, that can process input data or information, respectively, with high speed and energy efficiency. By utilizing different data injection, detection and system modification techniques, the devices 1 according to the invention perform different data processing techniques with high accuracy. The core of the technique, which is the synthesis of a nonlinear response with a linear system such as a linear optical system, is proven herein below by following the matrix formalism of light diffraction. Nevertheless, the same technique is valid for any linear system such as electrical or mechanical. The general conceptual schematic of the architecture comprises an input layer 2, one or more modulation layers 3, 3a, ... , and an output layer 4 that can be separated by propagation steps P1 , P2, ... , see Figure 1. The propagation step is a key element for coupling the input data or information (e.g. pixels) between the different layers 2, 3, 3a, .., 4 (see Figure 1). There is a multitude of realizations of the propagation step including but not limited to free space diffraction, linear or nonlinear propagation through a medium, and linear or nonlinear propagation through multiple coupled integrated 2-Dimensional (2D) or 3-Dimensional (3D), single or multimode waveguides and linear electrical or mechanical systems.

That is, the present invention presents devices 1 and methods to control linear transformations by repetitive modulation of a signal Sc, Smc using various techniques to synthesize a combination of linear and nonlinear processing that implements the processing of data such as, for instance, machine learning tasks. As embodiments, reconfigurable optical processors are presented that combine optimization algorithms, information acquisition, and architecture for repetitive modulation with various features of light (see Figures 3 to 13). These controlling and acquisition techniques for optical computing perform as optimized solutions for different machine learning tasks. A generalized embodiment for other domains such as electrical, mechanical, or hybrid is also presented (see Figure 14).

Figure 2 schematically depicts the input layer 2, modulation layers 3, 3a, ... and an output layer 4 of a device 1 according to the invention. This is a simplified schematic where each layer 2, 3, 3a, ... , 4 contains two pixels i1 , i2, t1 , t2, o1 , o2 for simplicity without losing generality for employing arbitrarily many pixels in each dimension available. Examples for the propagation include but are not limited to free space diffraction, linear or nonlinear propagation through a medium, and linear or nonlinear propagation through multiple coupled waveguides.

Herein below an analysis of a device 1 in the form of a simplified system, in particular in the form of an optical system is presented, where each layer 2, 3, 3a, ... , 4 consists of two pixels i1 , i2, t1 , t2, o1 , o2 (see Figure 2) and a propagation step P1 , P2, ... whose system response is expressed by a matrix with linear coefficients. We provide the investigation for Modulation layer number N=1 , N=2, and N=3 for ease of explanation and the conclusions are valid for arbitrary N by inductive reasoning. The system response for Modulation layer N=1 is the following: o = P2 x T x Pl x i

Where o is the output vector (Electric field), i is the input vector (Electric field), P1 is the propagation matrix 1, P2 is the propagation matrix 2, and T is the modulation matrix. For P1, we assume zero propagation for simplicity without losing generality and for P2 we use an arbitrary matrix. For a two-pixel per layer system we simply have the following:

Following similar steps for Modulation layer N=2 (assuming P3=P2 without loss of generality). Note that the data in layer N=2 is the same as layer N=1.

Hence, we reach a nonlinear relationship between the output field o_1: o₂ and the modulation layer comprising the input data t , t₂ . The essence of the invention is that this nonlinear relationship is between the input data introduced in the modulation layer 3, 3a, ... as complex transmittance and the electric field at the output layer 4. When intensity detection is employed as the acquisition method, the obtained output signal will be the absolute square of the field, providing a 4th order polynomial expansion of the input data inserted in the modulation layers 3, 3a for the specific case of 2 layers (N=2). Clearly, when input data is introduced in the modulation layers 3, 3a, ... , we obtain a nonlinear processing of the input data at the output layer 4 either by detecting the field with a detection device 7 (by a holographic recording) and/or the intensity (by a detection device in the form of a simple detector, which can be CMOS, CCD, etc.). We note that, recently, nonlinear response with linear scatterings within an integrating sphere is observed and explained by Born series in literature [Eliezer, Y., Ruhrmair, II., Wisiol, N., Bittner, S., & Cao, H. (2022). Exploiting a multiple-scattering system to implement nonlinearities, is referred to as structural nonlinearity [arXiv preprint arXiv:2208.08906.]. The input field i can be a programming parameter to change the effective transform. For simplicity, we will continue with a plane wave input without loss of generality:

In one embodiment, the modulation layer 3, 3a, ... is implemented with a Spatial Light Modulator (SLM). Conventional SLMs employ Liquid Crystal Displays (LCD) that modulate the phase of the light. For a phase-only modulation, we have the following relation for the modulation terms: ti = e^¹ where j= V-l.

In this case, the data becomes c . Then we have:

In the above expression, there is no polynomial expansion of the data at the output electric field. However, when we detect the intensity, we obtain:

In the above expression, the DC term refers to the grouping of the terms that do not depend on the SLM phase pattern ;.The constant terms C_t represent the electric field amplitude resulting from light propagation between layers. 0; is the additional phase bias of the constants arriving from propagation matrix. The elements of the propagation matrix are complex valued. Note that the intensity detection yields cosine terms. The integer multiplier (2) in the second cosine term comes from the fact that there are two modulation layers. Note that this term has a polynomial expansion: cos( 20) =2COS²( 0) - 1

Hence, intensity detection provides the nonlinearity (cosine) and where the multiple modulation layers 3, 3a, ... provide the polynomial expansion of the cosine term. Similarly, when adding a third modulation layer N=3, we have:

Noting that: cos( 30) =4cos³( 0) — 3cos( 0)

By induction, it is obvious that the polynomial expansion increases with the number of modulation layers N. In summary, in a first embodiment

• Complex modulation (amplitude and phase) or with only amplitude modulation in the different modulation layers 3, 3a, ... yield a nonlinear relationship between the output field and the modulation even without intensity detection.

In a further embodiment,

• Phase-only modulation in the different modulation layers 3, 3a, ... yield a nonlinear relationship between the output field and the modulation only when intensity detection is performed.

For the above-mentioned embodiments, we present an implementation for programming the non-linear transform:

With complex modulation or amplitude modulation: t i ., displ ,ay =t. i, d ,ata X S. i, acti .vati .on + JB i., acti .vati .on t i ., d.i.spl ,ay : dis rpla pyed r parameter on the modulation layer t i ., d ,ata : the data to he processed

S i . , acti .vati .on : Trainable scaling ° ^r parameters

B i ., acti .vati .on : Trainable bias ' parameters

With phase-only modulation:

B i . , acti .vati .on : Trainable bias ' parameters

Essentially, the trainable parameters Si, Bi are added to each pixels t1 , t2, ... of the modulation layers 3, 3a, ... . Its effect is analogous to the nonlinear activation in deep neural networks.

We employ systems that perform linear transformations on input data and programming commands. We use multiple such linear systems which use at least partially the same input data and trainable parameters as inputs so that the multiple linear transformations result in an overall nonlinear mapping. Alternatively, a single linear system can be used repeatedly using optical and/or electronic feedback. The input data and trainable parameters can be entered into each of the linear systems such as linear optical systems, for instance through an electronic-to-optical interface or any other suitable method. The output data can be recorded on detectors such as optical detectors, for example.

After setting up the basis, we describe the several embodiments for the implementation of the conceptual architecture (the first embodiment) and the explained nonlinear processing with linear systems not limited to the only optical systems which can be realized also by electrical and mechanical linear systems. The second embodiment employs modulation layers 3, 3a, ... in the form of an SLM or multiple SLMs arranged in a plane p where the modulation layers 3, 3a, ...are unfolded and displayed on dynamic (reconfigurable by input electrical signal) SLM panels. Examples of SLMs include but not limited to Liquid Crystal devices for phase-only modulation as well as Digital Micromirror Devices (DMD), photonic crystal spatial waveform modulators and grating light valves.

The pass through consecutive modulation layers 3, 3a, ... are realized by a reflective element 5 comprising a reflective surface 8 positioned across the panels (see Figure 3) where the propagation matrix boils down to free space diffraction. That is, figure 3 depicts an implementation of the optical computing framework by means of a (multitude of) dynamic/reconfigurable spatial light modulator(s) and a reflective element 5 comprising a reflective surface 8, which enables to unfold the modulation layers 3, 3a, ... on a single plane p. The figure depicts a carrier signal Sc in the form of an input beam, modulation layers 3, 3a, ... displayed on a dynamic/reconfigurable device, and an output signal So in the form of an output beam.

The reflective surface 8 can be flat but also can be curved to modify the free space propagation such as providing a focusing effect. The light source (not shown) can be a Gaussian beam or structured light by means of another SLM devices to introduce an additional control in the nonlinear processing. The trainable parameters can be optimized by having the configuration realized in a physical differentiable simulation model (i.e. digital twin), where error backpropagation can be employed, which enables the all-optical inference and/or smart encoding by displaying the data modulated by the trainable parameters on SLM(s). The output beam So can be detected by a photo-detector array where each photo-detector site is reserved for a specific class for a classification task to carry out all-optical inference. The output beam So can also be detected by a camera where the detected intensity distribution or the complex electric field distribution in case of a holographic recording, such as (but not limited to) off-axis digital holographic interferometry for all-optical regression or a latent space expression of the shown data sample where this latent space expression can be digitally post-processed.

The experimental results can be utilized to account for experimental imperfections in the simulation model/digital twin to fine tune the trainable parameters accordingly (see Figure 4). That is, figure 4 depicts a calculation loop of the trainable parameters, for example, with a digital twin approach. Panel a) depicts a digital twin implementation for error backpropagation St and panel B) depicts a schematics of a conceivable experimental setup where the trainable parameters are tested and discrepancies Sf between simulation and experiments are fed back to the digital twin to match the results. To fine-tune (or calibrate) trainable parameters, experimental imperfections can be measured (such as misalignment angle of the components with respect to each other) and these imperfections can be included in the digital twin. Alternatively, a machine learning approach can be employed to alter the forward pass of the digital twin to collectively account for experimental imperfections by having a loss function that compares the outputs of the forward pass of digital twin with the experimental outputs.

An example of processing for a sample from the MNIST handwritten dataset is provided as a schematic in Figure 5. In particular, an experimental sample result of the optical computing framework is depicted by means of four trainable modulation layers 3, 3a, ... displayed on the spatial light modulator. The figure depicts the carrier signal Sc in the form of an input beam, calculated modulation layers 3, 3a, ... for classification of the digit MNIST dataset, and an output signal So in the form of an output beam shape that is captured by a detection device 7 in the form of a camera. The figure-a depicts trainable parameters in the form of scaling masks 9, 9a, ... and bias masks 10, 10a, ... for four modulation layers 3, 3a, 3b, 3c. The figure-b demonstrated sample “7” modulated by these bias and scaling masks 9, 9a, ... 10, 10a, ... resulting in a focused spot on a certain location of the detector plane of the detector 7, which is aimed during training for the classification task. The calculation of trainable parameters can be performed via other suitable methods as well such as Surrogate optimization [Oguz, I., Hsieh, J.L., Dine, N.U., Tegin, II., Yildirim, M., Gigli, C., Moser, C. and Psaltis, D., 2022. Programming nonlinear propagation for efficient optical learning machines. arXiv preprint arXiv:2208.04951],

As follows from this example, the input layer 2 can be effectively provided by the carrier signal Sc such as an optical plane wave for example, the modulation layers 3, 3a, ... is where the input data is placed (in electronic format via a SLM for example) and the output layer 4 is basically the result of input, modulation and propagation to the output. The trainable parameters can be in the digital domain. For example, for multiple bounces, each modulation layer 3, 3a, ... (portion of the SLM) can be configured to be a product of the input data multiplied by trainable parameters onto which is added a bias. The input data can be in electronic format and be “imprinted” onto the optical field. There is no non-linear transformation of the optical field by diffraction, however the optical field is non-linearly related to the data presented multiple times.

The experimental implementation in the laboratory environment is depicted in Figure 6. That is, figure 6 depicts a realization of an optical computing framework by means of a dynamic/reconfigurable spatial light modulator 3 and a reflective element 5 in the form of a mirror comprising a reflective surface 8, which enables to unfold the modulation layers 3, 3a, ... on a single plane. The figure depicts the carrier signal Sc in the form of the input beam, modulation layers 3, 3a, ... displayed on a dynamic/reconfigurable device, and an output signal So in the form of an output beam. The figure depicts two different views of the platform realized in laboratory environment.

The third embodiment employs modulation layers 3, 3a, ... comprising materials that can preserve their state passively until they are modified again and modulate light according to their states. A typical example of such nonlinear optical materials are Phase Change Material (PCM) and they can be also used to optically modulate light in addition to the electrical modulation, which is utilized as previously mentioned by liquid crystal devices and DM Ds. In this case, there are two light sources (not shown) for inference and data writing on PCM. Following the same unfolding strategy explained above, we build a cavity with either dichroic mirrors 8 (relying on having different wavelengths for data writing beam and the inference beam) or partially reflecting mirrors (where the wavelengths for data writing beam and the inference beam can be the same) and the PCM plane (see Figure 7). That is, figure 7 depicts an implementation of an optical computing framework by means of an optically reconfigurable material of variable complex transmittance and two reflective elements 8 comprising reflective surfaces, which enables to unfold the modulation layers 3, 3a, ... on a single plane. The figure depicts the carrier signal Sc in the form of an input beam, modulation layers 3, 3a, ... written on a slab of phase change material, and an output signal So in the form of an output beam. The shown components are at least two- dimensional. There is an indicated direction of a data writing beam Sw in the figure, but it can be either way (bottom-to-up or up-to-bottom). Note that the phase change of a PCM modulates both the absorption and refractive index of the material yielding a complex modulation.

Another way to utilize a PCM is relying on the refractive index change, subsequently yielding different response in reflections because of Fresnel reflection laws. This simply removes the need for one of the mirrors (see Figure 8). That is, figure 8 depicts an implementation of the optical computing framework by means of an optically reconfigurable material of variable reflection properties and a reflective element 5 comprising a reflective surface 8, which enables to unfold the modulation layers 3, 3a, ... on a single plane. The figure depicts the carrier signal Sc in the form of an input beam, modulation layers 3, 3a, ... written on a slab of phase change material, and an output signal So in the form of an output beam. The shown components are at least two-dimensional. There is an indicated direction of a data writing beam Sw in the figure, but it can be either way (bottom-to-up or up-to-bottom).

For both cases regarding using PCM, the data writing beam can modulate the PCM either by scanning a focal spot or displaying the target modulation by a 2-dimensional display device.

The above-mentioned embodiments can be implemented in a monolithic fashion as exemplified in Figure 9 where the modulation layers 3, 3a, ... are positioned in one side or as exemplified in Figure 10 where the modulation layers 3, 3a, ... are distributed in both sides. That is, figure 9 depicts a monolithic implementation of the optical computing framework by means of a (multitude of) dynamic/reconfigurable spatial light modulator(s), transparent material and a coated surface for necessary reflectivity. The figure depicts the carrier signal in the form of an input beam, modulation layers displayed on a dynamic/reconfigurable device, and an output signal in the form of an output beam. The shown displays are two-dimensional. The angled input and output facets of the transparent medium are arranged to minimize Fresnel reflections from the input and output surfaces.

Figure 10 depicts an implementation of the optical computing framework by means of a (multitude of) dynamic/reconfigurable spatial light modulator(s) 3, 3a, ... facing across each other and a transparent material 11 in between for a monolithic configuration or free space. The figure depicts the carrier signal Sc in the form of the input beam, modulation layers 3, 3a, ... displayed on a dynamic/reconfigurable devices, and an output signal So in the form of an output beam. The shown displays are two-dimensional. The angled input and output facets 12, 13 of the transparent material 11 are arranged to minimize Fresnel reflections from the input and output surfaces.

Possible unified practical realization of this framework with and without light source 14, electronics 15 and multi-layer cavity are illustrated in Figures 11 and 12. That is, figure 11 depicts a compact monolithic implementation of the optical computing framework by means of a device 1 comprising a light source 14, spatial light modulator (SLM) 3, photodetector array 7 (camera), transparent material 11 , mirror 5 , and necessary electronics 15 arranged in close proximity. Figure 12 depicts a compact monolithic implementation of the optical computing framework by means of ambient light, spatial light modulator 3, transparent material 11 and a mirror 5 with necessary electronics 15 in close proximity. The device 1 furthermore comprises an imaging module 16 and a photodetector array 7 (camera). This scheme uses ambient light captured via the imaging module 16 for further processing without using on board light source.

The fourth embodiment is using waveguides for propagation and Mach-Zehnder Interferometers (MZIs) for complex modulation. The MZI mesh is an architecture that can intertwine the propagation and complex modulation steps described above. In general, one can consider rectangular and triangular mesh architectures as explained in [Pai, S., Bartlett, B., Solgaard, O., & Miller, D. A. (2019). Matrix optimization on universal unitary photonic devices. Physical Review Applied, 11(6), 064044.]. All other possibilities can be expressed as a combination of the two. We propose, instead of performing linear operations as investigated already in the literature, performing nonlinear processing by introducing the input data modulated by trainable parameters into the MZI modulators as complex modulation transmittance where the carrier signal Sc in the form of input light can be but not necessarily limited to unmodulated continuous wave laser beams (see Figure 13). That is, figure 13 depicts an implementation of the optical computing framework by means of a Mach-Zehnder Interferometer (MZI) mesh. The figure depicts the input light source(s), modulation layers 3, 3a, ... as sub-groups of MZI units 6, and output signals So in the form of output beams. The data is represented on the MZI units 6 in modulation layers 3, 3a, ... as a complex transmittance of the MZI unit 6. In the depicted example the MZI unit 6 consists of two inputs, two modulators (electro-optical, thermal, etc.), two coupler regions, and two outputs, which is schematically depicted in the Figure. The input data can be combined with the trainable parameters in different manners and presented to MZI units 6. That is, similarly to above-mentioned methods, the carrier signal Sc in the form of input light can be modulated by trainable parameters to improve the success metric of the undertaken machine learning task. Two modulators can perform an arbitrary linear transform between the input and output nodes if the MZI unit 6 in complex domain. In the MZI mesh, either input data and trainable parameters can be combined by simply having trainable parameters act on input data to be presented in the MZI units 6, or input data and propagation parameters can be introduced explicitly on different stages (for example modulation layers 1 , 3, 5, ... for data presentation and modulation layers 2, 4, 6, ... for trainable parameters, see Figure 8) .

The fifth embodiment details the implementation of the technique using various linear systems 17, such as electrical or mechanical, where nonlinearity is created by repetitive linear transforms. The architecture is presented in Figure 14, which is a generalized format of the optical configuration depicted in Figure 8. That is, figure 14 depicts the implementation of a generic computing framework by means of linear systems 17. The whole device 1 is composed of linear systems 17 and modulation layers 3, 3a, ... accept input data i_data and trainable parameters t_trainable. That is, the modulation layers 3, 3a, ... can be constructed with any linear physical system that enables a controllable transfer matrix between its input and output. These modulation layers 3, 3a, ... are operated in the same manner as the previous optical systems for inputting data and trainable parameters such as bias. Modulating a modulation layer 3, 3a, ... is achieved by modifying the elements of the corresponding transfer matrix based on input data and trainable parameters, e.g. bias parameters, such as capacitance, inductance, or resistance in an electrical system. While it is not mandatory to have a linear system 17 in between modulation layers 3, 3a, ... , they can be inserted for practical reasons. Since all units are linear, the implementation can be condensed by merging the modulation layer 3, 3a, ... with a linear system. Hence, in summary it can be said that the present invention provides a novel reconfigurable low power nonlinear data processing device such as an optical computing device. The device is based on the use of linear transformations in a specific manner to synthesize arbitrary nonlinearities, which can be easily programmed to perform a variety of computations.

The full potential of optical computers, which can greatly enhance processing speed and energy efficiency by using light instead of electrons, can be realized with a platform that allows both programmable linear and nonlinear operations using low-power, compact and simple light sources. To address this challenge, we present an optical technique that achieves the equivalent of optical nonlinearity more efficiently. The essence of the proposed technique relies on multiple linear scattering that uses low optical power to effectively synthesize a nonlinear operation in the optical domain. By exploiting this relationship, arbitrary nonlinear transformations can be programmed digitally, and light can perform an all-optical computation without requiring electronic-to-optical conversion or high-power lightmatter interactions. With this approach, an all-optical programmable computing framework is introduced. Our approach enables the implementation of fully programmable, all-optical linear and nonlinear operations on a single platform with low power light sources and it also makes it possible to perform complex computations more efficiently and cost-effectively.

LIST OF REFERENCE SIGNS device 17 linear system input layer , 3a, modulation layer Sc carrier signal output layer Smc modulated carrier signal reflective element So output signal

Mach-Zehnder Interferometer Sf feedback signal unit Sw data writing beam detection device P1, P2, ... propagation step reflective surface scaling masks p plane 0 bias masks 1 transparent material i1 , i2 pixels 2 input facet t1, t2 pixels 3 output facet o1, o2 pixels 4 light source i_data input data 5 electronics i_trainable trainable parameter6 imaging module

Claims

1 . A device (1) for processing data comprising:

- at least one input layer (2),

- at least one modulation layer (3),

- at least one output layer (4), wherein the input layer (2) is configured to receive at least one carrier signal (Sc), wherein the modulation layer (3) comprises input data, wherein the modulation layer (3) is configured to modulate the carrier signal (Sc) such, that a modulated carrier signal (Smc) comprising the input data being linearly transformed is generated, characterised in that the device (1) is configured such, that the modulated carrier signal (Smc) is repeatedly modulated with the input data and linearly transformed, wherein an output signal (So) comprising output data is produced at the output layer (4), and wherein the output data comprises or consists of the input data being nonlinearly processed.

2. The device (1) according to claim 1 , wherein the modulation layer (3) is configured to repeatedly modulate the modulated carrier signal (Smc) and preferably upon receipt of a feedback signal (Sf), whereby the input data is repeatedly linearly transformed.

3. The device (1) according to any one of the preceding claims, wherein the device (1) comprises at least two modulation layers (3, 3a) being arranged in a cascading manner, whereby the modulated carrier signal (Smc) is repeatedly modulated in a cascading manner and the input data is repeatedly linearly transformed.

4. The device (1) according to any one of the preceding claims, wherein the input layer (2) and the modulation layer (3) are arranged such, that the carrier signal (Sc) is allowed to propagate in at least one propagation step (P1) between the input layer (2) and the modulation layer (3), whereby different dimensions of input data are coupled, and/or wherein the modulation layer (3) and the output layer (4) are arranged such, that the modulated carrier signal (Smc) is allowed to propagate in at least one propagation step (P2) between the modulation layer (3) and the output layer (4), whereby different dimensions of input data are coupled.

5. The device (1) according to claim 4, wherein the device (1) is configured such that the carrier signal (Sc) and/or the modulated carrier signal (Smc) is propagating via at least one of:

- a free space diffraction,

- a linear or nonlinear propagation through a medium,

- a linear or nonlinear propagation through one or more preferably coupled waveguides, or

- a linear or nonlinear propagation through linear optical elements or linear electrical elements or linear mechanical elements.

6. The device (1) according to any one of the preceding claims, wherein two or more modulation layers (3, 3a) are arranged so as to extend in at least one common plane (p), and wherein at least one reflective element (5) is facing towards said modulation layers (3, 3a) such, that the carrier signal (Sc) and/or the modulated carrier signal (Smc) is reflected at said reflective element and allowed to propagate with respect to said modulation layers (3, 3a).

7. The device (1) according to any one of the preceding claims, wherein the modulation layer (3) is at least partially transmissive for the carrier signal (Sc) and/or the modulated carrier signal (Smc), and/or wherein the modulation layer (3) is at least partially reflective for the carrier signal (Sc) and/or the modulated carrier signal (Smc).

8. The device (1) according to any one of the preceding claims, wherein the carrier signal (Sc) is an optical signal, preferably laser light, LED light or ambient light.

9. The device (1) according to any one of the preceding claims, wherein the modulation layer (3) comprises or consists of at least one of a spatial light modulator, an electro-optical modulator, a thermal modulator, a Mach-Zehnder Interferometer unit (6), or a nonlinear optical material, and/or wherein the modulation layer (3) is configured to modulate at least one of an intensity, a phase, an amplitude, a frequency, or a polarization of the carrier signal (Sc) and/or the modulated carrier signal (Smc).

10. The device (1) according to claim 9 comprising a mesh of two or more Mach-Zehnder Interferometer units (6), wherein two or more Mach-Zehnder Interferometer units (6) of the mesh of Mach-Zehnder Interferometer units (6) are grouped together into one or more groups of Mach-Zehnder Interferometer units (6), and wherein, in each group of Mach-Zehnder Interferometer units (6), the modulation layers (3) of the Mach-Zehnder Interferometer units (6) are comprising the input data such that the modulated carrier signal (Smc) is modulated in a cascading manner by each group of Mach-Zehnder Interferometer units (6) having input data modulating the carrier signal (Sc) and/or the modulated carrier signal (Smc).

11 . The device (1) according to any one of the preceding claims, wherein the device (1) comprises at least one detection device (7) being configured to detect the output signal (So), and wherein said detection device (7) comprises or consists of the output layer (4), and/or wherein the detection device (7) comprises or consists of at least one optical detector and/or is configured for detecting an electric field and/or for holographic recording.

12. The device (1) according to any one of the preceding claims, wherein the input data is combined with at least one trainable parameter, wherein the device (1) is configured to determine at least one feedback signal (Sf), and wherein the device (1) is further configured to transmit the feedback signal (Sf) to the modulation layer (3) in order to train the trainable parameter based on the feedback signal (Sf).

13. The device (1) according to claims 11 or 12, wherein the output signal (So) being detected by the detection device (7) constitutes a final output signal comprising or consisting of finally processed data, or wherein the device (1) further comprises at least one layer of a digital network and is configured to propagate the output signal (So) being detected by the detection device (7) at least once through the layer of the digital network, whereby at least one final output signal comprising or consisting of finally processed data is generated.

14. The device (1) according to claim 13, wherein the device (1) comprises multi-layer networks and is configured to propagate the output signal (So) at least once through the multi-layer networks.

15. The device (1) according to any one of the preceding claims, wherein the device (1) is configured to produce the output data comprising or consisting of a reduced representation of the input data via optical nonlinear processing, and wherein said output data is preferably further processed by at least one layer of a digital network, and/or wherein the device (1) is configured as a frontend nonlinear processing device.

16. The device (1) according to any one of the preceding claims, wherein the input layer (2), the modulation layer (3) and the output layer (4) are monolithically integrated.

17. A method of manufacturing a device (1) for processing data, preferably a device (1) as claimed in any one of the preceding claims, wherein the method comprises the steps of:

- Providing at least one input layer (2),

- Providing at least one modulation layer (3),

- Providing at least one output layer (4), wherein the input layer (2) is configured to receive at least one carrier signal (Sc), wherein the modulation layer (3) comprises input data, wherein the modulation layer (3) is configured to modulate the carrier signal (Sc) such, that a modulated carrier signal (Smc) comprising the input data being linearly transformed is generated, characterised in that the device (1) is configured such, that the modulated carrier signal (Smc) is repeatedly modulated with the input data, wherein an output signal (So) comprising output data is produced at the output layer (4), and wherein the output data comprises or consists of the input data being nonlinearly processed.

18. A method of processing data comprising the steps of:

- Providing at least one device (1) for processing data as claimed in any one of claims

1 to 16, and

- Providing at least one carrier signal (Sc), wherein the input layer (2) receives the carrier signal (Sc), wherein the modulation layer (3) modulates the carrier signal (Sc) such, that a modulated carrier signal (Smc) comprising the input data being linearly transformed is generated, characterised in that the device (1) repeatedly modulates and linearly transforms the modulated carrier signal (Smc), wherein an output signal (So) comprising output data is produced at the output layer (4), and wherein the output data comprises or consists of the input data being nonlinearly processed.