Compressed Event Sensing (CES) Volumes for Event Cameras

Abstract

Deep learning has made significant progress in event-driven applications. But to match standard vision networks, most approaches rely on aggregating events into grid-like representations, which obscure crucial temporal information and limit overall performance. To address this issue, we propose a novel event representation called compressed event sensing (CES) volumes. CES volumes preserve the high temporal resolution of event streams by leveraging the sparsity property of events and the principles of compressed sensing theory. They effectively capture the frequency characteristics of events in low-dimensional representations, which can be accurately decoded to raw high-dimensional event signals. In addition, our theoretical analysis show that, when integrated with a neural network, CES volumes demonstrates greater expressive power under the neural tangent kernel approximation. Through synthetic phantom validation on dense frame regression and two downstream applications involving intensity-image reconstruction and object recognition tasks, we demonstrate the superior performance of CES volumes compared to state-of-the-art event representations.

Proof of Theorem 1

Theorem 1

Given a non-zero distinct s-sparse dataset ${\mathfrak {X}}=\{\vec {x_i}\}_{i=1}^I$, let ${\mathfrak {H}}_a$ and ${\mathfrak {H}}_b$ be the RKHS associated with the NTK of same-architecture fully-connected network with $\Psi _a^T\vec {x}$, and $\Psi _b^T\vec {x}$ as input, where $\Psi _a$ holds the non-degenerate property while $\Psi _b$ does not, the following subset inclusion relation hold:

$$\begin{aligned} {\mathfrak {H}}_b \subsetneq {\mathfrak {H}}_a. \end{aligned}$$

(16)

We first introduce two key ingredients of the proof:

Lemma 1

(Theorem 2.17 in Saitoh et al. (2016)) Let $K_a,K_b\, E \times E \rightarrow {\mathbb {C}}$ be two positive semi-definite kernels. Then the following two statements are equivalent:

• 1. The Hilbert space ${\mathfrak {H}}_{b}$ is a subset of ${\mathfrak {H}}_{a}$

• 2. There exist $\gamma > 0$, such that

  $$\begin{aligned} K_b \preceq \gamma ^2 K_a. \end{aligned}$$

  (17)

Lemma 2

(Proposition 2 in Jacot et al. (2018), Theorem 6 in Luís et al. (2024)) For a fully-connected network adopting a non-polynomial Lipschitz nonlinearity activation function $\sigma$, for any input dimension $n_0$, the limiting NTK is strictly positive definite if the number of layer $L \ge 2$.

Proof of Theorem 1

According to Lemma 1, to obtain ${\mathfrak {H}}_b \subsetneq {\mathfrak {H}}_a$, we require proof that $\gamma ^2 K_a - K_b$ is a positive semidefinite kernel for some $\gamma > 0$, whereas $\gamma ^2 K_b - K_a$ is not a positive semidefinite kernel for all $\gamma > 0$.

Consider arbitrary non-empty subset of ${\mathfrak {X}}$, $\{\vec {x_i}\}_{i=1}^r \subset {\mathfrak {X}}$, for $1 \le r \le I$, the NTK matrix ${\textbf{K}}$ with size $r \times r$ could be constructed for kernel $K_a$ and $K_b$, whose entries are

$$\begin{aligned} {\textbf{K}}_{a}^{i,j} = K_a (\vec {x_i}, \vec {x_j});~~ {\textbf{K}}_{b}^{i,j} = K_b (\vec {x_i}, \vec {x_j}). \end{aligned}$$

(18)

As introduced in the proposed NTK model, deep learning methods first represent events using sensing matrix $\Psi$, and then feed the representation into a neural network g. We refer to the NTK of the network g as $K_g(\cdot , \cdot )$. Therefore, for two different sensing matrices $\Psi _a$ and $\Psi _b$, the NTK of the whole networks can be represented as

$$\begin{aligned} \begin{aligned} K_{a}(\vec {x_i}, \vec {x_j})&= K_g(\Psi _a^T \vec {x_i}, \Psi _a^T \vec {x_j})\\ K_{b}(\vec {x_i}, \vec {x_j})&= K_g(\Psi _b^T \vec {x_i}, \Psi _b^T \vec {x_j}). \\ \end{aligned} \end{aligned}$$

(19)

According to Lemma 2, when we adopt the same network settings to Jacot et al. (2018), we obtain that the NTK of the network $K_g$ is a strictly positive definite for distinct network inputs.

Since $\Psi _a$ holds the non-degenerate property as described in Equation (11), the compressed representations $\Psi _a^T \vec {x_i}$ are distinct. Therefore, the NTK matrix ${\textbf{K}}_a$ is positive definite with eigenvalues $\lambda _a^i >0$. Whereas, $\Psi _b$ does not hold this property, i.e., there might exist “degenerate" vector pairs $\vec {x_i}$ and $\vec {x_j}$ such that $\Psi _b^T \vec {x_i} = \Psi _b^T \vec {x_j}$, leading to identical values in i th and j th rows of the NTK matrix ${\textbf{K}}_b$. And thus, the eigenvalues $\lambda _b^i \ge 0$.

Therefore, for each non-empty subset $\{\vec {x_i}\}_{i=1}^r$ of ${\mathfrak {X}}$, $\gamma ^2 {\textbf{K}}_a - {\textbf{K}}_b$ is positive semidefinite when

$$\begin{aligned} \gamma = \sqrt{ \frac{\max _i \lambda _b^i}{\min _i \lambda _a^i}}. \end{aligned}$$

(20)

Here, let the $\gamma _{max}$ be the maximum of $\gamma$ respective to all the non-empty subsets of ${\mathfrak {X}}$. Based on the definition of positive semidefinite kernels (see Definition 12.6 in Martin (2019)), $\gamma _{max}^2 K_a - K_b$ is a positive semidefinite kernel on ${\mathfrak {X}}$. This enables us to apply Lemma 1 to obtain that

$$\begin{aligned} {\mathfrak {H}}_b \subseteq {\mathfrak {H}}_a, \end{aligned}$$

(21)

Conversely, since the eigenvalues of ${\textbf{K}}_b$ contains zero in the degenerate case, for all $\gamma > 0$, $\gamma ^2 {\textbf{K}}_b - {\textbf{K}}_a$ is not positive semidefinite. Thus, for all $\gamma > 0$, the kernel function $\gamma ^2 K_b - K_a$ is not positive semidefinite. According to Lemma 1, ${\mathfrak {H}}_a$ is not a subset of ${\mathfrak {H}}_b$. Combining ${\mathfrak {H}}_b \subseteq {\mathfrak {H}}_a$, we can come to ${\mathfrak {H}}_a \ne {\mathfrak {H}}_b$, thereby concluding the proof. $\square$