Kyle Daruwalla

New preprint: Walking the Weight Manifold!

2025-05-29T00:00:00+00:00

A lot of my work at CSHL involve thinking how to embed structure into neural networks in a principled and controllable manner. This work does just that, taking inspiration from neuromodulation! We built not a single neural network, but a topologically contrained ensemble that learns concurrently. Check it out on arXiv!

New paper accepted at NeurIPS 2024!

2024-11-06T00:00:00+00:00

Our work on the Neural Tangent Ensemble is accepted as a spotlight at NeurIPS 2024! Bayesian ensembles are great for many applications such as continual learning, but it is computationally intensive to have an ensemble of large neural networks. In this work, we use neural tangent kernel (NTK) theory to show that a single network can be thought of as an ensemble of functions. We call this the Neural Tangent Ensemble (NTE) and derive a belief update rule for weighting members of the ensemble. Turns out this rule is closely related to (single sample) SGD! Our work focuses on continual learning as an example application, but the NTE is applicable anywhere you would use a Bayesian ensemble. Read the paper here, visit us at NeurIPS, or contact us if you have any questions about the NTE!

New paper accepted at UniReps NeurIPS Workshop!

2024-10-10T00:00:00+00:00

After working on representational geomtery at a summer workshop, CiCi approached us about collaborating on a paper exploring changes in representational geometry during grokking. Read the paper here or join us for the discussion at UniReps in Vancouver!

New preprint: Cheese 3D!

2024-05-01T00:00:00+00:00

I'm excited to share the first bit of work from my time at Cold Spring Harbor Laboratory. In an effort to learn neuroscience through osmosis, I've been collaborating with the Hou Lab to study facial movements in mice. In comparison to other behaviors, facial movements cover a wide spatial and temporal dynamic range, and in this work, we carefully developed a 3D pose estimation pipeline that is accurate and robust across these spatiotemporal scales. Read the preprint here, and contact us if you're interested in using our method!

Completed my PhD!

2022-06-19T00:00:00+00:00

I defended my thesis on Building Energy Efficient Computers with Brain-Inspired Computing Models (I plan to upload a "pretty" copy of the PDF soon). After a long break from research, I'll be starting as a NeuroAI Scholar at Cold Spring Harbor Lab in August! I'm very excited to be working in a collaborative environment with real neuroscientists. Please contact me if you think we can have an interesting collaboration!

Invited talk at Cold Spring Harbor Labs

2022-02-09T00:00:00+00:00

I gave an invited talk on my research on building energy efficient computers at Cold Spring Harbor Labs (CSHL). This was a fun opportunity, and I greatly enjoyed the feedback and discussion. You can find my talk slides here (note that the future work sections have been removed since they include ongoing proposals).

Poster at SNUFA '21

2021-11-26T00:00:00+00:00

I presented my work on a biologically plausible learning rule as a poster at the Spiking Neural networks as Universal Function Approximators (SNUFA '21). There were many great talks and posters presented, but I was most excited by the presentations on spiking neural networks (SNNs) being applied to solve real problems:

the European Space Agency is looking at SNNs to solve graph problems (first time I've seen a graph neural network (GNN) equivalent for spikes!)
StereoSpike uses two event-based cameras with a U-Net architecture to perceive depth. This work was especially cool for me, because I briefly looked at stereoscopic perception using conventional computer vision algorithms at the start of my Ph.D. We were trying to build a bitstream computing circuit as a stepping stone to SNNs, so it's cool to see someone find a solution to the problem years later.

Our work

I presented our work on learning in biological networks by optimizing the information bottleneck. The success of deep learning has emphasized the importance of depth when training networks to solve complex problems. Depth isn't an issue for artificial neural networks (ANNs), because back-propagation provides a systematic way to assigning credit regardless of depth. Currently, there is not an accepted, biologically plausible back-propagation equivalent for SNNs, though we can use surrogate gradients when the plausibility constraint is removed.

Our work took inspiration from HSIC training for ANNs. Instead of computing a loss at the very end of the network, then propagating the loss backwards, HSIC training optimizes each layer independently. Every layer is updated to minimize

$$ cal(L)_"HSIC" = upright(HSIC)(vecb(z)^ell, vecb(x)) - gamma upright(HSIC)(vecb(z)^ell, vecb(y)) $$

where $vecb(z)^ell$ is the output of the layer $ell$, and $vecb(x)$/$vecb(y)$ are the input/output of the entire network, respectively. We show the gradient descent update for this objective can be decomposed into two components—a local Hebbian component and a layer-wise global modulatory signal.

One challenge to applying this rule for SNNs directly is that the HSIC is computed over a batch of samples, but biological networks see samples sequentially, one-at-a-time. To overcome this, we encode a batch as a window of samples over time (i.e. a batch size of $N$ corresponds to the last $N$ samples presented to the network). We show that the local component depends only on the current sample, and the global component depends on the prior samples. Then, we propose using an auxiliary reservoir network to compute the global component as shown below.

Our learning rule is a three-factor Hebbian rule. It contains a local component that depends on the current sample, and a global component that depends on past samples. A reservoir is used to compute the global component.

Note that the reservoir can be trained a priori using random data, and it does not need to be trained during the main learning task (though it can be). Check out our poster or preprint for more details!