Spiking Neural Network (SNN) 学习笔记

Points

• Input:

  • One counter-intuitive fact is that almost all the mainstream SNN frameworks’ input shape is not a series of 1-D spike signals. Actually, the input shape is pretty much like other ANNs, but with an additional T time dimension. Here is the common input shape: T x N x C x H x W, where T means time, N means batch size, C means channel number, and H, W stands for the input image size. Except for the first two dimensions (T, N), the rest of the dimensions are quite flexible and can be modified as needed, and we can simplify the input shape as T x N x X.
  • For the Loihi SNN hardware platform, the input size is a bit different, but it’s a simple permutation: N x X x T.

• Another counter-intuitive side of SNN is, SNN is usually not used solely. Here, I mean people don’t solely use SNN neurons to build their entire model. The neuron-alone model can be potentially more efficient, but it is not good at capturing local and global patterns, and features. Instead, neurons here are only treated as a substitute for activation functions in ANNs.

• Characteristics of SNN:

  • Generative Property of SNN: Generative, as its name shows, for a trained SNN, for an output neuron of a certain output class, if we scale the values of all the input neurons connected to this neuron and rearrange them properly, we can get a general pattern image of this class. This pattern could be clear and distinctive, or blurry and non-distinguishable, which reflects how well this class is trained. Like the image below, the last one is worse compared to the first three. This property will be used for demonstrating the results.

    

  • Variable Threshold: Each pattern (e.g., in MNIST, the different number) has a different number of activations. For example, the number 1 has less activation (white pixels in the 28x28 image) than the number 8, generally speaking. Therefore, those classes with more activation will overshadow all the other classes with less activation. To avoid this kind of inter-class imbalance, we need to set a different threshold for each class, which is calculated based on the number of activation each class contains.

  • Lateral Inhibition: Many different neurons in the same layer could get excited at different time stamps, however, when one neuron gets excited, this mechanism (lateral inhibition) will reduce the activity and inhibit other neurons in the same layer to get excited. This property is also called Winner-Takes-All(WTA). In biology, the neuron gets excited first and lowers down the membrane potential of other neurons in the same layer.

How to Rewrite an ANN to SNN

• First, select a proper neuron type (let’s call it Nx), and replace all the activation functions in your original ANN model with Nx.

  The literature [P1] provides a theoretical basis for analyzing the conversion of ANN to SNN. The theory shows that the IF neuron in SNN is an unbiased estimator of the ReLU activation function over time.

• Then, remove all batch normalization layers.

• If you plan to eventually deploy the model to the Loihi platform, you should set all the bias items in related layers to False, like Conv and Linear. This is because bias is not supported in Loihi.

STDP

• The algorithm that is commonly used in neuron training is called Spike Time Dependent Plasticity (STDP, 突触时间依赖可塑性)。

• STDP can only be applied to the training of neurons (SNN layers), but not other ANN layers used in the same network.

• STDP is actually a biological process used by the brain to modify its neural connections (synapses). Since the unmatched learning efficiency of the brain has been appreciated for decades, this rule was incorporated in ANNs to train a neural network. Modeling of weights is based on the following two rules -

  • Any synapse that contributes to the firing of a post-synaptic neuron should be made strong i.e its value should be increased.
  • Synapses that don’t contribute to the firing of a post-synaptic neuron should be diminished i.e its value should be decreased.

  Here is an explanation of how this algorithm works:

  Consider the scenario depicted in this figure

  

  Four neurons connect to a single neuron by synapse. Each pre-synaptic neuron is firing at its own rate and the spikes are sent forward by the corresponding synapse. The intensity of the spike translated to the post-synaptic neuron depends upon the strength of the connecting synapse. Now, because of the input spikes membrane potential of the post-synaptic neuron increases and sends out a spike after crossing the threshold. At the time when the post-synaptic neuron spikes, we’ll monitor which all pre-synaptic neurons helped it to fire. This could be done by observing which pre-synaptic neurons sent out spikes before post-synaptic neurons spiked. This way they helped in the post-synaptic spike by increasing the membrane potential and hence the corresponding synapse is strengthened. The factor by which the weight of the synapse is increased is inversely proportional to the time difference between post-synaptic and pre-synaptic spikes given by this graph

  

Parameters

Building a Spiking Neural Network from scratch is not an easy job. There are several parameters that need to be tuned and taken care of. Combinations of so many parameters make it worse. Some of the major parameters that play an important role in the dynamics of a network are -

• Learning Rate
• Threshold Potential
• Weight Initialization
• Number of Spikes Per Sample
• Range of Weights

Frequently-used Activation Layers/Neurons

• IF stands for Integrate-and-Fire. It is a simple model in which a neuron integrates incoming inputs over time and generates a spike when the membrane potential reaches a threshold value.
• LIF stands for Leaky Integrate-and-Fire. It is an extension of the IF model that takes into account the leakage of charge through the neuron membrane over time. This model is widely used in SNNs due to its simplicity and efficiency.
• PLIF stands for Poisson Leaky Integrate-and-Fire. It is a model that takes into account the stochastic nature of synaptic inputs in biological neurons. In this model, synaptic inputs are modeled as a Poisson process, and the membrane potential of the neuron is determined by the integration of these inputs over time, taking into account the leakage of charge through the membrane.

Transform Traditional RGB Images to Events

• Methods:
  1. Show images on a paper/monitor, and then turn on/off the light.
  2. Show images on a paper/monitor, and move the image/camera/sensor horizontally to keep the depth the same.
  3. Show images on a paper/monitor, and rotate the camera.
• Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades: The original paper proposed for generating N-MNIST (Event camera version MNIST). It is collected by using an actual DVS to capture the MNIST images shown on a monitor, which slightly rotates the DVS to generate brightness change.
• For normal RGB images, we can also generate spikes for a certain period of time based on a certain distribution (e.g., Poisson Distribution) of pixel positions with a fixed frequency.

Specialized Hardware

• Intel Loihi 2: A hardware device designed for neuromorphic computation.
  • Asynchronous neurons applied.
  • Come with a new software platform, Lava.
  • Lava is NOT compatible with other platforms like PyTorch. Conversion needed.
  • Official Site
  • Specification.
  • Pit Holes

spikingjelly

• Link

• Different from what I expected to be the input (a series of 1-D features along the time axis), the input shape of spikingjelly is very much like a normal input shape (N x C x H x W, or even T x N x C x H x W).

• The training strategy is not any different from other Pytorch models.

  optimizer.zero_grad()
  y = net(x)
  loss = criterion(y, label)
  loss.backward()
  optimizer.step()

• To transform a CNN-based model to an SNN model, just remove all the activation layers and replace them with any neurons you like (e.g., LIF).

• An SNN neuron layer, actually, can be seen as an array of neurons that has the same shape as your input. Let’s call it a sub-neuron. Each sub-neuron works separately, and neighboring neurons don’t share anything with them. In the entire training/prediction process, each sub-neuron can be seen as an RNN. The spatial understanding and local feature capture ability are brought by normal CNN layers like Conv2d.

• You can select each neuron’s step mode from single-step to multi-step. The first one doesn’t take care of the time axis and you have to manually manage the behavior along the time axis, and the latter simply can be seen as a wrapper of standard loop-based stepping on a time series.

• SpikingJelly supports both Gradient descent and STDP training scheme. Actually, you can even merge these two training strategies, train neuron layers using STDP and train other CNN layers using GD. Link

• It supports direct conversion to the Intel Loihi platform (Link). You can even train an ANN at first and convert it to SNN with a bunch of patterns. (Link)

Helpful Software/Packages

• AWESOME: Computational Neuro Science

• FRAMEWORK 惊蛰 Spiking Jelly: A PyTorch-based SNN framework.

• LIBRARY Tonic: A tool to facilitate the download, manipulation and loading of event-based/spike-based data. It’s like PyTorch Vision but for neuromorphic data.

• PACKAGE BindsNET is a Python package used for simulating spiking neural networks (SNNs) on CPUs or GPUs using PyTorch Tensor functionality. Similar to Spiking Jelly and need more comparison. This package is developed by Hava’s group, so potentially easier to deploy.

References

Papers

1. Rueckauer B, Lungu I-A, Hu Y, Pfeiffer M and Liu S-C (2017) Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification. Front. Neurosci. 11:682.

Blogs

1. Realamirhe’s Tutorial on ANN&SNN
2. Spiking-Neural-Network: A python implementation of hardware efficient spiking neural network. It contains a good introduction to SNN.
3. 惊蛰 Spiking Jelly: General tutorial is also provided.