Paper on combining "cortical" WTA with transformers

Science Numenta Theory
1

I guess this is similar to spatial pooling in HTM?

“Winner-take-all” is a biological mechanism that occurs when one or a few neurons within a set (i.e., the one/ones with the highest activation level) influence the outcome of a computation. The more active neurons essentially suppress the activity of other neurons, becoming the only cells contributing to a specific decision or computation.

Iqbal and his colleagues tried to realistically mimic this biological computation using neuromorphic hardware and then use it to improve the performance of well-established machine learning models. To do this, they used IBM’s TrueNorth neuromorphic hardware chip, which is especially designed to mimic the brain’s organization.

A new approach that realistically mimics neocortex computations using AI

Capturing biologically-grounded WTA computations from the brain (A) to design neural state machines (B) and translating them onto IBM’s TrueNorth neuromorphic hardware chip (C) as well as implementing the WTA as a neural layer (D) in AI architectures. Credit: Iqbal et al.

“Our biophysical network model aims to capture the key features of neocortical circuits, focusing on the interactions between excitatory neurons and four major types of inhibitory neurons,” explained Iqbal.

“The model incorporates experimentally measured properties of these neurons and their connections in the visual cortex. Its key feature is the ability to implement ‘soft winner-take-all’ computations, where the strongest inputs are amplified while weaker ones are suppressed.”

By performing these brain-inspired computations, the team’s approach can enhance important signals, while filtering out noise. The key advantage of their NeuroAI system is that it introduces a new biologically-grounded and yet computationally efficient approach to processing visual information, which could help to improve the performance of AI models.

“One of our most exciting achievements was the successful implementation of our brain-inspired computations on IBM’s TrueNorth neuromorphic chip,” said Iqbal.

“This demonstrates that we can translate principles from neuroscience to real hardware. We were also thrilled to see significant improvements in the performance of Vision Transformers and other deep learning models when we incorporated our winner-take-all inspired processing. For example, the models became much better at generalizing to new types of data they hadn’t been trained on—a key challenge in AI.”

Iqbal and his colleagues combined the soft winner takes all computations performed using their approach with a vision transformer-based model. They found that their approach significantly improved the model’s performance on a digital classification task for completely “unseen” data through zero-shot learning.

3 Likes
2

Thank you for your analysis, @bkaz.

I didn’t realize there was a classification of inhibitory neurons. It seems that they take inhibition quite a few steps further than the way it is used in HTM.

2 Likes
3

Glad you liked it. Yes, those interneurons are very interesting, and definitely not fully understood.

Lateral inhibition is generally interpreted as a suppresion of redundancy, and I think that’s correct in a temporal sense. But in terms of more durable and important spatial representations, it seems to be a way to represent a connectivity-based cluster via single exemplar: the neuron with the highest activation, or the earliest-firing one. That requires establishing the cluster first: the laterally branching axons through which co-activated neurons inhibit each other. Such network must be established through mutual reinforcement, not inhibition.

So it should be a two-step process: first co-activation must form the cluster / clique through laterally-mediated Hebbian learning. That cluster is not really redundant, the neurons represent spatially distinct inputs, but it can have a compressed higher-level representation through exclusive activation of its exemplar.
That’s only after the cluster / clique / ensemble is established, so it seems that we need alternating phases of lateral reinforcement and lateral inhibition, through the same branching axons, probably in layer II-III. Not sure if this interpretation is novel?

2 Likes
4

From Claude, not really sure:

Let me analyze how different interneuron types could potentially orchestrate this alternating process:

  1. Fast-spiking PV+ basket cells:
  • Could be key in the inhibitory phase, creating the “winner-take-all” effect
  • Their fast dynamics and strong perisomatic inhibition make them ideal for rapid selection of exemplars
  • Their widespread connections to local pyramidal cells support their role in mediating competition within a cluster
  1. SOM+ Martinotti cells:
  • Targeting distal dendrites of pyramidal cells
  • Might be more involved in the reinforcement phase by:
    • Providing disinhibition of local circuits when activated by strong, consistent input
    • Their frequency-dependent facilitation could help sustain activity during cluster formation
    • The slower dynamics could help maintain stability during learning
  1. VIP+ interneurons:
  • Could act as a switch between phases by:
    • Inhibiting SOM+ cells during the inhibitory phase to allow PV-mediated competition
    • Being suppressed during the reinforcement phase to allow SOM-mediated facilitation
  • Their sensitivity to top-down inputs could help coordinate this alternation based on behavioral state
  1. CCK+ basket cells:
  • Their sensitivity to endocannabinoids and slower dynamics might help:
    • Regulate the overall excitability during the reinforcement phase
    • Fine-tune the balance between excitation and inhibition within forming clusters
2 Likes