Compositionality: Numenta's new framework vs actual deep learning models

Community Lounge
1

Speaking about compositionality, the advances brought by the new framework of Numenta could be really significant, in regard to what deep learning experts are saying.

On one side, the new paper from Numenta “A Framework for Intelligence and Cortical Function Based on Grid Cells in the Neocortex” proposes a theory that could explain the compositionality of objects (and compositionality of langage by extension).

Numenta - October 2018
_https://numenta.com/neuroscience-research/research-publications/papers/a-framework-for-intelligence-and-cortical-function-based-on-grid-cells-in-the-neocortex_

We propose an extension of grid cells, called “displacement cells”. Displacement cells enable us to learn how objects are composed of other objects, also known as object compositionality

On the deep learning side, the famous Yoshua Bengio has just published (as a co-author) a paper underlying the difficulty of learning the compositional properties in langage with deep learning methods :

_Chevalier-Boisvert, Bahdanau, Bengio - October 2018 - http://export.arxiv.org/pdf/1810.08272_

We put forward strong evidence that current deep learning methods are
not yet sufficiently sample efficient when it comes to learning a language with
compositional properties

Are you aware of other AI models that integrate compositionality in their approach ?

I’m really curious to see how those approaches will evolve !

1 Like
2

Yes, this is a hot topic right now. Google guys are publishing papers on “graph networks”, essentially combining relational reasoning with deep learning primitives, see here: https://arxiv.org/abs/1806.01261

I think there is a general consensus out there that some kind of knowledge representation or compositionality is definitely a good thing.

As a simple proof-of-concept I would love to see some kind of knowledge graph encoded as SDR and then HTM or Hebbian learning based system applied.

2 Likes
3

Thanks acatovic. I didn’t know about graph networks.

From the Deepmind paper “Relational inductive biais, deep learning and graph networks” :

  • Research on Graph Networks has been ongoing since 2005
  • Deepmind propose a graph network framework to unify existing approaches based on graphs and to provide a straightforward interface for assembling graph networks into sophisticated architectures
  • Graph Networks perform operations on structured representations with an invariance on node/edge permutations
  • Structured representations can be abstracted as tuples (entities as nodes, relations as edges, global attributes)
  • Entities (nodes) can also be a structured representation, allowing compositionality with hierarchy
  • It is still unclear how to convert sensory data into more structured representations like graphs (needed to apply Graph Networks operations)

Concerning HTM, I don’t understand how structured representations could be encoded.

In the stapler example, there are 2 entities and 1 relation.

Can we say the following ?

  • The allocentric location of each entity is represented in L6 grid cells
  • Each entity has a stable representation in L2
  • The relation between those 2 entities is believed to be represented by the displacement cells in L5

But the SDR of L2 cells is an overlapping of the SDR of the 2 entities of the stapler.
How displacement cells knows to which entities they are linked if the SDR of L2 cells consists in the union of more than 3 entities ?
I feel that SDR are good for modelling “sets” (via union) but not “graphs”.

More generally, how do object representations interact with each other in a single column (that can represent thousands of objects !)

Any thought ?

2 Likes
4

In HTM theory, we can represent different objects simultaneously by adding their SDRs. This is a union of SDRs representing a set of objects.

But how a SDR could represent a graph of objects, meaning that the relations between the objects are not necessary symetrical ?

Here are some reflexions in progress (which also help me to consolidate my understanding of the HTM theory):

1/ One way would be to model the graph as a sequence (with the minicolumns & predictive cells of the HTM theory). It would only work for acyclic and relatively short graphs. This should not be an issue for most composional objects.

2/ A second way could be to have three stable layers:

  • one layer for the set of objects (L2 ?)
  • one layer for the set of relations between objects (deduced from displacement cells in L5 ?)
  • one layer for mapping the two previous ones (?)

3/ Another way could be to model only one object per column, and then rely on the hierarchy. But this in not in line with the theory that each column can model thousand of objects.

Does it make sense or am I misinterprating the theory ?

5

I understand the theoretical manipulation of SDRs - taking this set of bits and doing various operations with that set of bit.

If you can explain this then there should be a path to answering the graph question

How is that actually accomplished in the biology?
Or is it?

In real brains we know that objects are manipulated because we can observe the macro behavior of living being with brains. Does that really reduce to this pattern of activation somehow being compared to other sets of SDRs in a section of cortex. If so - how? I feel like I know a bit about neurology and I can’t really imagine how this is happening.

I feel like this is the about the same as Deep Learning - SDR math captures in interesting property of the brain and extends it into a useful gadget (See Cortical IO for example) - but it really is not how the brain does things.