There have been a number of papers about the emergence of geometric form inside neural networks over the past few years:
Here is one of the later ones: Reddit - The heart of the internet
1 Like
I wrote this document about ReLU and associative memory:
https://archive.org/details/re-lu-as-a-switch-associative-memory
Then you can have the emergence of geometric forms and geometric factorization in hierarchical associative memory. If you can dig on that.
1 Like
Look at all of the current large language models and look at the whole host of different encoding schemes that are used at the input stage for encoding the text. Then figure out if it makes any sort of difference as to what scheme that was actually used.
Then apply that exact same reasoning to the consideration of model invariant learning of geometric forms, which are effectively just temporally compressed sequences that are backed into via the current approach to learning with backprop. They all learn the same forms.
All of the models are backing into the same patterns, regardless as to the architecture unless it does not work for language.
Where and in what layers parts are learnt will obviously vary but the abstract form will be consistent in the same way the encoding scheme does not really matter for language.
I can’t see the document, the UK have blocked argive.org. We live in 1984…
1 Like
You can view any matrix as an associative memory mapping xᵢ to yᵢ. And you can use some training algorithm like SGV or the Moore-Penrose pseudoinverse,
You can say then that each ReLU decision can be viewed as a one or zero entry in a diagonal matrix. Then a ReLU layer is DW. Where W is the original weight matrix.
Once D is known, D and W can be multiplied together to form a single matrix (associative memory) acting on some input. That is really conditional associative memory even though approximately half of the outputs of DWx are zero. Maybe it helps to consider a readout matrix R giving RDWx as conditional associative memory.
Anyway then a ReLU neural network is RDₙWₙ…D₂W₂D₁W₁. Once the D entries are known then you can do the combining matrix multiplies to get a single matrix (C) mapping y=Cx.
It is not unreasonable to view that as hierarchical associative memory.
That would pessimistically make ReLU neural networks a type of parrot.
However recent papers have shown the emergence of geometric forms in neural networks. And then test time data tends to fall on the same geometric forms giving a correct generalized results.
Hierarchical associative memory allows factorization of those geometric forms giving even better generalization and even reasoning.
Is the human brain also a geometry machine, with hierarchical memory and some training mechanism that allow the emergence of geometric forms? Obviously there will be a lot of inductive biases and priors built in from the biological evolution of form and function.
You could reasonably put an argument that current large neural networks have more geometry and more geometric factorization (layers) than the human brain.
I ran the argument through chatGPT to make the presentation smoother. It didn’t really change anything much: https://archive.org/details/re-lu-neural-networks-as-hierarchical-associative-memory