I have recently been trying to imagine how generally intelligent structural phenomena such as hierarchy, slowly mutating protocols and an affinity towards increased choice can flow out of the raw elements of networks themselves (or mathematical principles of graph theory).
Its got me thinking about how the Numenta model of a neuron differs from the neurons used in NN today.
As I understand it, and I only understand what I have accumulated in passing, that the Neumenta neuron has proximal and distal connections which are treated differently (mainly have different effects on the cell to fire or not). Moreover, the Numenta neuron has essentially 3 states instead of 2. Inactive, active and predictive. If this summary is not accurate please let me know.
( TDLR: )
It occurred to me last night, that the Numenta model could be generalized as an ‘activation weighting over distance function.’ That is, a neuron has three signatures:
- a number of other neurons it listens to,
- a signature (distance ratios) of how many far away neurons vs. nearby neurons it listens to, and
- a signature (distance weights) of how much of an effect those neurons have upon this neuron to fire by distance. (That is, weights indicating how many neurons must fire at the same time at every distance and combination of distances to activate this neuron).
Allow me to explain:
I’m first of all working off the assumption that anything can be modeled by/as a network, which I think we should take as an initial assumption because we use a physical network to understand all that we understand. Thus we actually know that anything we can model can be modeled by/as a network (anything else, we can’t model we have no choice but to ignore).
Any generic network has 2 parts, of course: nodes and edges. In a fully connected network every node listens to every other node. Most networks are not fully connected though and therefore each neuron has some signature (distance ratios) of how far away all the neurons are that it listens to.
Let’s talk about this distance metric. I’ve shown this image on this forum before but I think it does a good job of showing distance:
A is close to C, B, and D but pretty far away from E. Perhaps you ask, “Why? A has a direct connection to all of them, shouldn’t they be equally distant with respect to A?” In a sense, yes, but the distance metric I mean to highlight is ‘interconnectedness.’ If A didn’t have a connection to D, it would mean very little because it can get to it pretty quickly by going through B. However, if A didn’t have a direct connection to E, it would have to make quite the journey to get at the information disseminating from E; there are 6 nodes in between A and E without that direct connection.
So you can see the “distance signature” of any node, meaning how many nodes it connects to that are far away vs how many are nearby is highly affected by how many nodes it connects to, and how many nodes the average node connects to, and furthermore, is highly affected by the “distance signatures” of the specific nodes it connects to.
What does a network with highly variable distance signatures for its nodes end up looking like in the end? A brain with different “cell types.” That is why I wonder if this metric, along with an activation signature by distance is away, or in basic terms perhaps, the way to generalize the Numenta (or brain) model neuron.
I’m suggesting, (and I’d like to know if I’m up in the night or on to something), that with, perhaps only 3 dials you can generate a network of any structure or repeating structure possible. Those dials or parameters being the signature of each cell:
- connection variable: how many other cells does this cell listen to?
- distance signature: what is the ratio of distances of nodes it listens to? (What is it’s affinity to listening to nodes that are distant from each other?)
- weighting signature: by distance how much does an activation count to activate this node?
All of these metrics can be expressed in terms of the metrics each node connects to, they don’t have to be constants. In this way can’t you simply specify a distribution by percentage and determine the shape the network would be?
Can’t you say, “give me a network where x% of nodes have this parameter-signature, and y% have this parameter-signature and z% have this parameter-signature.” and produce a network that attempts to come nearest those signatures and those percentages given the actual data that is provided by the environment? Doesn’t it feel like giving these metrics you define a sphere in the state space of the network’s shape that it attempts to revolve around?
Anyway, I know this is quite the tangential theory but I’m just trying to find a way to generalize the Numenta neuron because I think by doing so you might be able to express the possibility space of the shape of the whole network most generally. (Which in turn would help with automatically generating AGI structures).
What are your thoughts?