HTM Based Autonomous Agent

[sunguralikaan]

I have been travelling the last couple of days so I couldn’t respond quickly.

@kaikun

Agreed. But your motor-layer where you choose the 3 winner cells from had the same size as the other layers (512).

The motor layer consists of only 30 neurons and no columns. It is as shown in page 28. There are 4 rows just to visualize different states of the same neurons. There is actually a single row of neurons.

So the motor layer can be treated as 30*1 (30 columns and 1 neurons per column). The functionality can be realized by a full layer but I just simplified it this way.

@kaikun

Do you believe it hurts the architecture much if the apical connections from layer 5 voluntary active neurons to motor neurons to excite/inhibit are not learned but constantly mapped instead?

I thought of this myself too at some point. It works but there is a catch. What happens when the layer 5 activation changes slightly because of a new pattern or boosting? In this case, there will not be a motor command mapped to the slightly changed layer 5 activation for whatever reason. It can work if the layers are highly stable but you basically remove noise robustness and any change requires a new mapping which kind of contradicts with what HTM does.

@kaikun

The main confusion is caused by the fact that we sample in D1/D2 using a Spatial Pooler from layer 5. This means we will have a complete different columnar activation than in layer 5 as the connections are randomly initialized. Then even though the neural activation is still the prediction for layer 5 the indices can not really be mapped to layer 5 as they are completely different distributed. Thus it seems not systematic to me that basal and apical depolarizations aim to the same.

I think I understand your confusion.

Each layer 5 activation corresponds to different activations on D1 and D2. Suppose that at time t, L5 has activation L5(t), D1 has activation D1(t) and D2 has activation D2(t). Activation D1(t) and D2(t) takes activation L5(t) as their input. Therefore D1(t), D2(t) and L5(t) all have differing activations. However, there is a relation; both D1(t) and D2(t) occurs when they get L5(t) as their input so they encode activation L5(t) on their own unique way. As the time goes on, D1 and D2 learns all layer 5 activations. On the temporal memory side, D1(t) and D2(t) takes their distal input from L5(t-1).

@kaikun

Then even though the neural activation is still the prediction for layer 5 the indices can not really be mapped to layer 5 as they are completely different distributed.

This is like motor layer association with layer 5 with a single difference. For motor layer you associate L5(t) with Motor(t). In this case, you associate D1(t-1) with L5(t) through apical connections instead of D1(t). Same with D2.

So, there are also apical connections forming to L5(t) from D1(t-1) and D2(t-1). So any activation occurring in D1 or D2, depolarizes cells in L5 that are expected to be active in the next time. What you end up is at any given time there are predictive cells in L5(t) that are distally depolarized by the activation from L5(t-1) and apically depolarized by the activation from D1(t-1) and D2(t-1).

You can achieve the same thing by directly using the same columnar activation of L5 on D1 and D2, however that does not seem to be how biology does as one activation is in the cortex and one is in the striatum. These should be mapped but not the same.

@kaikun

I also have the problem currently that the mouse movement does not have enough effect on the sensory encoding and layer 4 neural activation to make a difference.

I believe this is a general problem also when the task would be more complex and detailed that individual pixels become more important but get vanished by the Spatial Pooler. In general some “Attention” mechanism could be tried here to give a certain image region more importance in the encoding (or using saccade and a limited perceptual field as you had).

This is a VERY crucial problem. I had this sort of a problem from the beginning. Currently, you can only get around this by redesigning your encoders. This is a research area on its own. In my case, I was interested on the parts of the image that changed so I tried implementing an event based visual sensor here. This allowed the agent to sense what actually changed (what matters). Maybe you can come up with an encoder that ‘magnifies’ what you need until we can have a crack at the attention problem.

@kaikun

In my case I could encode the mouse coordinates next to the image visuals and then combine the input at layer 4 sensory input, sizing them on the importance for the feature. Or I increase the layer size and sparsity enough to encode the details in the activation. Or do it specifically for the pixels around the mouse (attention like)

Any of these can be a starting point. You will probably understand what is important about the input after some trials. I really spent weeks trying to come up with something kind of universal to “zoom” on to the important bits in the input. However, biology has very sophisticated tools tailored just this task such as the retina and thalamus. If you are interested in how the eye does it, you can read on neuromorphic vision sensors but then again, this is another area of research.