2D Object Recognition Project

For me it worked great after a few mistakes i made. Its an really interessting and nice application you’ve made.
I
When you add a lot more grid and normal cells to the model and make more iterations and move a longer path with UDRL movements the anomaly score drops down a lot. The next interesting step could be a temporal pooler, so we could see if there is a stable representation over an specific learned object. I had a look but i cant find any kind of temporal pooler in the htm.core. Does anyone of you guys know?

2 Likes

I am not sure about what you mean by term “temporal pooler”.
As far as i know, in htm.core we have these algorithms:

  • spatial pooler : proximal input activates columns, it basically extracts sematic meaning etc…
  • temporal memory : using active columns and distal input (both internal and external) it determines winner cells & predictive cells

In current state of this experiment we use 1 SP and 1 TM on sensory layer.

i tought that we can realise the “object pool” by just using the spatial pooler, with input connected to sensory layer

4 Likes

This will pool related bits that are separated spatially within a single timestep, but it won’t pool related bits that occur in different timesteps. Since each feature/location is active in different timesteps, I’m not sure you could refer to this as an object pool.

As @weiglt indicated, a true object pool would need to pool representations that are separated temporally as well. I and others are working on an algorithm for this.

There may be options in the NuPIC research code as well, not sure. I recall copying an algorithm from the research code a couple years back, which was a modified SP with activity that followed a logarithmic decay (called a Union Pooler, I believe). It didn’t suit my purposes at the time, but may be worth trying here.

7 Likes

It may be helpful to explain the problem I ran into. In my case, I was trying to pool TM output in the context of a hierarchy. A key feature of hierarchy (my personal belief – don’t know if it is shared by others) is that the more frequently an “object” is encountered, the further down the hierarchy its associated representations should be pushed. This would need to be done in such a way that two “objects” sharing similar sub-components should have an equivalent amount of overlap in their pooled representations.

Thus, the specific behavior I was trying to achieve was that as an “object” (a sequence in my case) is repeatedly encountered, more and more of it would become consolidated into fewer and fewer representations in the pooling layer. The problem I encountered was related to this and the implications that it has when using it to pool TM output.

The reason this algorithm didn’t work for my purposes is because the elements of a sequence always happen in the same order. Thus, aligning the current representation with previous elements in the sequence using logarithmic decay would never stabilize, because earlier representations in the sequence will always end up being reinforced more strongly than later representations. No matter how many times the sequence was repeated, it could never converge onto a single representation.

That may not be an issue for this project, though. In this case, the feature/locations are encountered in different orders depending on how the sensor is moved. Thus it may be possible for convergence onto a single representation given enough movements over the object.

4 Likes

Don’t know if this helps, but when I study a piece for piano, I try to play particularly difficult small two-to-four-measure parts in different ways: I play them faster, slower, in different rythms if possible, sometimes an octave higher or lower even, and on as many different piano’s I can get access to.

Knowing a bit of HTM, I imagine that my brain is making multiple SDR copies of a very similar routine, so that in case of stress (at an exam or while my piano teacher is yelling at me again), my brain has back-up copies to rely on, so that I can keep playing.

I don’t think of it as different hierarchical levels, but more as paralel copies. With voting (however voting works in the brain) this kind of seems logical.

I also wonder how the pathways towards the particular SDRs are chosen (in lack of a better word), and if getting lost in the pathway leads to new copies somehow. (… if that makes any sense).

2 Likes

Can you share where you are working on this? Can we track the progress?
It is supposed to be implemented in htm.core in future?

Yes i think you mean this code, union temporal pooling:

2 Likes

See this project visualized in pictures on this repo HTMpandaVis.

As @weiglt suggested, i have increased number of columns to 5000 (and with the same small simple movements that we have here from the start) the anomaly score drops significantly after 10 steps.
So i guess that this works, but now need to solve following:

  • how to implement object pool - should we use the union pooler? or is here any more usable code?
  • how to implement distal input to location layer (arrow no.4 on picture)

Note: location layer is now just grid cell encoder

2 Likes

Hi @Zbysekz

Is this project still alive?

I would love to continue this line. But getting some issues when getting started.
Here is my fork in case someone can shed some light.

When launching getting this issue when launching : python main.py

Iteration:0
Sensor at 3, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:1
Sensor at 3, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:2
Sensor at 4, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:3
Sensor at 5, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:4
Sensor at 6, 4:
Feature UP: 0
SDR( 700 ) 42, 65, 71, 73, 226, 256, 408, 491, 493, 570, 633, 652, 668
Iteration:5
Sensor at 7, 4:
Feature UP: 0
SDR( 700 ) 42, 65, 71, 73, 226, 256, 408, 491, 493, 570, 633, 652, 668

Could someone tell if this data is coherent? Is it correct? Is just to know if I managed to setup the env correctly?

Also had a strange issue when installing the htm.core version thorugh th epipenv:

Adding htm.core to Pipfile’s [packages]…
:heavy_check_mark: Installation Succeeded
Pipfile.lock (b8e016) out of date, updating to (819f84)…
Locking [dev-packages] dependencies…
:heavy_check_mark: Success!
Locking [packages] dependencies…
✘ Locking Failed!
[pipenv.exceptions.ResolutionFailure]: File “/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/resolver.py”, line 69, in resolve
[pipenv.exceptions.ResolutionFailure]: req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]: File “/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.py”, line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]: req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]: File “/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.py”, line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]: resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]: File “/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.py”, line 395, in resolve
[pipenv.exceptions.ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches htm-core[examples]
[pipenv.exceptions.ResolutionFailure]: No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with pipenv lock --clear, then try the original command again. Alternatively, you can use pipenv install --skip-lock to bypass this mechanism, then run pipenv graph to inspect the situation. Hint: try pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches htm-core[examples]
No versions found
Was https://pypi.org/simple reachable?
[pipenv.exceptions.ResolutionFailure]: req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]: File “/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.py”, line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]: req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]: File “/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.py”, line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]: resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]: File “/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.py”, line 395, in resolve
[pipenv.exceptions.ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches htm-core[examples]
[pipenv.exceptions.ResolutionFailure]: No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with pipenv lock --clear, then try the original command again. Alternatively, you can use pipenv install --skip-lock to bypass this mechanism, then run pipenv graph to inspect the situation. Hint: try pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches htm-core[examples]
No versions found
Was https://pypi.org/simple reachable?

Is this normal? Beciase Afterwards the main.py work, but, its a strange issue as far as I can see.

I would like to have some insiight about the results to know if they are correct and how to interpret them so I can build appon this. Thanks

1 Like

Hello @duckfrost,
yes, i would say the project is alive, just it is little bit waiting for answers for the “hard” questions that i posted before:

  • how to implement object pool - should we use the union pooler? or is here any more usable code?
  • how to implement distal input to location layer (arrow no.4 on picture)

I am putting the little time that i can spare now for improving the HTMpanda Vis and this is the main testable code that i am using for it.

Any help on any frontier is appreciated.

When launching getting this issue when launching : python main.py

What issue do you mean? The log you posted seems to be not the latest master.
I guess this is just iteration, sensor position and SDR representing the encoded sensor feature.

Also note: I made some major updates related to calculation order (when you sense feature and when you make prediction and for what data the prediction is made) This was not correct since now. This is in the branch pandaVisIntegration and not yet in the master (i will merge it soon)

About the htm.core installation, post it as github issue here https://github.com/htm-community/htm.core
But for the first look, you should be using python3.6 and greater, but in the log appears python2.7, which shouldn’t.
I will make a note in the readme that python3.6< must be used in this project.

Great! I just want to maybe create a very understandable, well documented example. That way I get to know the newest elements of HTM better and we also make a quickstart/great place to start with HTM’s. So any help in getting to your level of understanding of the code and the HTM structure is appreciates. I worked with the old version of NUPIC years ago and the theory has changed significantly.

The “issue” I just ment that the main when executed, it appears this image:

And to continue each iteration have to close the window to make appear the next one. I dont know ifthis is the behaviour that the example has to have. Though it was a live, dynamic and interactive demo moving with the keyboard around an environment. Knowing this is important to understand better the code and to know where I can work on ( very interested in creating a HTM learning 2D/3D environment ).

I’m using the pipenv which is using python 3.7, and installed the htm.core like this:

[[source]]
name = “pypi”
url = “https://pypi.org/simple
verify_ssl = true

[dev-packages]
pytest = “"
flake8 = "

black = “*”

[packages]
PyYAML = “"
numpy = "

matplotlib = “"
htm-core = {index = “https://test.pypi.org/simple/”,extras = [“examples”],version = "
”}

[requires]
python_version = “3.7”

[pipenv]
allow_prereleases = true

I’m trying to first have something that anyone can just setup the env and start working also.

As for the big questions, once I land and understand better the code and the current HTM structure, thats just a matter of spendinng time on it, which I will be delighted to ;).

And the Panda visualization how canI set that up also to use it?

Lots of questions ;).

Ok fine :+1:
You don’t have the latest version, in the latest these maps should change each 1 sec for example (although tested only on linux in interactive console & separately). It should look like the green-black maps i posted in this thread before.

The 3D environment you talk about, is the HTMpandaVis. You must run panda and this script simultaneously, they connect through TCP socket together.
See readme, there should be everything you may need.

1 Like

Hi,

I finally get into the phase about checking out the grid cell implementation.
Thank you for sharing me this amazing project link (totally impressed about this brainstorm and how good this community is).
But I got some questions too.
I saw from previous post, you implemented the encode of the position -> SDR
Is the position represented in grid cell module?
How did you implement the grid cell module (from coordinates -> grid cell module)?
Can you explain what is the algorithm of the encoder?
I saw that the movement is using GCM, can you elaborate the mechanism behind that?
Also, in the repo, I cloned it and may I ask what I should expect to see or to do about this repo?
Is this project completed?

Thank you very much.

You might want to read the location paper from front to back. And the HTM school video is a good complimentary

I saw from previous post, you implemented the encode of the position -> SDR
Is the position represented in grid cell module?
How did you implement the grid cell module (from coordinates -> grid cell module)?
Can you explain what is the algorithm of the encoder?

I’ve just simply implemented the @dmac work from this PR, he created grid cell encoder


Here is the main source code of the encoder, it’s python and has quite some comments in it:

I saw that the movement is using GCM, can you elaborate the mechanism behind that?
Also, in the repo, I cloned it and may I ask what I should expect to see or to do about this repo?

In the readme is stated, that you can just run python main.py in the htm-2d-object-modeling/python/
The simulation should start, and you should see plots.
The position of agent is encoded with this encoder into SDR which is used as distal input to the SensoryLayer.

Is this project completed?

No it’s far from it’s finish, see status from the post above

I’ve been trying to grow my knowledge a bit around spiking neural networks and how they work. In the process of that, I ran across one method that was used for encoding MNIST (handwriting) samples which I think would be perfectly useable within HTM as well; we could simply split the color channels (BGR), threshold them, flatten into 1d array, then randomize in the same way the Nengo framework is doing. Afterward, concatenate the encodings from each channel and send the binary pattern into HTM.

https://www.nengo.ai/nengo-dl/examples/lmu.html

Thought I’d leave this here, in case it may be useful in your project.

EDIT:
As a side note, if folks are doing this in Python, consistently distributing the bits of an encoded image using a seed, note the answers in this link here:

1 Like

Progress update:

  • Temporal memory was replaced with ApicalTiebreakTM from ported numenta research advanced code and it works well. Now with context going only from location layer, the sensory layer learns sensations@locations independent on the order of sensations.

The open questions:

  • how to implement object pool - column pooler from advanced code? or ascii_stability ?
  • how to implement location layer - ThresholdedGaussian2DLocationModule from advanced code? CAN network?

Please any thoughts or links are welcome!

3 Likes

Object pool … ascii_stability

I wrote that, and IIRC I never got distal dendrites to work on that model.

1 Like

Bringing some update after long delay.

I have used the code from “advanced” section of htm.core (ported from htmresearch), using network API.

Created this Network API scheme:

System firstly learned 5 objects like “cup”, “palmpilot” and “boat”. Then it was able successfully classify them.
a

In this gif, one can see “narowing” of the unions in L6.

And here are some plots: (inferring starts at iteration 156)


10 Likes

:-D.

5 Likes

Hello can you please inform me that how you are obtaining the out put?

Do L4 & L2 both hav = Spatial Poller + Temporal Memory ?

From L2 where I have to send apical data input to L4 . To a SP or TM .
How feedback input will work in L4. What’s it role.

Actually I thought in case of Multilayer its working like that

Sensory Input -> encoder -> bitArray -->L4 SP → SDR of Active coloums -->L4 TM → SDR of Active Cells → :L2 SP–> SDR of Active coloums --:> L2 TM -->SDR of Active Cells → Classifier–> Output

But I haven’t realized how your network is working. How it performs output. And the role of Apical Feed back from L2 to L4.

I will be grateful if you can explain it for me.
Thank you

2 Likes