Trying to get this demo running in my local setup. Any virtual env setup script you might have to make this process? Because I run into compilation and instalation issues : https://github.com/htm-community/htm.core/issues/702 , https://github.com/htm-community/htm-2d-object-modeling/issues/10.
Any help would be appreciated. I would love to contribute and develop for this project, but have to get the setup running and get to understand all the elements in play. Thanks a lot in advance
Great! 
i think there is a lot to do, donât hesitate to ask if you are not sure about something. I am now currently working on the vis tool, with the time that i am able to reserve for that
For me it worked great after a few mistakes i made. Its an really interessting and nice application youâve made.
I
When you add a lot more grid and normal cells to the model and make more iterations and move a longer path with UDRL movements the anomaly score drops down a lot. The next interesting step could be a temporal pooler, so we could see if there is a stable representation over an specific learned object. I had a look but i cant find any kind of temporal pooler in the htm.core. Does anyone of you guys know?
I am not sure about what you mean by term âtemporal poolerâ.
As far as i know, in htm.core we have these algorithms:
In current state of this experiment we use 1 SP and 1 TM on sensory layer.
i tought that we can realise the âobject poolâ by just using the spatial pooler, with input connected to sensory layer
This will pool related bits that are separated spatially within a single timestep, but it wonât pool related bits that occur in different timesteps. Since each feature/location is active in different timesteps, Iâm not sure you could refer to this as an object pool.
As @weiglt indicated, a true object pool would need to pool representations that are separated temporally as well. I and others are working on an algorithm for this.
There may be options in the NuPIC research code as well, not sure. I recall copying an algorithm from the research code a couple years back, which was a modified SP with activity that followed a logarithmic decay (called a Union Pooler, I believe). It didnât suit my purposes at the time, but may be worth trying here.
It may be helpful to explain the problem I ran into. In my case, I was trying to pool TM output in the context of a hierarchy. A key feature of hierarchy (my personal belief â donât know if it is shared by others) is that the more frequently an âobjectâ is encountered, the further down the hierarchy its associated representations should be pushed. This would need to be done in such a way that two âobjectsâ sharing similar sub-components should have an equivalent amount of overlap in their pooled representations.
Thus, the specific behavior I was trying to achieve was that as an âobjectâ (a sequence in my case) is repeatedly encountered, more and more of it would become consolidated into fewer and fewer representations in the pooling layer. The problem I encountered was related to this and the implications that it has when using it to pool TM output.
The reason this algorithm didnât work for my purposes is because the elements of a sequence always happen in the same order. Thus, aligning the current representation with previous elements in the sequence using logarithmic decay would never stabilize, because earlier representations in the sequence will always end up being reinforced more strongly than later representations. No matter how many times the sequence was repeated, it could never converge onto a single representation.
That may not be an issue for this project, though. In this case, the feature/locations are encountered in different orders depending on how the sensor is moved. Thus it may be possible for convergence onto a single representation given enough movements over the object.
Donât know if this helps, but when I study a piece for piano, I try to play particularly difficult small two-to-four-measure parts in different ways: I play them faster, slower, in different rythms if possible, sometimes an octave higher or lower even, and on as many different pianoâs I can get access to.
Knowing a bit of HTM, I imagine that my brain is making multiple SDR copies of a very similar routine, so that in case of stress (at an exam or while my piano teacher is yelling at me again), my brain has back-up copies to rely on, so that I can keep playing.
I donât think of it as different hierarchical levels, but more as paralel copies. With voting (however voting works in the brain) this kind of seems logical.
I also wonder how the pathways towards the particular SDRs are chosen (in lack of a better word), and if getting lost in the pathway leads to new copies somehow. (⌠if that makes any sense).
Can you share where you are working on this? Can we track the progress?
It is supposed to be implemented in htm.core in future?
Yes i think you mean this code, union temporal pooling:
See this project visualized in pictures on this repo HTMpandaVis.
As @weiglt suggested, i have increased number of columns to 5000 (and with the same small simple movements that we have here from the start) the anomaly score drops significantly after 10 steps.
So i guess that this works, but now need to solve following:
Note: location layer is now just grid cell encoder
Hi @Zbysekz
Is this project still alive?
I would love to continue this line. But getting some issues when getting started.
Here is my fork in case someone can shed some light.
When launching getting this issue when launching : python main.py
Iteration:0
Sensor at 3, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:1
Sensor at 3, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:2
Sensor at 4, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:3
Sensor at 5, 4:
Feature UP: 1
SDR( 700 ) 21, 25, 63, 231, 234, 258, 264, 293, 343, 367, 454, 464, 621, 668
Iteration:4
Sensor at 6, 4:
Feature UP: 0
SDR( 700 ) 42, 65, 71, 73, 226, 256, 408, 491, 493, 570, 633, 652, 668
Iteration:5
Sensor at 7, 4:
Feature UP: 0
SDR( 700 ) 42, 65, 71, 73, 226, 256, 408, 491, 493, 570, 633, 652, 668
Could someone tell if this data is coherent? Is it correct? Is just to know if I managed to setup the env correctly?
Also had a strange issue when installing the htm.core version thorugh th epipenv:
Adding htm.core to Pipfileâs [packages]âŚ
Installation Succeeded
Pipfile.lock (b8e016) out of date, updating to (819f84)âŚ
Locking [dev-packages] dependenciesâŚ
Locking [packages] dependenciesâŚ
â Locking Failed!
[pipenv.exceptions.ResolutionFailure]: File â/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/resolver.pyâ, line 69, in resolve
[pipenv.exceptions.ResolutionFailure]: req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]: File â/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.pyâ, line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]: req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]: File â/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.pyâ, line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]: resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]: File â/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.pyâ, line 395, in resolve
[pipenv.exceptions.ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches htm-core[examples]
[pipenv.exceptions.ResolutionFailure]: No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with pipenv lock --clear, then try the original command again. Alternatively, you can use pipenv install --skip-lock to bypass this mechanism, then run pipenv graph to inspect the situation. Hint: try pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches htm-core[examples]
No versions found
Was Simple index reachable?
[pipenv.exceptions.ResolutionFailure]: req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]: File â/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.pyâ, line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]: req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]: File â/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.pyâ, line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]: resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]: File â/home/rdaneel/.local/lib/python2.7/site-packages/pipenv/utils.pyâ, line 395, in resolve
[pipenv.exceptions.ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches htm-core[examples]
[pipenv.exceptions.ResolutionFailure]: No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with pipenv lock --clear, then try the original command again. Alternatively, you can use pipenv install --skip-lock to bypass this mechanism, then run pipenv graph to inspect the situation. Hint: try pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches htm-core[examples]
No versions found
Was Simple index reachable?
Is this normal? Beciase Afterwards the main.py work, but, its a strange issue as far as I can see.
I would like to have some insiight about the results to know if they are correct and how to interpret them so I can build appon this. Thanks
Hello @duckfrost,
yes, i would say the project is alive, just it is little bit waiting for answers for the âhardâ questions that i posted before:
- how to implement object pool - should we use the union pooler? or is here any more usable code?
- how to implement distal input to location layer (arrow no.4 on picture)
I am putting the little time that i can spare now for improving the HTMpanda Vis and this is the main testable code that i am using for it.
Any help on any frontier is appreciated.
When launching getting this issue when launching : python main.py
What issue do you mean? The log you posted seems to be not the latest master.
I guess this is just iteration, sensor position and SDR representing the encoded sensor feature.
Also note: I made some major updates related to calculation order (when you sense feature and when you make prediction and for what data the prediction is made) This was not correct since now. This is in the branch pandaVisIntegration and not yet in the master (i will merge it soon)
About the htm.core installation, post it as github issue here GitHub - htm-community/htm.core: Actively developed Hierarchical Temporal Memory (HTM) community fork (continuation) of NuPIC. Implementation for C++ and Python
But for the first look, you should be using python3.6 and greater, but in the log appears python2.7, which shouldnât.
I will make a note in the readme that python3.6< must be used in this project.
Great! I just want to maybe create a very understandable, well documented example. That way I get to know the newest elements of HTM better and we also make a quickstart/great place to start with HTMâs. So any help in getting to your level of understanding of the code and the HTM structure is appreciates. I worked with the old version of NUPIC years ago and the theory has changed significantly.
The âissueâ I just ment that the main when executed, it appears this image:
And to continue each iteration have to close the window to make appear the next one. I dont know ifthis is the behaviour that the example has to have. Though it was a live, dynamic and interactive demo moving with the keyboard around an environment. Knowing this is important to understand better the code and to know where I can work on ( very interested in creating a HTM learning 2D/3D environment ).
Iâm using the pipenv which is using python 3.7, and installed the htm.core like this:
[[source]]
name = âpypiâ
url = âSimple indexâ
verify_ssl = true[dev-packages]
pytest = â"
flake8 = "â
black = â*â[packages]
PyYAML = â"
numpy = "â
matplotlib = â"
htm-core = {index = âSimple indexâ,extras = [âexamplesâ],version = "â}[requires]
python_version = â3.7â[pipenv]
allow_prereleases = true
Iâm trying to first have something that anyone can just setup the env and start working also.
As for the big questions, once I land and understand better the code and the current HTM structure, thats just a matter of spendinng time on it, which I will be delighted to ;).
And the Panda visualization how canI set that up also to use it?
Lots of questions ;).
Ok fine 
You donât have the latest version, in the latest these maps should change each 1 sec for example (although tested only on linux in interactive console & separately). It should look like the green-black maps i posted in this thread before.
The 3D environment you talk about, is the HTMpandaVis. You must run panda and this script simultaneously, they connect through TCP socket together.
See readme, there should be everything you may need.
Hi,
I finally get into the phase about checking out the grid cell implementation.
Thank you for sharing me this amazing project link (totally impressed about this brainstorm and how good this community is).
But I got some questions too.
I saw from previous post, you implemented the encode of the position -> SDR
Is the position represented in grid cell module?
How did you implement the grid cell module (from coordinates -> grid cell module)?
Can you explain what is the algorithm of the encoder?
I saw that the movement is using GCM, can you elaborate the mechanism behind that?
Also, in the repo, I cloned it and may I ask what I should expect to see or to do about this repo?
Is this project completed?
Thank you very much.
You might want to read the location paper from front to back. And the HTM school video is a good complimentary
I saw from previous post, you implemented the encode of the position â SDR
Is the position represented in grid cell module?
How did you implement the grid cell module (from coordinates â grid cell module)?
Can you explain what is the algorithm of the encoder?
Iâve just simply implemented the @dmac work from this PR, he created grid cell encoder
Here is the main source code of the encoder, itâs python and has quite some comments in it:
I saw that the movement is using GCM, can you elaborate the mechanism behind that?
Also, in the repo, I cloned it and may I ask what I should expect to see or to do about this repo?
In the readme is stated, that you can just run python main.py in the htm-2d-object-modeling/python/
The simulation should start, and you should see plots.
The position of agent is encoded with this encoder into SDR which is used as distal input to the SensoryLayer.
Is this project completed?
No itâs far from itâs finish, see status from the post above
Iâve been trying to grow my knowledge a bit around spiking neural networks and how they work. In the process of that, I ran across one method that was used for encoding MNIST (handwriting) samples which I think would be perfectly useable within HTM as well; we could simply split the color channels (BGR), threshold them, flatten into 1d array, then randomize in the same way the Nengo framework is doing. Afterward, concatenate the encodings from each channel and send the binary pattern into HTM.
https://www.nengo.ai/nengo-dl/examples/lmu.html
Thought Iâd leave this here, in case it may be useful in your project.
EDIT:
As a side note, if folks are doing this in Python, consistently distributing the bits of an encoded image using a seed, note the answers in this link here:
Progress update:
The open questions:
Please any thoughts or links are welcome!
Object pool ⌠ascii_stability
I wrote that, and IIRC I never got distal dendrites to work on that model.
Bringing some update after long delay.
I have used the code from âadvancedâ section of htm.core (ported from htmresearch), using network API.
Created this Network API scheme:
System firstly learned 5 objects like âcupâ, âpalmpilotâ and âboatâ. Then it was able successfully classify them.
In this gif, one can see ânarowingâ of the unions in L6.
And here are some plots: (inferring starts at iteration 156)
