Hello again, I’m back with more questions related to coding my own HTM implementation in pure Python3. I’ve gotten the spatial pooling algorithm part done at this point, but I’m having some issues with optimization. I’m hoping someone can help me figure out how best way to handle the sparse distribution of synapses during the update and overlap phase. I’m currently using numpy arrays for everything, and I’ve tried 2 different methods so far:
I initially tried using np.random.choice() with a static seed for each minicolumn (where the seed is the index of the minicolumn) on the input array to extract an input array for each individual minicolumn. This seems to be very memory-intensive and CPU-intensive due to the np.random.choice() method. I considered using a mapping array to map only certain indices of the input array to each minicolumn, but that still seems like it would be very slow.
I switched to using a binary mask that gets generated for each neuron that gets applied to the permanence matrix after updating each time. This is much faster than the above method, but it seems to be completely counter-productive to the idea of sparse connections in the first place - to reduce computation cost. With this method, I have connected, active, input, and permanence arrays for each neuron that are ALL the same size as the initial input array, so it eats tons of memory.
So how do I handle this issue? Am I thinking about this the wrong way? I tried looking at the Nupic code, but I can’t make any sense of it. Is it better to split up the input array for each minicolumn? If so, what is the fastest method for doing this through each iteration?
We’re both working on the same problem! The thing I’m planning to try is using scipy-based sparse array objects. I dont know much about them yet, but I think they’re more memory efficient and can still be operated as if they’re numpy arrays.
Edit: I’ve heard that they’re efficient and act like np arrays. I haven’t even installed scipy yet so anything I say about it is basically hearsay
Another option: It may actually be worth considering writing your own implementation of a sparse array object. Even with poor optimization, a simple nonzero-index-list object might even be faster than regular arrays, given the extreme sparsity…maybe?
Yeah I’ve come up with a couple of theories for solutions on my own, but I was hoping someone here could help me understand how it’s implemented in the existing framework. I’m assuming this part of the processing is being done on the lower level C stuff and isn’t really feasible for a strict Python implementation. I really hope that isn’t the case, as I am trying to avoid having to learn an entirely new programming language just for this.
All that said, I’d suggest (for the purpose of building your own HTM) you skip the SP at first. Just get an encoder working (like simple scalar or categorical encoder) and feed those vectors right into TM. Getting a working TM is more critical for a usable HTM system. For example with categorical data, using an SP should make little if any difference to system performance over encoder vectors – since there’s no overlap between categories.
I’d setup some categorical test sequences with some categories showing up in different contexts – like the classic “A,B,C,D,X,B,C,Y”. This will test the system’s ability to learn the context and come to predict the values precisely. I’d feed this sequence in on repeat and track both the anomaly scores and the number of predicted values. What should happen is the anomaly scores start at 1 & settle down to 0, and the number of predictions start at 0 & settle down to 1, while hitting 2 in between. This should be when the system sees “C” and predicts both “D” and “Y”, before it has learned enough context to choose just 1.
Your post was extremely helpful, thank you. I think the key information I was looking for is in the first code block you posted. It looks like this implementation is using the same masking method I was talking about earlier, but just using a single matrix for all connections instead of storing multiple matrices. I have yet to refactor my code down from all-object-oriented (I broke everything out into neurons, minicolumns, and layer types for visualization purposes initially). Now that I know that’s the method being used, I’ll give that a shot.
As for focusing on TM, I’m not really ready to think about that part just yet. I’m learning as much about coding as I am about HTM through this exercise so I want to keep the complexity to a minimum while still being able to understand everything from the bottom up. My python code looks strikingly similar to the pseudocode in BAMI, which is probably part of my problem. Once I have a better understanding of how to use these tools like matrices in Python, then I can tackle the TM portion.
Thanks again for the thorough response. I’ll report back with what I come up with.
Edit: Rereading through the full code, I’m thinking I’m going to run into a brick wall with this “custom math binding”? I’m really unfamiliar with this type of thing; I’m assuming this is a binding to some C code that runs in a specific way? Should I just import these classes and use them myself? Is there some material to better understand how these work? Should I just ignore it and use Numpy arrays and accept the difference in speed? I apologize for my ignorance. I’m hoping once I get all this foundational stuff out of the way, I can start contributing.
IIRC NuPIC contains an entire sparse matrix library and python bindings for it.
The HTM.Core community fork of NuPIC has a special piece of code called the “Connections” class. It is bespoke for dealing with sparse synapses, and it has been fairly well optimized. HTM.Core does not contain any sparse matrices. Instead the Connections class stores its data as lists-of-lists.
@dmac Thanks for that information. It looks like this is being done in C++, which I am not familiar with. I guess I will stick with using numpy arrays for now, then check out scipy sparse arrays to see if it will speed it up, then I’ll just import the custom Nupic sparse arrays and see how much of a difference those make.
The connections class has python bindings but that class is considered internal to the library, so the API is marked as unstable and liable to change (though unlikely) and it is not as well documented as the rest of the library.
