Trying to implement HTM theory using Julia

Thank you @breznak
I have posted a query about implementation aspect but it does make more sense to post this entire question on the forum. I’ll do it soon.
I would like to ask you all for advice here the same. Any input is much appreciated.

Sure, here are a couple of things that come to mind:

  1. Encode SDRs as an array of indexes to the “1” bits (versus a large boolean array)

  2. When scoring columns, iterate over the active cells in the input space to score their connected columns (versus iterating over every column to score them based on their connected active cells in the input space).

  3. For depolarizing cells in TM, transmit the activity while activating the cells at time T (versus iterating over all cells in active columns at time T+1).

3 Likes

Thanks @Paul_Lamb, will use some of those insights for re-work in the community nupic.cpp!
I definitely want the 1/ Sparse arrays.

As the author of HTM in another lang, did you use a variant of the “Connections” class? My Q is for using it in SP: Implement SpatialPooler with Connections backend

Thank you.
Will certainly make use of those suggestions. I was aware of 1 since Jeff Hawkins mentioned that in one of his talks. 2 and 3 are certainly interesting. Will try those first.

I had similar idea as to the implementation of SP and TM. But I had arrived at the same conclusion by thinking backwards that every TM columnar layer is essentially an SP and only a single SP unit(minicolumn) will be selected from the TM column as active after selecting winning “cells”.

For some potential dead-ends, read through this thread if you haven’t already. I proposed some potential optimizations, and there was some very good discussion on what the side effects would be.

2 Likes

@abshej Why did you send this as a private message to everyone instead of just using the forum? If you don’t mind, I’m just going to make this a public forum post in #htm-hackers.

2 Likes

My bad, I had a different thought process behind doing so.

I don’t mind at all. Please do so.

As far as getting started it may help to know the theory level behind the code first.

The Numenta site has an excellent resource page. If you have not seen it look here:
https://numenta.com/papers-videos-and-more/

I got a lot out of the BAMI living book there:

Coding is an implementation level of the theory.

What do I mean by “level?” See the link to Marr’s three level in this post:

1 Like

Thank you.
I have read some parts of BaMI, HTM white paper and also some neuroscience research papers. Although, frankly, I didn’t finish reading any of them properly. I’ll go through them again. Thanks for the reference to the post. Will surely look into it.

A month in the laboratory can often save an hour in the library.
F. H. Westheimer

http://threeplusone.com/gec/quotes
https://en.wikiquote.org/wiki/Frank_Westheimer

1 Like

Please ignore this. It is outdated.

1 Like

what is the go-to version? BaMI?

1 Like

There are mutliple ways to encode SDRs, with pros and cons.

A list of indexes has the smallest memory usage and is faster to iterate through, but determining if a certain bit in the SDR is active is time consuming. Even if you pre-sort the array and do a bisecting search, it takes O(log(N)) time per query, where N is the number of active bits in the SDR.

Dense boolean arrays use more memory and are slower to iterate through but determining if a certain bit in the SDR is active takes a constant amount of time. This operation is as fast as your systems RAM and does not change if you increase the number of bits in the SDR. Dense boolean arrays can also be quickly reshaped and sliced up in all sorts of ways.

Converting between these two representations takes a linear amount of time with respect to the number of neurons and the memory usage of a dense boolean array is 1 byte per cell.

An example of where both representations are useful is when the HTM learns: it iterates through the dendrite segments which should learn (an operation best suited for a list of indexes), and for each pre-synaptic input to a learning segment the HTM needs to know if that input was active (an operation best suited for a dense boolean array).

1 Like

One of the reasons I like the path table approach is that table is retained in CPU cache and runs very fast.
The path table is +/- computed offset for each element of the path list.
As I mentioned above - if you build the active synapse list for each cell you don’t search after doing that processing step.

1 Like

When is a newer version of BaMI or reference papers about the new additions to the theory including the apical dendrites and if possible, material related to allocentric location signals and grid cells’ computation, coming out?

1 Like

I honestly have no idea. I would like to work on BAMI if I run out of videos to do, but I’m not sure I will. Even if I did work on BAMI it would be to convert it from PDF into a web asset, and revising and completing content up to and including sequence memory.

But all new research will first come out as papers, usually with associated code samples.

:persevere: :persevere: :persevere:
I see, thank you.

7 posts were merged into an existing topic: Multiple proximal dendrites and neural model for SP