How to easily implement your HTM?

Engineering
1

Hi, this is my first post. I want to implement the latest HTM, but I want a 1 compact and detailed pseudo code. Until BAMI is updated, is there such a resource?

Do you use fully-connected layers?
Is it possible to do it as tensor operations (To just use any TF & PyTorch library)?

I think a “Implement HTM in 10 lines” tutorial would attract people to you.

1 Like
2

Currently, I’m re-reading BAMI.

3

I will suggest this JavaScript implementation built by @Paul_Lamb here:

It is clean and simple, and should help you get started.

Re-reading BAMI would be a good thing. :slight_smile:

Never. Fully connected layers produce dense representations. HTM depends on sparse representations.

Yes, see our current discussions:

CHALLENGE! If anyone can do this I will reward you!

3 Likes
4

Thanks. :smiley:

5

Implementation options:

  1. OOP style (Like HTM.js): classes for column, synapse, neuron, etc and each class has a vector with list of connected items.
  2. Some way to do it once with math ops.

Which is easier?

6

I’m currently searching through available implementations.

7

It is pretty much a personal preference, honestly.

8

It’s not too difficult to implement a relatively complement HTM. (Compared to NNs.) But it still is a lot of work.

My personal preference is to go with a Data oriented Design (Like what PyTorch and a lot of other DL library does. It promotes deep knowledge of the underneath system and reduces dependency).
As for math operations… No, standard numpy operations are not enough (or at least vert inefficient) to implement HTM. In my HTM implementation, I sotre synapses as ND arrays and write custom operations for them. This is faster than OOP that the data is stored close to each other, reducing cache misses and along with other benefits.

4 Likes
9

I think python would be ideal because it’s easy, popular and powerful, but people have talked about implementing HTM algorithms in languages that would easily allow it to scale across clusters like Elixir and Pony.

I wonder how amenable it is to GPU matrix operations. The best longterm architecture maybe something like a cluster of GPU machines.

1 Like
10

It’s something I’m currently working on, so might not be accurate. HTM can be somewhat efficient on GPU by storing cell state in the GPU’s local memory and stream the synapses from VRAM. This can significantly improve the access pattern and have very short access time. Local memory can be accessed ~3 cycles and linear VRAM access can be down to ~60 cycles vs ~100 cycles for random access if I recall correctly.
With that said, it is not an matrix operation and has to implemented in OpenCL/CUDA directly. I’ve dig a bit trough TVM and other matrix/tensor libraries. It’s nearly impossible to them to generate the code for such operation (mainly because they see local memory as a temporary buffer, not memory for long term use). I’ll release my work when it’s done.

2 Likes
11

I’m considering either Python or Rust:

  • Python advantages: PyTorch & TF. I use both.
  • Rust advantage: NDArray & Rustsim. They seem to give some flexibility, but I never used them before and I need to improve my system programming skills.

It’s highly likely I’ll choose Python.
But re-reading once wasn’t enough to fully understand it. Sorry, I need to more re-readings. :sweat_smile:

2 Likes
12

Hey @sherif7810 it’s definitely good to get a full understanding of the system before you dive into it, saves you from having to refactor a million times. So by all means read and reread as many times. I read papers on it several times as well. I also do this thing where I build out tiny chunks of code on little independent sections when I’m exploring the ideas, then a full project after I’m confident about each concept.

As for what language and style, definitely a language you are comfortable in. No need to complicate things by trying to learn the ins and outs of a language and HTMs at the same time. Object Oriented Design will be good for your first build because its easy to add and change stuff, but @marty1885 is definitely right, I’m newer to it, but Data Oriented Design is much much more efficient, especially when you have lots of elements with the same operation being done over all of them.

If you decide to look into Data Oriented Design, make sure you build the system in a design you are comfortable and familiar with, DOD requires you to understand the system in its entirety and some pre planning, but the efficiency rewards are amazing. You should also take a look at this video before you jump into DOD just so you can kind of see the reasoning behind the style. This particular video isn’t explicitly about DOD but the things he talks about are relevant even if you use OOP.

But, I would encourage you to use Data Oriented Design if you end up making a production formalized version.

13

A post was split to a new topic: Please review my HTM implementation in Rust