HTM Cheat Sheet

reference

#1

This is a community-created and managed reference card about HTM theory and implementation for easy lookup for common information. Please add to it if you have something interesting to add.
If you see “?” and know the answer, please fill it up.


Brain

  • 5% of a human’stotal body weight, it uses 20% of the body’s oxygen.
  • Total number of neurons in the brain : 86 billion
  • Energy consumption : 20 W
  • 1 sq. mm. contains : 170 000 neurons

Neocortex

  • 75% of the brain
  • Total number of neurons in the cerebral cortex : 21-26 billion
  • Total number of neurons in the neuro-cortex : ??
  • Total number of connections : ??
  • Total number of synapses : 100-180 Trillion (depends on sex and age)
  • Neo-cortical sheet is about 1000 cm^2 in area and 2.5 mm thick
  • One mm^2 of cortex has about 100K neurons and 1B synapses
  • Cortex is distributed in 6 layers (there are actually more like nine layers 1,2,3a,3b,4,5a,5b,6a,6b)

Cortical Region

  • Although different regions of the neocortex process different inputs (vision, hearing, touch, language, etc.) at a fundamental level these are all variations of the same problem, and are solved by the same neural algorithms.
  • The regions are defined by connectivity. Regions pass information to each other by sending bundles of nerve fibers into the white matter just below the neocortex. The nerve fibers reenter at another neocortical region. The connections between regions define a logical hierarchy.
  • Neuroanatomy tells us that every region of the neocortex has both sensory and motor functions. Therefore, vision, hearing, and touch are integrated sensory-motor senses; we can’t build systems that see and hear like humans do without incorporating movement of the eyes, body, and limbs.

Cortical Column

  • A cortical column is about 1.0 - 1.5 mm^2 in area and contains about 2000+ mini-columns
  • HTM: 2048 minicolumns under 1 spatial pooler
  • All cortical columns are learning a complete model of the world of everything they get exposed to and they are all doing it in parallel. Each cortical column basically learns the same thing in parallel and votes (via layer 2 communication).

Mini-column

  • About 30-50 microns wide with 100-120 neurons across all 6 layers
  • HTM: 32 cells per mini-column in layer 3 implementation

Neuron (Cell in HTM)

  • Pyramidal neurons are the most common type of neuron in the neocortex
  • Average number of connections : 1000 - 10 000 ??
  • Average number of dendrite segments : ?
  • Average number of synapses per dendrite segments : ?
  • The pyramidal neuron is the core information processing element of the neocortex, and synapses are the substrate of memory

Note: Need to add apical dendrites and synapses to the cell model visualization

Encoder

  • Takes sensor inputs and converts them into SDRs
  • HTM Examples: Scalar Encoder, Random Distributed Scalar Encoder (RDSE)
  • Take stimulus from the environment and translate them into a stream of SDRs that are neural activity going to the brain.
  • An encoder takes some type of data–it could be a number, time, temperature, image, or GPS location–and turns it into a sparse distributed representation that can be digested by the HTM learning algorithms. The HTM learning algorithms will work with any kind of sensory data as long as it is encoded into proper SDRs.

Biological Examples:

  • Vision: Retina
  • Hearing: Cochlea
  • Touch: Nerves

HTM Implementation Parameters

  • Num Columns (N): 2048
  • Num Cells per Column (M): 32
  • Num of active bits (w) : 41
  • Sparsity (w/N) : 2%
  • Dendritic Segment Activation Threshold (θ): 15
  • Initial Synaptic Permanence: 0.21
  • Connection Threshold for Synaptic Permanence: 0.5
  • Synaptic Permanence Increment and Decrement: +/- 0.1
  • Synaptic Permanence Decrement for Predicted Inactive Segments: 0.01
  • Maximum Number of Segments per Cell: 128
  • Maximum Number of Synapses per Segment: 128
  • Maximum Number of New Synapses Added at each Step: 32

Tree Neurons
HTM School Episode 7: Spatial Pooling Input Space & Connections
#2

I moved this from #other-topics to #htm-hackers.

Not a bad idea… maybe we change the title of this topic to “HTM Cheat Sheet” and make it a wiki?


#3

Good idea. It should specify

  • the respective name of the parameter in Nupic,
  • its estimated size in the cortex, and
  • its suggested size in the implementation, especially whether it’s absolute or relative in nature (e.g. depending on the overall region size / # of columns).

#4

I’ve actually been working on my own HTM notes to help wrap my head around the theory. Some of it may be direct quotes from members of this forum (Jeff comes to mind). It’s still very much a WIP, but I hope some of it is of benefit to the cheat sheet! I will be updating as I go so hopefully I’ll remember to update this post, too.

I moved the rest of this content up into the main wiki - Matt

Side Question: Why are permanence values floats from 0.0 to 1.0? Why not an int8 from 0 to 100?


#5

@ddigiorg This is awesome! Thanks for sharing.

  • Dendritic Segment Activation Threshold (θ): 15

I suppose this is the number of synapses per segment that need to be active (= permanence exceeds threshold) for the segment to become active, correct?

  • Initial Synaptic Permanence: 0.21

Here my understanding was that the initial permanence would be a random value chosen around the threshold. Citing the SP whitepaper:

> Prior to receiving any inputs, the code is initialized by computing a list of initial potential synapses for each column. This consists of a random set of inputs selected from the input space. Each input is represented by a synapse and assigned a random permanence value. The random permanence values are chosen with two criteria. First, the values are chosen to be in a small range around connectedPerm (the minimum permanence value at which a synapse is considered “connected”).

Above refers to proximal synapse initialization though, but I assume distal would be the same.

  • Synaptic Permanence Increment and Decrement: +/- 0.1
  • Synaptic Permanence Decrement for Predicted Inactive Segments: 0.01

Why are 2 different increments/decrements needed? Didn’t see that in the whitepaper.

  • Maximum Number of Segments per Cell: 128
  • Maximum Number of Synapses per Segment: 128

If these are the maximum values, what are the values you start with?

Also, I’d expect these values to have a relationship to the overall number of cells. So if the region has 2048*32=65k cells, the number of synapses would be defined as 0.2% of the total cell count. The reason I’m stressing this point is that in the implementation, this actually should not be a separate parameter to be set because it’s automatically derived from the column/cell count.

Side Question: Why are permanence values floats from 0.0 to 1.0? Why not an int8 from 0 to 100?

With ints you could always have 100 steps only but with floats the range is endless depending on how small you set the increment (learning rate).


#6

Thanks @lindmatt! I got the HTM Implementation Parameters from the Appendix of Numenta’s Continuous online sequence learning with an unsupervised neural network model.

I suppose this is the number of synapses per segment that need to be active (= permanence exceeds threshold) for the segment to become active, correct?

Yes, Dendritic Segment Activation Threshold indicates there must be above 15 active synapses above the permanence threshold.

Here my understanding was that the initial permanence would be a random value chosen around the threshold.

I agree that the Initial Synaptic Permanance being a random value chosen around the threshold. Not sure why it wouldn’t be and I think it is actually implemented that way, but I’m not certain.

Above refers to proximal synapse initialization though, but I assume distal would be the same.

The proximal synapses for each column are initialized to a random subset of input bits. However, the basal synapses for each cell are not initialized. Instead (Please correct me if I’m wrong) basal synapses start with 0 segments per cell and 0 synapses per segment and grow or die dynamically in the Temporal Memory algorithm because initializing so many basal synapses all at once takes a looooong time and uses up a lot of memory.

With ints you could always have 100 steps only but with floats the range is endless depending on how small you set the increment (learning rate).

True, but then again one could just use larger integers for a wider range, right? I thought ints are generally a faster representation (totally making this up and probably really embarrassing myself on a basic Comp Sci concept so I will have to research this) but I guess it doesn’t really matter in the end. It’s just a representation of bits after all.

EDIT: I should probably mention that everything I say should be taken with a grain of salt. I am still learning and it’s very possible I could be missing something. Please correct me if I’m wrong because I’d like to learn as much as possible!

Dave


#7

@ddigiorg

The proximal synapses for each column are initialized to a random subset of input bits. However, the basal synapses for each cell are not initialized. Instead (Please correct me if I’m wrong) basal synapses start with 0 segments per cell and 0 synapses per segment and grow or die dynamically in the Temporal Memory algorithm because initializing so many basal synapses all at once takes a looooong time and uses up a lot of memory.

I think we’re talking about 2 different things here:

  1. Initial Permanence values – I was referring to both proximal and distal permanences being initialized with a random value around the threshold.

  2. Initial Segment + Synapse structure – you’re referring to how many segments and synapses per segment the algorithm starts with.

Of course, both are linked, i.e. if you do start from 0 segments and 0 synapses, you don’t need to initialize any permanences. :grinning:


#8

Oops, you’re right. It’s midnight on a Friday night and my reading comprehension has been found wanting…

Time for beer. It kills the weaker brain cells first! :laughing:


#9

Ok all, I converted @mraptor’s first post to a wiki and moved @ddigiorg’s content into it as well. It is now up to you to organize the content and plan how it should be structured. Use this thread to discuss the wiki, but treat the first post as the actual wiki content.


#10

Cortical Column is equivalent to hierarchy of HTM-regions, right ??
Mini-column is equivalent to column inside HTM-region ?


#11

Yes that’s right. Let me be explicit just in case I’m mistaken: a Cortical Column (a “macrocolumn” or “hypercolumn” in the biological neocortex) consists of many minicolumns. The HTM that we’re used to that has a Spatial Pooler and Temporal Memory is a model of layer 3 in a Cortical Column. In a HTM Region, many Cortical Columns are connected in parallel and “vote” on the Region’s output. It’s the Regions themselves that are hierarchical.

I definitely will need to make a better graphic to reflect this a bit more explicitly. So many things to keep in mind!


#12

I thought it is the other way around !

HTM-Region (SP+TM) is 2048 mini-columns and CC is hierarchy of HTM-regions.


Memory layout strategies for a Cell
#13

I modified the graphic under the “Neocortex” heading. I think this is more clear.

In HTM per my understanding a Cortical Column has 2048 mini-columns. The Spatial Pooler selects active minicolumns across the entire CC. Temporal Memory replicates layer 3 neurons in the biological neocortex. The hierarchy comes from the connection of Cortical Regions.

I’d love to have someone from Numenta confirm or correct this.


#14

Hey @Paul_Lamb, have you thought about updating this with more of the graphics you’ve created? You might post them here for discussion if you think they are relevant.


#15

Sure, that’s probably a good way to discuss components of HTM and fact-check any assumptions. I’ll put some together to try and depict graphically where the HTM Implementation Parameters apply in the system.


#16

Does somebody knows those numbers ?


#17

Perhaps this paper could help:


#18

3 posts were split to a new topic: Do minicolumns span layers?


#21

9 posts were split to a new topic: Cell segments vs synapses