HTM Benchmarking Measurement Unit(s) Analogous to Convolutional ANN's Multiply-Accumulates/Sec?

Accepting that as HTM theory advances, the very definition of “HTM” changes, there are certain operations that seem to remain as primitives and, in that sense, are analogous to multiply-accumulates/second for convolutional neural nets – which approximate “connections per second” for ANNS, with the remaining operation of a non-linear function like sigmoid or RELU being insignificant to complete an ANN cycle. For that reason, measuring the multiply-accumulates/second is a useful benchmark for ANNs.

Is there a group of operations that, taken together, can comprise an analogous benchmarking unit of measure for HTM?

I suppose that depends on the flavor of HTM one decides to take. Current implementations in the official and community versions of HTM include (in my opinion) more information than is necessary to create a functioning hebbian-based learning system (which I’ve tried to outline in the “We need new hardware” thread). Rather than keep a list of connection strengths, keep a ring buffer of connections only. When a cell “wins”, it goes through its list of “connections” to increment the counters of the cells for those each of those connections (which are then used during cell selection for winning columns).

Nothing in the math requires multiplication.

Fundamental operations are:

  1. binary AND (to check for overlap between column’s distal connections and current input space encoding)
  2. accumulator (Add), for when those overlaps result in successful ANDs
    ** Combination of the 1 & 2 results in a ‘sum’ module to find overlap score for a minicolumn
  3. Min value register/variable
  4. Max value register/variable
  5. Floating average register/variable
    ** Combination of 3, 4, and 5 can be used for both k-means column selection, and choosing winning cells within a winning column.
  6. Sorting (of column overlap scores)

The basic operations then are:

  1. [SPATIAL MEMORY] For each minicolumn within a pool, AND+accumulate binary comparison with column connections to input space, and current encoding of that input space. Save overlap score (somewhere).

  2. [SPATIAL MEMORY] When all columns have calculated overlap scores, which could be continuously updating a average/min/max score values as those operations take place (and discarding from consideration any columns which fall below that average score), sort columns based on their scores, taking top N amount of winning columns (equaling about 2% of total columns).

  3. [SPATIAL MEMORY] For winning columns, strengthen connections to input space where connections and encoding bits overlap. (Optionally, depending on use) Decrease connection strengths for bits that do not overlap with current encoding.

  4. [TEMPORAL MEMORY] For winning columns, check activations for cells in column; if one is predictive, strengthen that cell by adding it to previous winner cells’ ring buffers. Store current winning cell to some known location for use by next round’s winning cells.
    ** This is where my approach varies most greatly from mainstream.

So for spatial pooler, that would be:

  • “input space bitsize” x “columns” number of accumulator operations.
  • “column sort”’ x “number of qualifying columns” affect speed of sort and number of operations
  • “input space bitsize” x “column connectivity pct” x “number of winning columns” number of strength updates.

Adding/Including temporal memory:

  • “number winning columns” x “cells per minicolumn” sorting and winning cell selection
  • “number winning columns” x “number of prev winning cells” number of “strength” updates
  • “number of winning cells” x “ring buffer size” ** → ** number “connections” updates.

I invite others to add to this, but for my own take, these are my operations that I’m counting.

1 Like