Memory Palaces, grid cells, and exploiting sequences and HTM for sophisticated processing beyond mere recall

I’ve recently learned to use Method of Loci and other memory systems. While I’ve been having fun with them, I started wondering whether they’re limited to recall.

In a way, these systems are a method of efficiently and directly programming our brains by exploiting evolved, built-in functions like spatial awareness, facial and emotional recognition, location memory, and so forth. Memory grandmasters can memorize entire decks of cards in amazing time, the record just under 13 seconds.

The methods exploit the brain’s face recognition, novelty detection, sparse distribution, and spatial memory to attach arbitrary concepts to SDRs. After learning several methods, and knowing what I do of HTM, it’s a blazingly obvious validation of the function of sparsity in cognition.

Method of Loci uses imagined journeys with a persistent route, often with unique combinations of people, objects, colors, emotions, or associated words. Short term memory gives us access to the parts of the brain where we are memorizing items in long term memory.

Knowing what we do about how HTM works, is there a way of exploiting these methods for higher order processing? What experiments could we perform that could provide direct experimental confirmation of the HTM as the brain’s operating algorithm?

Memory athletes often use outrageous associations, exploiting sparsity and preventing fuzzy overlapping with existing memories - using sex, violence, and profanity in mnemonics to achieve this. The use of existing knowledge, such as people you know, celebrities, actions, arbitrary lists like signs of the Zodiac, and so forth, can be combined to create ever more complex representations of arbitrary items like card orders, phone numbers, addresses, math formulas, words in foreign languages, and so on. Mnemonic recall is a trivially acquired skill, and you get better at it the more you use it.

I don’t know how to set up higher order processing quite yet, but this seems like the place to figure it out.

If Method of Loci / Peg systems are first order functions, second order functions and so forth should be possible. More sophisticated calculations could be incorporated into functional Loci, allowing us to perform arbitrary mental math, like multiplying long numbers. I don’t think there’s any reason more sophisticated operations can’t be programmed abstractly.

Amazing mental feats performed by savants, such as factorization, that implies there are specific functions being performed in the brain that might be replicated through direct “programming” of this sort. The idea is to develop a skill that operates subconsciously, performing a sequence of operations, in which a result is recalled instead of consciously produced.

As an example of method of Loci:

So, imagine a living room. In the living room, turning clockwise from the center of the room, there’s a large flatscreen TV, a leafy green potted plant, a window with a red curtain, a brown leather recliner, a blue suede couch, an oak desk, a chrome coatrack, and finally a painting of Jeff Hawkins on the wall. That gives us 8 Loci/Pegs to attach items to:

  1. TV
  2. Plant
  3. Window
  4. Recliner
  5. Couch
  6. Desk
  7. Coat rack
  8. Painting

Once you’ve memorized this little list, and are able to visually traverse it backwards and forwards, you can then associate arbitrary memories with each item very quickly and easily, and recall the items at a later time with hardly any effort. Furnishing dozens or hundreds of rooms, buildings, or journeys results in a trivially accessible filing system that gives direct access to thousands of items. The items stored at each loci can be sophisticated constructs themselves, with encodings that represent more than individual factoids. Search for PAO and The Major System for examples.

If abstract lists can be learned, and then correlated and developed into specific hierarchical arrangements, then learning of new skills could be turned into very specific sequences of memorization. Instead of rote memorization of times tables, for example, a multiplication function could be broken into hierarchical structures, and then {something} could be done to obtain the output.

Does anyone have ideas for what that {something} could be?

I’m thinking of something like a dynamic loci, in which the associated memory is generated by operating on items stored in input loci. Imagine storing numbers in the above living room example, then adding a closet, and when you imagine opening the closet door, the sum of all the stored numbers in the room can be retrieved.

Let me know if you’ve got any experiments to try. Thanks for taking a look!

1 Like

You might be interested in the idea of knowledge frames, such as have been used before in symbolic processing systems. The example of this taken to the extreme is perhaps Cyc, which exhibits traits that look a lot like common sense. Doug Lenat, founder of cyc, describes his approach here.

What it sounds like is that you’re looking for HTM-like embedding/indexing of knowledge. I’ve some thoughts around that, but have done nothing substantial on the topic.

1 Like

Cyc was the first AI system I cut my teeth on, and has informed my understanding of hierarchical processing. It definitely demonstrates the power of networks, and that style of networking will play a part in functional loci. HTM can be viewed as a generative ontology with an implicit temporal component, and the temporal component can substitute for other, arbitrary features of a problem being solved.

The times table, learned by rote, creates a temporal, “if » then” multiple input calculation arrangement of synaptic connections.

Mnemonic techniques like a memory palace create a sequence that temporally leads from the imagined, abstracted loci to the arbitrary construct comprising whatever it is you want to memorize.

In a way, the PAO system ( ) creates functional loci, requiring unfolding of a concept beyond recall. The system requires rote mapping of each of its features (numbers, letters, words, card suits) to some unique person, action, or object. It’s the same process that memorizing the times table gives you the ability to multiply 1 digit numbers, and eventually abstract it as a foundation for multiplying larger numbers.

Synesthesia is a background player in mnemonics. Some memory athletes learn color associations with numbers, linking their qualiative / experiential memories with arbitrary concepts for recall, imbuing objective concepts like numbers with color, taste, emotion, texture, or sound. That is a validation of the HTM theory of cognition, a far as I can tell.

I want to program a function into my brain, like 10 digit binary multiplication, such that I can place the inputs in a memory palace, and then look into another place in my mind to retrieve the output.

Knowing that HTM is the operating algorithm, what would the regional and hierarchical arrangements look like if you were to implement this with nupic?

You’d need three memory palaces - two having 10 loci for inputs, and the third having 20 loci for the output. The specific thing I want to figure out is how to train the brain to populate the output loci with the correct answer, without having to consciously process the information beyond placing the inputs. I think this particular experiment would be sufficient to prove the process.