Learning high order sequences

Science Tangential Theories
1

Hi,

First time poster. My background is physics (long, long ago I got a BSc hons in physics and maths), and now I’m an independent researcher working on a knowledge representation language. I’ve been following Jeff and teams HTM work for a while now, and I’m a big fan! Sure the deep learning guys keep producing amazing results left and right, so they are obviously on to something, but I just don’t buy into their model of neural networks.

I’ve been wanting to implement the idea of high order sequence memory in my language for a long time, but only recently worked out how to do it. I hope this is not too off-topic?

Summary of this is a pretty table after my code has learnt “A B C D E” and “X B C Y Z”:

+-----+----------------+----------------+----------------+----------------+
| ket | step-1         | step-2         | step-3         | step-4         |
+-----+----------------+----------------+----------------+----------------+
| A   | B              | 1.00 C         | 0.87 D, 0.13 Y | 0.87 E, 0.13 Z |
| B   | 1.00 C         | 0.50 D, 0.50 Y | 0.50 E, 0.50 Z |                |
| C   | 0.50 D, 0.50 Y | 0.50 E, 0.50 Z |                |                |
| D   | 1.00 E         |                |                |                |
| E   |                |                |                |                |
| X   | B              | 1.00 C         | 0.87 Y, 0.13 D | 0.87 Z, 0.13 E |
| Y   | 1.00 Z         |                |                |                |
| Z   |                |                |                |                |
+-----+----------------+----------------+----------------+----------------+

Full blog post is here: http://semantic-db.blogspot.com.au/2016/07/learning-sequences-htm-style-using-if.html

thanks,
-Garry.

1 Like
2

Just a brief update. I automated my code a little. Now I can just specify a list of sequences, and then magic happens :grinning:

Here is a taste:
data = [“count one two three four five six seven”,“Fibonacci one one two three five eight thirteen”,“factorial one two six twenty-four one-hundred-twenty”]

t = 0.5:

+--------------------+--------+----------+----------------------------+----------------------------+-------------------------+----------------------------+---------------+
| ket                | step-1 | step-2   | step-3                     | step-4                     | step-5                  | step-6                     | step-7        |
+--------------------+--------+----------+----------------------------+----------------------------+-------------------------+----------------------------+---------------+
| count              | one    | 1.00 two | 1.00 three                 | 1.00 four                  | 1.00 five               | 1.00 six, 0.03 twenty-four | 1.00 seven    |
| one                |        |          |                            |                            |                         |                            |               |
| two                |        |          |                            |                            |                         |                            |               |
| three              |        |          |                            |                            |                         |                            |               |
| four               |        |          |                            |                            |                         |                            |               |
| five               |        |          |                            |                            |                         |                            |               |
| six                |        |          |                            |                            |                         |                            |               |
| seven              |        |          |                            |                            |                         |                            |               |
| Fibonacci          | one    | 1.00 one | 1.00 two                   | 1.00 three                 | 1.00 five               | 1.00 eight                 | 1.00 thirteen |
| eight              |        |          |                            |                            |                         |                            |               |
| thirteen           |        |          |                            |                            |                         |                            |               |
| factorial          | one    | 1.00 two | 1.00 six, 0.03 twenty-four | 1.00 twenty-four, 0.03 six | 1.00 one-hundred-twenty |                            |               |
| twenty-four        |        |          |                            |                            |                         |                            |               |
| one-hundred-twenty |        |          |                            |                            |                         |                            |               |
+--------------------+--------+----------+----------------------------+----------------------------+-------------------------+----------------------------+---------------+

t = 0.05:

+--------------------+---------------------------------------------+---------------------------------------------------------+--------------------------------------------------------------------+
| ket                | step-1                                      | step-2                                                  | step-3                                                             |
+--------------------+---------------------------------------------+---------------------------------------------------------+--------------------------------------------------------------------+
| count              | one                                         | 1.00 two, 0.03 one-hundred-twenty                       | 0.90 three, 0.10 six                                               |
| one                | 0.75 two, 0.25 one, 0.02 one-hundred-twenty | 0.53 three, 0.26 six, 0.21 two, 0.01 one-hundred-twenty | 0.28 four, 0.28 five, 0.24 twenty-four, 0.20 three                 |
| two                | 0.67 three, 0.33 six                        | 0.35 four, 0.35 five, 0.31 twenty-four                  | 0.32 five, 0.32 eight, 0.29 one-hundred-twenty, 0.07 six, 0.01 two |
| three              | 0.50 four, 0.50 five                        | 0.45 five, 0.45 eight, 0.10 six                         | 0.41 thirteen, 0.41 six, 0.10 seven, 0.09 eight                    |
| four               | 1.00 five                                   | 0.82 six, 0.18 eight                                    | 0.82 seven, 0.20 thirteen                                          |
| five               | 0.50 six, 0.50 eight                        | 0.51 seven, 0.51 thirteen                               |                                                                    |
| six                | 0.50 seven, 0.50 twenty-four, 0.01 thirteen | 1.00 one-hundred-twenty, 0.03 two                       |                                                                    |
| seven              |                                             |                                                         |                                                                    |
| Fibonacci          | one                                         | 0.83 one, 0.17 two, 0.00 one-hundred-twenty             | 0.76 two, 0.08 three, 0.08 six, 0.08 one, 0.02 one-hundred-twenty  |
| eight              | 1.00 thirteen, 0.02 seven                   |                                                         |                                                                    |
| thirteen           |                                             |                                                         |                                                                    |
| factorial          | one                                         | 0.91 two, 0.09 one, 0.02 one-hundred-twenty             | 0.75 six, 0.17 three, 0.08 two, 0.00 one-hundred-twenty            |
| twenty-four        | 1.00 one-hundred-twenty, 0.02 two           |                                                         |                                                                    |
| one-hundred-twenty |                                             |                                                         |                                                                    |
+--------------------+---------------------------------------------+---------------------------------------------------------+--------------------------------------------------------------------+

Pretty, huh?

Here are the tables, if you are having line wrap issues:
count-fibonacci-factorial–t-0_5.txt
count-fibonacci-factorial–t-0_05.txt
count-fibonacci-factorial–t-0_05–3-steps.txt

-Garry.

3

Garry, you might take a look at this language demo I prepped for our 2nd hackathon in Fall 2013:

This link starts in at 21m and talks about POS tagging and streaming POS into NuPIC. Sorry the AV is pretty bad, this was one of my first events. The repo I was working out of is now here:

But I haven’t run it in years, so YMMV.

BTW, “CEPT” is now “Cortical.io”.

4

Thanks, interesting, especially Cortical.io’s word SDR’s!
I presume they are not publicly available.
I wouldn’t mind reproducing your animal vegetable prediction example, for a start.
There are a bunch of other things I would like to try with them too.

As for predicting plurals, my hunch is there are no patterns for irregular plurals.
That’s why kids say “foots” and “tooths”. If there was a pattern I presume a kid would have found it – that’s what kids brains are very good at, finding patterns.

-Garry

5

Technically there are patterns for most irregular plurals – those patterns originated from other languages and worked their way into English through evolution of the language and cultural interactions. That said, identifying a higher-order pattern for when to use one of these lower patterns is probably not possible without encoding some semantics around the evolution of the English language. For example, taking the Umlaut pattern in your examples (changing a back rounded vowel [o, u] to the corresponding front rounded vowel [œ, y] in anticipation of a front vowel [i, e] in the next syllable):
tooth -> teeth
foot -> feet
goose -> geese

Without any additional information, one would logically predict that the plural of “boot” should be “beet”. However, if you had some way of encoding semantics of Germanic etymology, some higher-level pattern that isn’t immediately obvious might be found. Of course figuring out how to encode those semantics (or even if there is enough information available to do so) probably makes it a virtual impossibility.

1 Like
6

Actually, you can sign up for an API key here.

After this event, I talked to the Cortical.io guys and realized that they do not encode the idea of plural into their system.

7

Should I try to explain what my tables mean?
The first column, which I call the ket column, is the list of objects my system knows about after training on these three sequences:
“count one two three four five six seven”,
“Fibonacci one one two three five eight thirteen”,
“factorial one two six twenty-four one-hundred-twenty”

The step-k columns are then the predicted results k-steps from an input, where the float is the probability.
eg, given “count” as the starting point, the code predicts “one” with 100% certainty as the next step, and “two” with 100% certainty as the second step.

Now, let’s look at the second table.
Here if we start with “two” it predicts “three” with 67% chance, and “six” with 33%
If we start with “five” it predicts 50/50 between “six” and “eight”.
If we look at our three sequences, these probabilities match up nicely.

Notes:

  1. I am just using my interpretation of the HTM high order sequence algorithm to get these results
  2. the sequences don’t have to be words, they can be almost anything.

Next on my list is to work out how to encode online learning (currently it is batch learning), and anomaly prediction. I don’t yet know how to do either.