# Power-of-Two-Based Permutation Logic

**URL:** <https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618>\
**Category:** Lounge\
**Created:** [November 22, 2016, 12:58am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618 "2016-11-22T00:58:58Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 22, 2016, 12:58am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/1 "2016-11-22T00:58:58Z")

</div>

I’m just reading this at the moment:  
[Power-of-Two-Based Permutation Logic](http://journal.frontiersin.org/article/10.3389/fnsys.2016.00095/full)

Generally though I am more looking at papers about saddle points etcetera in deep neural nets.

[https://papers.nips.cc/paper/5486-identifying-and-attacking-the-saddle-point-problem-in-high-dimensional-non-convex-optimization.pdf](https://papers.nips.cc/paper/5486-identifying-and-attacking-the-saddle-point-problem-in-high-dimensional-non-convex-optimization.pdf)

The basic idea is that you never actually end up being trapped in a local minimum while training a large neural net. There is always some vector direction you can move the weights in to get an improvement. Ie. you can’t go wrong using some variation of the random hill climbing algorithm (for descending!!)  
I guess it ends up that the structure of a deep neural net is stupid and the algorithm you train it with is stupid. However if you throw enough floating point operations at it with a GPU cluster you can train it to do things in a few weeks that would take an equivalent biological system millions of evolution to be able to do.

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 24, 2016, 2:02am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/2 "2016-11-24T02:02:17Z")

</div>

When I try random hill climbing with small nets it doesn’t work too well (or at all.) When I try it with large nets it seems to go far more smoothly. The scaling effects look to be counterintuitive. I guess that is the reason people abandoned the idea in the 1980’s when they could only experiment with small nets.

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 24, 2016, 5:43am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/3 "2016-11-24T05:43:14Z")

</div>

The biological paper then is suggesting (sparse) locality sensitive hashing. Okay, I’ll think about it that way. Maybe I’ll investigate multiplexing and demultiplexing as well.

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 25, 2016, 12:44am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/4 "2016-11-25T00:44:29Z")

</div>

I suppose repeatedly demultiplexing from 4 to 15 (+ 1 null) states is a very expansive and very sparsity inducing. In some forms of machine learning you create millions of variations of the input data and then choose some of those variations that best explain the target. Of course the chosen items can be demultiplexed into new variations and used to further explain the target.

Anyway you could have made something like that in the early 1970’s:[http://www.ti.com/lit/ds/symlink/74ac11138.pdf](http://www.ti.com/lit/ds/symlink/74ac11138.pdf)

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 27, 2016, 8:29am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/5 "2016-11-27T08:29:00Z")

</div>

So recurrence plus demultiplexing to create a reservoir and then linear readout. It that it?  
Well, who can be sure yet.

[https://youtu.be/Seme-955jYk](https://youtu.be/Seme-955jYk)

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 27, 2016, 3:56pm UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/6 "2016-11-27T15:56:05Z")

</div>

I’ll save you the trouble of having to look for the paper:  
[http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004967](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004967)

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 28, 2016, 2:48am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/7 "2016-11-28T02:48:03Z")

</div>

[http://journal.frontiersin.org/article/10.3389/fncom.2010.00024/full](http://journal.frontiersin.org/article/10.3389/fncom.2010.00024/full)

And just as an aside:  
[https://en.wikipedia.org/wiki/Chaos\_computing](https://en.wikipedia.org/wiki/Chaos_computing)

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 28, 2016, 8:38am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/8 "2016-11-28T08:38:27Z")

</div>

It kinda, sorta sounds as if Hawkins/Ahmad synapse building results in micro-correlation learning in the reservoir. Making the reservoir less sparse over time and increasing the number of non-linearities that can be drawn from to make predictions. That is a dual learning system that works together to create great richness in the system yet being very simple to train. And in fact having very obvious and biologically plausible learning mechanisms.  
Are kinda and sorta valid scientific terms?

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 30, 2016, 5:08am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/9 "2016-11-30T05:08:59Z")

</div>

I’m kinda using your forum as a collecting point for related ideas, hope it’s not annoying.  
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2773171/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2773171/)  
I guess a simplification would be to use Hawkins/Ahmad type neurons as micro-correlators but not to feed the outputs back into the (random) reservoir. Then learn read-out projections from the whole kit and kaboodle to the wanted result. So while the reservoir might only contain a cue for a bright light and a cue for a loud noise a Hawkins/Ahmad neuron projecting into the reservoir could (unsupervised) learn any correlations in the environment between the two cues (simultaneous or one followed the other.) You are enriching the reservoir with unsupervised correlation learning. Obviously if that neuron were to fire it would be good idea to initiate fleeing behavior.  
Such a myriad of possibilities though, AI is all about the structure.

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 30, 2016, 5:30am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/10 "2016-11-30T05:30:08Z")

</div>

[http://journal.frontiersin.org/article/10.3389/fncom.2015.00036/full](http://journal.frontiersin.org/article/10.3389/fncom.2015.00036/full)

---

<div class="post-metadata">

**Author:** ![Sean\_O\_Connor](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)\
**Post date:** [November 30, 2016, 7:18am UTC](https://discourse.numenta.org/t/power-of-two-based-permutation-logic/1618/11 "2016-11-30T07:18:54Z")

</div>

[https://archive.org/details/Redwood\_Center\_2010\_02\_24\_Gordon\_Pipa](https://archive.org/details/Redwood_Center_2010_02_24_Gordon_Pipa)  
I’ll leave it there for the moment and think about how best to code such ideas.