# When I look at what deep neural networks do
**URL:** https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498
**Category:** Lounge
**Created:** [February 5, 2018, 1:52pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498 "2018-02-05T13:52:24Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)
#### Post date: [February 5, 2018, 1:52pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/1 "2018-02-05T13:52:24Z")
When I look at what deep neural networks do what I see at each layer is a linear mapping followed by a pull to one of the attractor states. That gives decision regions based on the boundaries between the attractor states. Also it seems to me there are problems with lots of unwanted boundaries that are hard to get rid of.
Associative memory alone cannot replace what deep networks do in terms of conditional decision regions. Where if a vector is in one decision region in a given layer it will lead to a conditional choice between some attractor states in the next layer.
It could be though that if you gave associative memory a helping hand it could find conditional decision regions robustly and efficiently. That would be very shocking for the deep learning community. One tool to do that might be to have agreeing associative memories. If two (or more) memory systems agree and are wrong then do something. If two memory systems agree and are right then do something else. If two memory systems disagree then do something else again. It should be possible to create conditional decision structures based on that.
---
### Author:  [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)
#### Post date: [February 6, 2018, 12:05am UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/2 "2018-02-06T00:05:49Z")
I suppose then with an 'agreeing" associative memory you can split its responses into known, (both equal but) wrong and unknown. For wrong responses you create an additional memory system to learn a conditional alternative. You know to use the conditional rather than the original because it agrees too (self signaling.)
For unknown you start again with a new unconditional associative memory to learn less common things than the first.
You could end up with a forest of rooted trees.
Such matters have a lot to do with self organizing maps (SOM.)
Work has been done before with SOM, but not so much on associative memory based SOM. Mainly because notions about extreme learning machines weren’t current at that time and only poor quality Hopfield AM was available.
It’s time to revisit these things and have a look at the literature.
---
### Author:  [@Bitking](https://discourse.numenta.org/u/Bitking)
#### Post date: [February 10, 2018, 3:07am UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/3 "2018-02-10T03:07:36Z")
This might interest you:
> [@How long does a cell stay depolarized?](https://discourse.numenta.org/t/how-long-does-a-cell-stay-depolarized/3446/5):
>
> I have had long walks talking to myself about these metabotropic critters. One amusing line on these receptors is as follows: They smear the current activation pattern out in time. If this function was happening to you right now you would be “enhancing” the current perceptions into a sort of “here and now.” You know, what you are feeling at this moment. Well - every moment. Functional description - project current activation into the future. If similar fields are firing at the same time they w…
---
### Author:  [@Bitking](https://discourse.numenta.org/u/Bitking)
#### Post date: [February 18, 2018, 5:15pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/4 "2018-02-18T17:15:37Z")
> [@Untangling Sequences: Behavior vs. External Causes](https://discourse.numenta.org/t/untangling-sequences-behavior-vs-external-causes/2883/5):
>
> Also it seems to me there are problems with lots of unwanted boundaries that are hard to get rid of.
>
> Associative memory alone cannot replace what deep networks do in terms of conditional decision regions. Where if a vector is in one decision region in a given layer it will lead to a conditional choice between some attractor states in the next layer.
This made me think about the works of William H. Calvin. He did some pioneering work on a hexagonal organization of neural information and described some of the exact situations you describe. If you note the similarity to the current work on grids his work becomes very relevant to the current discussion.
COMPRESSING THE CEREBRAL CODE - William H. Calvin
[http://williamcalvin.com/socns94.html](http://williamcalvin.com/socns94.html)
[http://williamcalvin.com/bk9/bk9ch6.htm](http://williamcalvin.com/bk9/bk9ch6.htm)
The entire book:
> **[THE CEREBRAL CODE by William H. Calvin (MIT Press 1996)](http://williamcalvin.com/bk9/)**
>
> Chapter titles: The Representation Problem and the Copying Solution, Cloning in Cerebral Cortex, A Compressed Code Emerges, Managing the Cerebral Commons, Resonating with your Chaotic Memories, Partitioning the Playfield, The Brownian Notion,...
---
### Author:  [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)
#### Post date: [February 21, 2018, 1:19am UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/5 "2018-02-21T01:19:37Z")
Thanks for the links. If deep neural networks are all about compounded decision regions (defined by basins of attraction) and that is all, then there might be more efficient ways of doing things.
The next project I am trying is to let evolution very directly specify the decision regions, their size and location and then combine that with what I call if-except if decision trees.
I tried those special decision trees on text before and the results were quite impressive, with semi-readable text via prediction as output, always with correct spelling and a limited amount of sentence structure. However higher order reasoning about grammar etc was not possible because there was no search mechanism to identify and link together concepts. Evolvable decision regions should allow that to happen, but might not be terrible efficient. Anyway I’ll try.
---
### Author:  [@rhyolight](https://discourse.numenta.org/u/rhyolight)
#### Post date: [February 21, 2018, 5:13pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/6 "2018-02-21T17:13:17Z")
I split this from another topic and I don’t know what to title the thread, so I just used @Sean_O_Connor’s firs sentence. Anyone may suggest a better title (or edit the title if you have permission).
---
### Author:  [@Bitking](https://discourse.numenta.org/u/Bitking)
#### Post date: [February 28, 2018, 11:59pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/7 "2018-02-28T23:59:13Z")
I always wonder what it is that people expect to happen with word generative networks without some “concept anchors” to drive the sentence construction.
Learning pairs of words leads to autocorrect at best.
There is a type of neural defect called fluent aphasia where the poor victim spouts off streams of gibberish that “sounds” like real speech. The grammar sub-system is working but it has nothing to work with.
> **[Wernicke's (Receptive) Aphasia - National Aphasia Association](https://www.aphasia.org/aphasia-resources/wernickes-aphasia/)**
>
> In this form of aphasia the ability to grasp the meaning of spoken words and sentences is impaired, while the ease of producing connected speech is not very affected. Therefore Wernicke’s aphasia is also referred to as ‘fluent aphasia’ or ‘receptive...
Now if you could create a chatbot using this general approach:
Use Parsey Mcparseface to pick out the elements of speech, post the found words into a “blackboard” and look for keywords.
Make a list of concepts structures to manipulate these elements into a semi-intelligent sounding conversation. Access to these structures would be based on the old frames concept; combined with some power data mining to build your frames.
> **[Frame (artificial intelligence)](https://en.wikipedia.org/wiki/Frame_(artificial_intelligence))**
>
> Frames were proposed by Marvin Minsky in his 1974 article "A Framework for Representing Knowledge." A frame is an artificial intelligence data structure used to divide knowledge into substructures by representing "stereotyped situations." Frames are the primary data structure used in artificial intelligence frame language.
> Frames are also an extensive part of knowledge representation and reasoning schemes. Frames were originally derived from semantic networks and are therefore part of structure ...
There would have to be some overarching goals to select and combine access to the frame structure.
Add some grammar constructions primed with the output of this frame parsing and it might be a fun toy.
---
### Author:  [@Paul\_Lamb](https://discourse.numenta.org/u/Paul_Lamb)
#### Post date: [March 1, 2018, 2:22pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/8 "2018-03-01T14:22:59Z")
> [@Bitking](#):
>
> There would have to be some overarching goals to select and combine access to the frame structure.
>
> Add some grammar constructions primed with the output of this frame parsing and it might be a fun toy.
Furby 3.0… (or 4.0? I forget how many variations there have been)
---
### Author:  [@Bitking](https://discourse.numenta.org/u/Bitking)
#### Post date: [March 1, 2018, 3:22pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/9 "2018-03-01T15:22:19Z")
Might end up being an expensive toy.
---
### Author:  [@Sean\_O\_Connor](https://discourse.numenta.org/u/Sean_O_Connor)
#### Post date: [March 1, 2018, 11:23pm UTC](https://discourse.numenta.org/t/when-i-look-at-what-deep-neural-networks-do/3498/10 "2018-03-01T23:23:00Z")
This chip will probably be less than $5.
[http://www.tomshardware.com/news/rockchip-rk3399pro-ai-chip,36270.html](http://www.tomshardware.com/news/rockchip-rk3399pro-ai-chip,36270.html)