As a small note about, “empirical evidence on which structures have higher
permenance or even life-long permenance”
There is evidence that suggests that the brain is a reductionist system,
and that over time information is encoded by pruning the synaptic forrest.
This comes from the fact that a childhood brain differs from an adult brain.
During childhood, and particularly during adolescence, a process
known as “synaptic pruning” occurs.
Although the brain http://www.human-memory.net/brain.html continues to
grow and develop, the overall number of neurons and synapses are reduced by
up to 50%, removing unnecessary neuronal structures and allowing them to be replaced by more complex and efficient structures, more suited to the
demands of adulthood.
This is something that I believe has not been mentioned, and I think it
deserves some consideration.
The pruning effect which takes place during adolesence in humans (and mammals) is activated by what is called neuronal apoptosis (neuronal programmed cell death). This genetically programmed process is stored in the genes and ensures that certain cells, destroy themselves after a certain period of time and perhaps also when given conditions are present in the cell. Apoptosis is not limited to the brain, it is necessary for the entire body to renovate itself, but in the brain we can observe a very highly regulated version of apoptosis which not only prunes away the right neuronal cells, but also unfolds during a very specific developmental period.
I fully agree with you, i.synapse, that we need to also pay much more attention to this very crucial and developmentally essential process in our brains. Without the correct pruning, the formation of an HTM with full blown hierarchy is most likely not possible. This means that understanding this process is crucial to modelling HTMs. (This probably does not apply to a single CC region, but it most likely does apply for any working hierachy of regions).
Another very interesting point is the fact that since this process is activated by specific genes working in a complex sequence with interdependencies, it is conceivable that understanding this genetic mechanism, would make it possible to control both neuronal growth and apoptosis and perhaps make a second phase of pre-puberty massive learning possible in adulthood. This would only make sense in cases, like the one of Mike May, with long-term vision deprivation followed by vision restoration in adulthood.
Children already have deep hierarchies, as far as I know. The frontal cortex is functional, but not well developed. Synaptic pruning might help without being crucial, which could help HTM a little. It might also be to reduce energy consumption, because the brain uses a lot of energy. Synaptic pruning could help optimize a full-scale HTM hierarchy.
Personally, I don’t think synaptic pruning needs a lot of investigation yet because hierarchies aren’t possible yet. It will be much easier to tell that it’s needed once hierarchies encounter problems. If there are never any problems for synaptic pruning to solve, that will save time.
I can understand your arguments regarding the secondary relevance of synaptic pruning in young brains, but do not share your certainty about the overall lack of relevance in this process. I agree that deep hierarchies are already formed in early childhood, but it is also true that these hierarchies do not yet support a more sophisticated degree of abstraction, as needed for higher cognitive tasks. Pruning is a gradual process and it really accelerates around age 6, exactly when these higher order tasks, start to become solvable. By the way, we are not only talking about a synaptic pruning but much more so, of a neuronal pruning. A very large portion of all neurons in the early brain self-destroy themselves via apoptosis (programmed cell death).
This is too significant and correlates with many crucial brain developmental stages, to simply be ignored, in my opinion. I have no evidence, but I personally do not beleive that the pruning phase affects the columnar connections within a mini-region (CC) as much as it does seem to give shape to the inter-regional connections, which build the hierarchies. I suspect that the hierarchies are already layed out prior to the pruning process, but the pruning eliminates a large series of unused or not needed hierarchical connections, which probably strengthens the remaining structures and perhaps allows them to further develop. If this holds true, then we probably also need to take account of additional usage-threshholds which determine which structures are kept and which ones need to be eliminated.
I would just like to say that these forums have been very welcoming and there is nothing like the pursuit of truth beauty and happiness with like minded curious individuals!
My initial reason for bringing up neuronal/synaptic pruning was to add to the HTM CLA model. Perhaps we can expand this discussion to include possible suspect processes. I was particularly intrigued by the work of Penrose & Stuart Hameroff on Fractal Consciousness and Orch OR.
The bubbling activity within neurons, specifically the microtubule structures represent a technical direction for consideration of long term memory encoding CaMKII, a sort of sparse memory that encodes at a different rate than the upper level activity. In the video, this is referred to as topological computing. I believe this expands the functional domain for computation of such systems. In the video by Hameroff, he outlines the algebras derived and other components. I was inspired by Jeff’s pursuit for biological inspiration, so I apologize if this is too much of a tangent.
microtubule: a microscopic tubular structure present in numbers in the cytoplasm of cells, sometimes aggregating to form more complex structures.
My team and I are investigating this for the purpose of integrating such automata into Nupic. This we believe will influence the probability skew and add a stochastic component to the synaptic harmonies, or perhaps even give rise to conscious machines! Just kidding !!
I don’t know a lot about neuroscience or cellular automata, so I’m a little confused. Could you give me more details about what you’re suggesting? What exactly do you mean by probability skew, and what would the role of the stochastic component be?
That type of pruning seems much more significant, so I agree that it could be crucial.
Based on a quick search, half of the cells die. I don’t know where they are located. If half of the cells in the neocortex die, that might be too many not to affect the connections within a CC. That could be quite useful if the activation dynamics change wildly, like they do during the early development of a HTM. Column state activation dynamics change a lot and columns stop bursting as much, so some cells will have a lot of useless dendritic segments. It might be useful to use extra redundancy and kill the cells which no longer function. For example, you could use 64 cells per column, usually with two cells on in each column, then kill half the cells to end up with typical HTM parameters and a selection of cells which had the right connections to best adapt to the changes in activation dynamics.
That doesn’t rule out pruning to affect inter-regional connections. By eliminating hierarchical connections, I assume you mean eliminating connectivity between regions, not just some synapses between each pair of regions. That could be useful in a couple of ways. If each level of the hierarchy starts with the same stability, higher levels might want inputs from very low levels so they can start training without relying solely on inputs which have passed through a bunch of untrained regions first, which might not convey the meaning of the input as well as lower regions. Another possible use for eliminating connections between entire regions is deciding how to do association between different senses. Two senses might need to be associated after one is at the third level of the hierarchy and the other is at the fifth level, for instance. This could change as the regions learn, as well.
Thanks for your response and your interesting insights about the possible effects caused by pruning. I personally am inclined toward your second paragraph above. And yes, I did mean “eliminating connectivity between regions” (not just some synapses). While the dynamics which you describe are very plausible, I think all of this hinges on the question regarding which neurons exactly are the ones which get pruned away (i.e. die). I have not yet found any evidence from empirical studies which clearly identify which neurons are typically pruned away in this process. It would be very useful to know what these pruned neurons all have in common. For example, are they mostly involved in connectivity between regions? Or are they mostly found within the CC regions? Or both in nearly equal parts? Then, irregardless of their locations, what traits or activity levels do they have in common? Knowing the original trigger for the pruning would be very helpful in understanding the effects and even the purpose, of pruning. Right now, all that I can say is that I suspect pruning is indeed very significant in the development of our brain, because it is a universal phenomenon in all healthy individuals and also, because nature does show us similar paradigms in other realms beyond neurobiology. In ecosystems the process of elimination in population regulation, does play a key role in shaping and stabilizing the system as a whole. And even natural selection relies on continous elimination of certain elements, which shapes and determines the traits of the adapted population. Maybe pruning also follows some of the principles of natural selection, and thereby allows the fittest structures to survive? I must say, that this thought is only based on my intuition, for I lack the empirical knowledge which could perhaps substantiate it.
Good idea and nice to know it was forked and not deleted. I had been wondering why the other thread had been set back in time. Perhaps a pointer could be set in the original threads when this is done. But I fully agree that we were drifting off-topic and into another interesting one. Thanks for keeping our thoughts well sorted.
This is just a short additional thought on the possible function/purpose for pruning: I think it is also very likely that energy conservation is one of the driving factors for the evolution of synaptic pruning. And also memory (or processing capacity) management, seems a likely purpose. In an analogy to computer programming, it is a sort of “one-time” garbage disposal, the way we do it in Java or other programming languages, when we wish to release memory used by un-needed variables. However, even if this holds true, the pruning triggers may still be very crucial to understand, for they may have a very fundamental influence on the generation and maturing of HTM across the regions.
Hi, thanks for the comment on the cell-death (apoptosis) taking place prior to birth. I would agree that this does change the role of apoptosis very significantly. May I ask what your source of information for this statement is? Because it was a long time ago, but I seem to recall reading about some very significant amounts of apoptosis (programmed cell-death) taking place between age 6 and puberty. I will also have to check my sources again. But thanks for the pointer and if you can mention a source, that would be very nice. Thanks.
Joe
I can’t remember the original source, but here’s one:
http://europepmc.org/articles/PMC3061193 - “About 50% of the neurons produced during neuronal development die by this process [apoptosis] before birth or shortly thereafter”.
According to https://en.wikipedia.org/wiki/Synaptic_pruning - “The infant brain will increase in size by a factor of up to 5 by adulthood. Two factors contribute to this growth: the growth of synaptic connections between neurons, and the myelination of nerve fibers; the total number of neurons, however, remains the same.” That seems to rule out massive neuronal apoptosis after birth, although it could be wrong.
!!
