Note about the memory effects seen when you use linear mappings as associative memory.
Prior memories can hang around for a long time, including random memories programmed in by random initialization.
That’s because storing new memories only slightly damages prior memories, then it’s just a hit count of how many new memory training examples have damaged a prior memory and by how much.
https://archive.org/details/linear-associative-memory-dynamics-and-ccslm
The online linear regression people already know the math of that.
Perhaps though are less aware of the mechanistic interpretations.
One application is concatenated context selected linear mappings: https://sciencelimelight.blogspot.com/2026/09/an-introduction-to-concatenated-context.html
A person with a lot of knowledge in the linear area is Andrew Saxe. And in one paper he discovered an adjacent system to CCSLM: https://arxiv.org/pdf/2207.10430
Trajectory remembrance in linear associative memory trained with SGD:
https://archive.org/details/trajectory-remembrance-in-linear-associative-memory
Basically linear associative memory will remember (fade memory) the route it took from its initial weight vector direction to the final weight vector direction after fitting the data. That is because forgetting stops when the data is fitted, there is no incentive for the weight vector to change further.
You should put your name on the PDFs you are publishing. Right now they’re just anonymous documents. Although I see the metadata is put into your archive.org post. It’s just not in the doc itself.
I just dont want to lose track of who wrote this stuff and when, if i ever come back to it.
1 Like
I’ll leave it neural. They are tech. notes with some ideas to think about.
There is also no date mark.
I don’t disagree with you, it’s just I was repeated accused of egotism and self-promotion when I did put my name on documents with concepts or ideas I felt motivated to share.
I have finished a line of thinking about concatenated context selected linear mappings. And obviously linear system learning applies to concatenated linear mappings, hence the renewed interest.
For ReLU neural networks:
ReLU(Wx)=DWx where D is a diagonal decision matrix with 0 or 1 entries depending on the corresponding ReLU decision. You are lifting the ReLU decisions out as a context channel and placing them in D.
The main data channel only ever sees D’s and W’s, the main data channel is only ever is processed by linear mappings. That is a remarkable restriction.
You can fuse D by basic matrix multiple with either adjacent weight matrix say, L=DW₁ or L=W₂D.
Then L is the linear mapping chosen by the context at a particular layer. Since D=DD you can play with the equations a little for fun, you just get a bunch of different ways of saying the same thing.
Each linear mapping L at each layer, under the pressure of backpropagation becomes a expert for its context chosen role.
Separating ReLU neural networks into a main data channel and a context channel gives a very clean way to understand how they work and allows major design variations.
That is a classic problem with linear associative memories, and your note hits the nail right on the head. When new memories only make tiny updates to existing matrix entries, noisy or randomly initialized legacy patterns linger forever and clutter up new data. I faced a very similar memory interference issue when building my project, BrainStem V8, and had to implement a few biologically inspired control loops to fix it.
Without active forgetting and homeostatic scaling, linear mappings almost always get overwhelmed by legacy noise. In BrainStem, we do not rely on passive overwrite to clean things up. Instead, we use active pruning based on explicit stability and evidence metrics like hypothesis stability scores, evidence count, and conflict count to actively sweep out unproductive patterns.
We also run cybernetic sleep and consolidation cycles inspired by slow wave sleep to handle synaptic downscaling. During these phases, patterns face selective pressure where strong, stable patterns get reinforced while weak background noise is systematically scaled down or wiped. On top of that, dynamic neuromodulators like GABA apply inhibitory pressure to stop uncontrolled activation, while dopamine and BDNF scale plasticity rates based on actual learning success rather than updating at a rigid, constant rate.
Since BrainStem is already running in Python, it might serve as a helpful architectural reference if you are looking for ways to keep associative memory structures balanced and noise free. Always down to swap ideas or chat more about memory consolidation if you want to connect!
Weight decay is commonly used in artificial neural networks as something that works without really knowing why. It is kind of drastic and global though.
I like your reinforcement idea, it sounds like a more efficient way of doing the same thing.
I find it very interesting that linear associative memory when used under capacity has response trajectory from where it was born to where it fully fits the data and freezes in place. Even though where it freezes is not a minimum norm solution.
That is a great connection regarding weight decay. Replay definitely acts as a more targeted, context-aware alternative to global weight decay because it selectively reinforces useful patterns instead of blindly shrinking everything.
Regarding that freezing trajectory in linear associative memory, that is actually a key difference in how BrainStem operates. Nothing in my system ever permanently freezes in place. Instead of settling into a static state, the system uses adenosine homeostasis and sleep replay for systematic downscaling.
This active downscaling systematically dampens energy and prunes unproductive noise across cycles. It prevents the state space from getting locked into rigid, non-optimal plateaus, ensuring the network maintains its plasticity and can continuously adapt to new context shifts without getting stuck.
Okay, that’s the dumbest accusation. Who cares? This is someone who is feeling inadequate and lashing out trying to bring you down too.
Regardless, if researchers are going to use you work, they absolutely require your name and date. This is not optional. We absolutely need to be able to properly cite where information comes from. By not putting your name and date on your notes, you’re making it more difficult for me because i need to make extra notes so your name is tracked in case I ever use it in the future for something to publish.
So really, please, put your name on your stuff. I have documents that have shown up on this forum from years ago that are no longer available, but the name and dates are properly attached so I can give proper citations. Those old half-formed notes from people on this forum have more relevance to me now than they did 8 years ago. You never know how your work is going to be used in the future.
I understand your position. Likely I will continue with the public domain extra approach though.
I found a student paper on arxiv that aligns with the technical note I made:
Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking