I’ve been exploring some ideas about locality sensitive hash based neural networks.
https://archive.org/details/atlas-lsh-neural-networks-an-intuitive-overview
You can click on “uploaded by” for more information.
Allow me to add some music:
https://youtu.be/sWblpsLZ-O8?si=wUvrHzpvJmbsf3Rd
I don’t know if Direct Feedback Alignment (DFA) would work with that kind of neural network.
If it did it could be a biologically plausable system.
For a given input x the Atlas looks up a route through the brain and lights up those areas. Then the input data x flows through the lit up regions to produce the final result.
If DFA could function then you don’t need a backpropagation chain. The error signal can be sent to each region directly. It could even be that a dose of dopamine could act as DFA.
I think in a biological system the Atlas would be evolved rather than random.
A professional researcher would be loath to provide a pretty picture of an idea. As a hobbyist I have more freedom.
https://archive.org/details/atlas-lsh
The thing I was worried about was generalization. Which I just tried in code a few minutes ago.
I think okay, a bit noisier I would say than a standard neural network. I tested with very out of distribution test time data though, so you would expect switching noise effects.
I did have to use weight decay, but that is normal.
If you get use to LSH ideas, Atlas LSH neural networks are an actual simplification of standard neural networks with more design options.
Basically, I would say a new type of neural network is available. A mind is born. Like the song title.
I can provide you with a chatGPT assisted summary, if you don’t mind.
Atlas LSH Neural Networks: Geometry as Context
Atlas LSH neural networks begin by taking a compact geometric sample of the input using locality-sensitive hashing (LSH). Each LSH bit can be viewed as asking a simple geometric question about the input vector—for example, which side of a randomly oriented boundary it lies on. Together, hundreds of these bits provide a sparse fingerprint of where the input sits in high-dimensional space.
These bits are not intended to describe the input directly. Instead, they provide context. Similar inputs tend to produce similar bit patterns, so the LSH acts as a geometry sampler that captures the approximate neighborhood of the input.
The network then uses this geometric context to decide which parameters or memories should participate in the computation. Rather than activating every weight, it performs conditional computation by routing information through the subset of parameters associated with the current geometric context.
One way to think about this is as an interaction between propositions and memory. The LSH bits represent propositions about the input’s location in state space, while the selected parameters represent memories associated with those propositions. Computation becomes a sequence of context-dependent memory lookups rather than a uniform application of every parameter.
Whether this proves to be a useful direction remains an empirical question, but it offers a simple conceptual picture: sample the geometry, infer the context, select the memories, and compute.