This paper on BDH (Baby Dragon Hatchling). It’s basically an attempt to replace the Transformer bottleneck with a scale-free graph of “neuron particles.”
The technical gist:
No KV Cache: Instead of storing context in a massive memory buffer, it uses synaptic states. It uses local Hebbian rules to update weights during inference.
Sparsity: It operates at ~5% activation. Everything is sparse and positive-only, which keeps it closer to SDRs (Sparse Distributed Representations) than traditional dense LLM vectors.
Graph Dynamics: It’s structured as a graph rather than a stack of layers. It uses an “integrate-and-fire” cycle (Firing → Competition → Update → Transmission).
Scaling: They managed to hit GPT-2 performance levels at 1B parameters. That’s the part that actually matters—it’s a biologically-plausible model that doesn’t fall apart at scale.
Interpretability: Because of the sparsity and local rules, the authors claim “monosemanticity.” You can basically trace a concept to a specific physical path in the graph rather than a high-dimensional mystery.
They’ve got a BDH-GPU implementation that maps these graph interactions into linear algebra kernels so it actually runs on current hardware.
The “Thermodynamic Limit” they mention actually prevents the local updates from diverging/exploding when you move past 1B parameters.
very interesting and promising development - found the fact very curious that you can just concat these models to produce a bigger one that combines knowledge of both parts into one model.
They seem to add a limited attempt at sparsity and a beginning of continuous learning (after classic pre-training with back propagation), but they still use point neurons.
OP’s message (@MTIzNDU2Nzg5) is completely AI generated. He left em-dashes, chatgpt arrows,.. You’re a disgusting misinformation bot.
Another forum written entirely by AI, and overhyping and lying about new papers. Pathway’s published BDH code is the gpu version, which is IDENTICAL to an SSM like Mamba, it is a regular recurrent state-space model. It uses gradient descent (Adam), with next-token prediction. Furthermore, weights during inference are static (NO CONTINIOUS LEARNING), there is no hebbian learning or plasticity. And the most shocking, there is no monosemanticity! Even worse, they never released the non-gpu version.
Their repo is filled with marketing BS. Even the ‘demos’ of it outperforming LLM’s at sudoko etc, are extremely unfair because the task was for a large language model to also understand spatial tasks, instead Pathway trained a regular DQN on a discrete task. The goal was AGI, instead we made a 10K params model with backpropagation trained only on that task. Genuinely every reinforcement learning architecture could outperform it.
