I have created a knowledgebase/graph neural network architecture (GIAANN prototype) which can be used to predict the next token in a sequence. It trains a set of columns for every new noun encountered in a textual corpus, and feature neurons for every contextual word (non-noun) directly surrounding each noun. It supports distinctions in dendrite proximity (SANI; sequentially activated neuronal inputs), and is based on neural assembly and cortical column theory.
It is currently citing “Hawkins, J. et al. (2011). Hierarchical Temporal Memory (HTM) Whitepaper (Version 0.2.1). Numenta”, although they may have more recent preferred citations.
Originally posted on Column/assembly graph neural network - Research and Theory - Thousand Brains Project
3 Likes
I created something along these lines a few years back (Jun 2022 post of an image you may understand) and after some distraction I have been workiing on it again more recently. I may be able to help.
This is a quick grok (4.3) summary from an older theoretical and code situation :
Overall Assessment
GIAANNpy is a strong, practical cortical-column + SANI-focused prototype that captures important pieces of the theory (columnar hierarchy, sequential/temporal sensitivity, dynamic local learning, no backprop). It would integrate well as a “cortex layer” in a larger system.
However, it is missing or under-emphasising the hippocampal consolidation engine (ripples → sequential-to-parallel/abstract transformation) and the basal-ganglia attention-pool layer (theta persistence, selection/commit, long-range state holding) that the theory identifies as critical for scaling to general intelligence, efficient abstraction, and human-like long-temporal-span reasoning.
The Cerebral codebase already implements more of the distributed memory + question protocol side; combining the two (Cerebral-style chunk orchestration + GIAANNpy-style SANI columnar processing + an explicit sleep/ripple consolidation pass + BG-style attention gating) would be a natural next step toward the fuller architecture we have been discussing.
I think you are closer to the right track on some aspects than traditional HTM approach, but there is an element of compression that more dense networks acheive, which the fire and wire forward approach does not necessarily achieve. Within my setup I embeded timing within the network to allow for sparse compute for signal propogation calcs, which reduced the overhead significantly, which in turn made it more random memory based and not tensor style matric math. I scaled up to 20bn in around 400GB across 15yr old servers with 20gb mellanox, all eBay bargains at the time for a good reason…
I believe if you look at language from a temporal dimension (without the academic labelling overhead), multi modal sensory input just runs within that whole stream. I’d be interested in providing you with all the research.
2 Likes
Hi BrainVx, thanks for reviewing the release.
I have noticed that your tech is capable of bringing together co-occurring concepts, which may be necessary for generalisation beyond named types. Although it is not clear from your diagram whether these hierarchical concepts are real nodes or merely representations of temporal distortion, GIAANN intentionally removed all higher level node assignment from its earlier prototype HFNLP (“SANIconceptNeurons”), as its design is limited by the total number of available neurons in the neocortex for language processing (~1B). By only assigning one neuron per noun (concept feature/column), it significantly reduces the total substrate/soma requirement. It can do this while still allowing the non-noun feature substrate to be contextually dependent.
More complex concepts are maintained in memory as a function of the current activation state of the network. It is expected that some level of test-set generalisation to novel phrase use will be possible as an emergent property of the shared network connectivity between similar concepts (words). It likewise was designed to maintain concepts as an effective bundle of words, rather than learning dense word vector expressions of these (as is common in contemporary artificial neural nets). However this poses potential issues for like-word generalisation (synonyms). A better test-set eval regime beyond next-token prediction is required to properly evaluate this issue; a) which is more suitable for whole-word tokenisation, and b) is capable of distinguishing between justifiable prediction and hallucination (i.e. making either false predictions or correct predictions without sufficient grounding within the training-set).
Regarding some of the other systems you mentioned, the current prototype can be thought of as a learning algorithm simulation of specifically the explicit language system in neocortex. For example, GIAANN implicitly assumes the presence of an inhibitory network in the implementation of its top-k feature selection within/across columns. There are various articles that either suggest or support inhibition as a mechanism for this purpose (more recently even; https://www.biorxiv.org/content/10.64898/2026.04.23.720412v1).
1 Like
@richardbaxter I see that this work is in a larger context of an algorithm called GIA and a organization called BAI.
Can you explain a little bit about that previous work and your overall mission?
1 Like
Sure; BAI (originally Baxter AI, now BAI Research Pty Ltd) is an Australian artificial intelligence research startup, primarily focused on biological constraints and performance (i.e. biological AI algorithm research). General Intelligence Algorithm (GIA) is an early graph network approach to language modelling, which grew up in a similar environment to OpenCog (2011+). GIAANN is a recent neural network implementation of GIA, that maintains a number of core biological constraints (substrate capacity, excitatory neurons, columnar structure, graph networking, segmented/dynamic dendrites etc), and exhibits a number of core biological features (concept organisation, low training samples, online learning, continual learning, free context windows, hebbian learning, robustness to hallucination, etc).
1 Like