How to feed the simulator with input data at least I think I can describe with some clarity. Here’s some pseudo-code I sent Charles Simon when I was trying to code language sequences into his project:
public override void Initialize()
{
ifstream corpus;
corpus.open(“corpus.txt”)
std::string word;
std::string nextword;
while( !corpus.eof() ) {
corpus >> word; //I'll format my corpus as word per line to save the effort of splitting lines
corpus >> nextword
while(nextword != ".") {
void AddSynapse(string word, string nextword);
word = nextword;
corpus >> nextword;
}
}
Clearly this didn’t work as written. But perhaps you can see my logic in it.
I also have the working code which Charlie wrote from this. I guess I could share that here too. I think it is part of the public project. But perhaps the underlying logic might be clearer with this fragment despite my bad syntax anyway.
Then add inhibition to this network until a random activation on any node causes global oscillation. Just inhibit the whole network, and vary the intensity until a random activation does not immediately activate the whole network, or completely die out.
For the output. It’s just a raster plot at this stage. To envisage a hierarchy we might start with something like the clustering displayed in the Brazilian paper. Fig. 2:
Here, I’ve taken a screenshot:
In fig. 2 you can see that spike times cluster into four distinct synchrony patterns.
Do you see that the shared context (shared sequence with “a call” in the example in recent messages) might cause “pay” and “make” to synchronize as a “cluster”?
attention
/
pay
/ \
(?) a call
\ /
make
\
an effort
There is no hierarchy in fig. 2. But that’s because there is no hierarchy in the community structure being explored. In the network being explored in that paper there are just four distinct clusters. I imagine if there were sub-clusters within the four main clusters, that would be visible as similar distinct synchrony patterns within each of the main synchrony patterns.
Then, to build a parse tree you might look at submitted sentences (prompts) and break them into sub-sequences which form clusters like “pay” and “make” do above.
So, for instance, given a (prompt) sequence “Place wood blocks under the wheel” you might find “wood blocks” participates in a cluster which synchronizes in the context between “Place” and “under the wheel”. Which is to say, the synchronization clusters which form might suggest a structure:
Place (wood blocks) under the wheel.
And so on to build a full parse. Looking for sub-strings which form synchronization clusters like those in fig. 2, above:
((Place (wood blocks)) (under (the wheel)))
This is the sort of meaningful structuring which transformers are surely doing implicitly, but do not expose. Which makes it difficult to interpret or guide the things they say.
Having that structure should aid us to make predictions about the next word in the sequence. The next element in a sequence is the current transformer output. That’s all the output. Transformers just poke out a continuation of the sequence, and if you don’t like the continuation you get, you have no choice but to try another prompt. Given such a structure we could influence how the sequence continued. For instance, we might have some specific protocol for what to do after you place wood blocks under the wheel (this example comes from an old machine translation task in auto manufacturers manuals.)
Prediction would be… whatever is most highly activated in the network as a possible continuation. Or, the best continuation might be whatever forms the closest super cluster about the clustering above. I’m not sure at this point. But the information would be in the network. We would just have to extract it.
Does that clarify at all?