2D MNIST Encoder

Engineering Machine Learning
6

Ok, is a bit of talk. Let’s say there are three stages, their names sound cryptic but I need to explain each stage.
In reality the execution order is different, here is the… “semantic” order. 1. is the last stage to encode the whole dataset, 2. prepares the encoder and 3. computes “balancing weights”

  1. Running a dense matrix projection of the image pixels.
    With a random matrix RM of shape e.g. 28x28x1000 floats applied to any 28x28 image it would produce a 1000 bit “Fly Hash” SDR . Algorithm is simple - for every pixel in the input image e.g. row 5x col 7 I multiply its value with the RM[5,7] and add the result to a total buffer.
    That is simply a dot product between image and RM
    The resulting 1000 floats vector I multiply it with “ballancing vector” (see 3. below) then select the highest P values e.g. 50 for 50/1000 resulting sparsity.
    T
  2. but, in order to make it sensitive to position in 2D (28x28) input space I first start with a 34x34x1000 random matrix and convolve every 7x7x1000 submatrix into a single 1000 bit line in the the final 28x28x1000 marix.
    So instead of applying convolutions to every image I convolve the encoder itself. Then use it as in fly hash (random projection) encoding described above. This makes adjacent lines in the RM more “close” to each other. I haven’t tested much with different size kernels, 7x7 seemed quite lucky :stuck_out_tongue:
  3. Final preparation stage I can call it “VU-meter balancing” this uses a subset of the dataset to figure out which values in the 1000 long dense vector need to be “stretched” or “compressed” in order to avoid having some bits in the SDR being over-represented and other under-represented. So if total activations average 100 / each sdr bit and some bit activates 125 times other 70 times these weights are adjusted so the have more even chances (each activating approx 100 times)
    This balancer Is ran on only 5000-10000 images once before actual encoding. The number is picked so the number of samples is significant yet, being quite costly, doesn’t impact too much overall encoding time.

I hope it makes sense.