I did try something similar to that–combining the invariant scalar label encoding with the images. I haven’t done much testing on its ability to recognize new digits, since what I’m actually interested in is whether the system can take the scalar label and produce an image to match. Image generation rather than image recognition. (See my last post: Input-invariant representations )