The key pixel problem will be fixed through large amounts of regularization.
MOAR DATA?
Not more data, more noise. Noise is good.
Can also use drop out which is just a form of noise injection.
1 Like
So what type of solution do you have in place that needs this fix? Just curious… what type of network are you using?
Not sure who you are asking but I will say our math folks always want strong regularization. For all input streams more generalize-able models are learned with regularization and less attack-able models. This may be less of an issue with sparse representations?
I’m not a math guy, can you describe what you mean by regularization?
The image is from wikipedia article on regularization with is pretty good.
If you use many terms in a polynomial you can get the over fit blue curve. If you use few terms in the polynomial you can get the green curve. This is not the ideal example in the ideal example the over fit curve would fit some points that the mild/slow/gentle curve does not fit. But the over fit fast/extreme/wildly changing curve is likely not a good interpolation for the points in-between the given data point nor for extrapolating to point beyond the given data.
Literally taking your data point and adding a small amount of noise (a random value selected from some probability distribution say a Gaussian centered around a small number relative to the data average) to each data point will regularize. That is will make it impossible for even the more complex function to exactly fit the data. It will instead have to go for a reasonable middle fit.
To check if you have over fit one generally divides that data set in two parts (say 60% and 40%). One part is used for fitting and the other is used to check the predicting ability of the fit. There are more fancy ways to do this making multiple data sets selected by random draw and checking the several fits to the several verification data set. Another method to avoid over fitting is to stop early. Do not train your model through all the epochs. There are an endless variety of way to weaken, break, gentle the fitting to avoid over fitting.
2 Likes