I’m working on some visualizations, and I wanted to talk about the duty cycle period for determining whether columns in the Spatial Pooler should be “boosted” (whether Homeostatic Regulation should be applied (thanks @ycui 
)).
Here is some context for this discussion if you don’t know what a “duty cycle period” is.
I’m trying to understand intuitively what it means to change the size of the duty cycle period. The default in NuPIC is 1000. But how would you characterize the effects of doubling the size? Or halving it?
Any discussion is much appreciated. 
PS: Yuwei’s new boosting code is still in research, but it will be integrated into NuPIC at some point. You can find it here: https://github.com/ywcui1990/nupic/blob/boostingRes/src/nupic/research/spatial_pooler.py#L1301-L1347
2 Likes
Here is how I think of it.
The input to the spatial pooler is a large high dimensional space. You can think of it as a big three dimensional blob and still get an intuition. Every input to the SP represents a point in that space, a point somewhere in the big blob. Every column in the SP also represents a point in the blob space. There might be 2000 of these “column points”. When a new input arrives the SP finds the 40 column points that are nearest to the input point. Voila, those 40 columns are the output of the spatial pooler.
A random SP distributes the column points randomly in the blob making sure all parts of the blob are covered equally. No matter where an input point is there will be 40 column points as nearby as any other input point. However, input points in the real world are unlikely to be distributed equally in the blob. There might be a big cluster of input points in one spot and vast deserts where no input points occur in other places in the blob.
When the SP learns it moves the winning column points closer to the input point. They might be heard saying “let’s move closer to the input point so next time we are sure to recognize it again.” This is good but when the column points move they make other parts of the blog less well covered. Think of it like placing Starbucks. In rural Nevada you space the Starbucks 20 miles apart, but in Reno they might be just across the street. When the SP learns, it slowly moves the column points to where the input points tend to be clustered.
Ok, if you got that image in your head we can discuss the the duty cycle of boosting. The duty cycle controls how patient the column points are before they start moving. Imagine if you were a Starbucks in the Nevada desert. If you haven’t had a customer in 5 days you say, “screw it, I am going to start dragging my shop towards Reno and take some of the Reno customers, I am tired of waiting”. If I increase the duty cycle from 5 to 10 days you will be more patient and wait 10 days before starting to move. Decrease the duty cycle and you will start moving before 5 days. There is a downside to moving your shop, when a input point (aka customer) does shows up in the desert they wont be near many Starbucks. That’s ok but the SP will not be able to distinguish two customers in the desert who might be 100 miles apart, whereas it can tell the difference between Reno customers that are just 100 feet apart.
In summary, the SP divides up the input space. Learning changes how the space is divided to better match the distribution of the observed input points. The more clustered the inputs are the more the columns move towards the clusters. Boosting decides how patient the SP is before moving the column points. If you really care about distinguishing the rare customers in the desert then make the boosting number big. If you care more about distinguishing the Reno customers and don’t mind lumping all the desert customers together, make boosting small. There is no universally correct answer to what should the boosting number be. It depends on the cost of missing the distinction between desert customers vs the cost of missing the distinction between Reno customers. If the desert customers are big spenders but only if you get their order right and Reno customers don’t seemed to mind being grouped together then make the boosting number high. You may have to wait a few days, but the payoff is big when the desert cowgirls and cowboys do walk through the swinging saloon door.
11 Likes