Bad Anomaly detection for complex periods data

Hi, I’m a user of nupic, and my job is anomaly detection dev.
In my situation, we have many complex periods data such as daily period/weekly period/monthly period, and when I tested the nupic algorithms with these data, it shows bad prediction and detected results.

I feed with 150 days data which span is minutes, after swarming and prediction, it’s result is below:


it’s unalarmed when it’s 04:00~04:10, which obviously is an anomaly events.
Is something wrong?

PS1:
my swarming config:

SWARM_DESCRIPTION = {
  "includedFields": [
    {
      "fieldName": "date",
      "fieldType": "datetime"
    },
    {
      "fieldName": "create",
      "fieldType": "int",
      "maxValue": 644395,
      "minValue": 0.0
    }
  ],
  "streamDef": {
    "info": "create",
    "version": 1,
    "streams": [
      {
        "info": "Rec Center",
        "source": "file://create.csv",
        "columns": [
          "*"
        ]
      }
    ]
  },

  "inferenceType": "TemporalAnomaly",
  "inferenceArgs": {
    "predictionSteps": [
      1
    ],
    "predictedField": "create"
  },
  "iterationCount": -1,
  "swarmSize": "large"
}
MODEL_PARAMS = \
{ 'aggregationInfo': { 'days': 0,
                       'fields': [],
                       'hours': 0,
                       'microseconds': 0,
                       'milliseconds': 0,
                       'minutes': 0,
                       'months': 0,
                       'seconds': 0,
                       'weeks': 0,
                       'years': 0},
  'model': 'CLA',
  'modelParams': { 'anomalyParams': { u'anomalyCacheRecords': None,
                                      u'autoDetectThreshold': None,
                                      u'autoDetectWaitRecords': None},
                   'clParams': { 'alpha': 0.04372980145038936,
                                 'clVerbosity': 0,
                                 'regionName': 'CLAClassifierRegion',
                                 'steps': '1'},
                   'inferenceType': 'TemporalAnomaly',
                   'sensorParams': { 'encoders': { u'create': { 'clipInput': True,
                                                                'fieldname': 'create',
                                                                'maxval': 644395,
                                                                'minval': 0.0,
                                                                'n': 502,
                                                                'name': 'create',
                                                                'type': 'ScalarEncoder',
                                                                'w': 21},
                                                   u'date_dayOfWeek': None,
                                                   u'date_timeOfDay': None,
                                                   u'date_weekend': None},
                                     'sensorAutoReset': None,
                                     'verbosity': 0},
                   'spEnable': True,
                   'spParams': { 'columnCount': 2048,
                                 'globalInhibition': 1,
                                 'inputWidth': 0,
                                 'maxBoost': 2.0,
                                 'numActiveColumnsPerInhArea': 40,
                                 'potentialPct': 0.8,
                                 'seed': 1956,
                                 'spVerbosity': 0,
                                 'spatialImp': 'cpp',
                                 'synPermActiveInc': 0.05,
                                 'synPermConnected': 0.1,
                                 'synPermInactiveDec': 0.09274388313941442},
                   'tpEnable': True,
                   'tpParams': { 'activationThreshold': 12,
                                 'cellsPerColumn': 32,
                                 'columnCount': 2048,
                                 'globalDecay': 0.0,
                                 'initialPerm': 0.21,
                                 'inputWidth': 2048,
                                 'maxAge': 0,
                                 'maxSegmentsPerCell': 128,
                                 'maxSynapsesPerSegment': 32,
                                 'minThreshold': 10,
                                 'newSynapseCount': 20,
                                 'outputType': 'normal',
                                 'pamLength': 5,
                                 'permanenceDec': 0.1,
                                 'permanenceInc': 0.1,
                                 'seed': 1960,
                                 'temporalImp': 'cpp',
                                 'verbosity': 0},
                   'trainSPNetOnlyIfRequested': False},
  'predictAheadTime': None,
  'version': 1}

How to set the timeofday and dayofweek?

Thx

1 Like

Please see my response on the new Don’t swarm for Anomaly models topic, but also, which line is the actual data being analyzed in the graph above? I see you are using @jeff’s visualization graph!

Great to see it being used! I haven’t had much time to attend to the viz tool since taking my recent job.

@rhyolight @jeff can I get this graph somewhere open source? looks super nice :blush::heart_eyes:

1 Like

We’ll have to ask the author @Leoysen_Liu.