# Getting an error about RandomDistributedScalarEncoder

**URL:** <https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669>\
**Category:** NuPIC\
**Tags:** question\
**Created:** [March 21, 2018, 9:59pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669 "2018-03-21T21:59:52Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![JustAnEngineerO](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/justanengineero/32/3323_2.png) [@JustAnEngineerO](https://discourse.numenta.org/u/JustAnEngineerO)\
**Post date:** [March 21, 2018, 9:59pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/1 "2018-03-21T21:59:52Z")

</div>

Hey guys, can someone sanity check our model params here? We got these from NAB, before I make an issue in github, are these right?

```
{  
   "aggregationInfo":{  
      "seconds":0,
      "fields":[  

      ],
      "months":0,
      "days":0,
      "years":0,
      "hours":0,
      "microseconds":0,
      "weeks":0,
      "minutes":0,
      "milliseconds":0
   },
   "model":"HTMPrediction",
   "version":1,
   "predictAheadTime":null,
   "modelParams":{  
      "sensorParams":{  
         "sensorAutoReset":null,
         "encoders":{  
            "value":{  
               "name":"value",
               "resolution":0.007692307692307693,
               "seed":42,
               "fieldname":"value",
               "type":"RandomDistributedScalarEncoder"
            },
            "timestamp_dayOfWeek":null,
            "timestamp_timeOfDay":{  
               "fieldname":"timestamp",
               "timeOfDay":[  
                  21,
                  9.49
               ],
               "type":"DateEncoder",
               "name":"timestamp"
            },
            "timestamp_weekend":null
         },
         "verbosity":0
      },
      "anomalyParams":{  
         "anomalyCacheRecords":null,
         "autoDetectThreshold":null,
         "autoDetectWaitRecords":5030
      },
      "spParams":{  
         "columnCount":2048,
         "synPermInactiveDec":0.0005,
         "spatialImp":"cpp",
         "inputWidth":0,
         "spVerbosity":0,
         "synPermConnected":0.2,
         "synPermActiveInc":0.003,
         "potentialPct":0.8,
         "numActiveColumnsPerInhArea":40,
         "boostStrength":0.0,
         "globalInhibition":1,
         "seed":1956
      },
      "trainSPNetOnlyIfRequested":false,
      "clParams":{  
         "alpha":0.035828933612158,
         "verbosity":0,
         "steps":"1",
         "regionName":"SDRClassifierRegion"
      },
      "tmParams":{  
         "columnCount":2048,
         "activationThreshold":13,
         "pamLength":3,
         "cellsPerColumn":32,
         "permanenceDec":0.1,
         "minThreshold":10,
         "inputWidth":2048,
         "maxSynapsesPerSegment":32,
         "outputType":"normal",
         "initialPerm":0.21,
         "globalDecay":0.0,
         "maxAge":0,
         "newSynapseCount":20,
         "maxSegmentsPerCell":128,
         "permanenceInc":0.1,
         "temporalImp":"cpp",
         "seed":1960,
         "verbosity":0
      },
      "tmEnable":true,
      "clEnable":false,
      "spEnable":true,
      "inferenceType":"TemporalAnomaly"
   }
}

```

Here is the error we are getting:

```
File "/opt/anomaly/anomalymodel.py", line 198, in process_sample
    result = self.htm_model.run({"value": sample.value, "timestamp": sample.timestamp})
  File "/usr/local/lib/python2.7/site-packages/nupic/frameworks/opf/htm_prediction_model.py", line 429, in run
    self._sensorCompute(inputRecord)
  File "/usr/local/lib/python2.7/site-packages/nupic/frameworks/opf/htm_prediction_model.py", line 514, in _sensorCompute
    sensor.compute()
  File "/usr/local/lib/python2.7/site-packages/nupic/engine/ __init__.py", line 433, in compute
    return self._region.compute()
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/engine_internal.py", line 1499, in compute
    return _engine_internal.Region_compute(self)
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/regions/PyRegion.py", line 184, in guardedCompute
    return self.compute(inputs, DictReadOnlyWrapper(outputs))
  File "/usr/local/lib/python2.7/site-packages/nupic/regions/record_sensor.py", line 377, in compute
    self.encoder.encodeIntoArray(data, outputs["dataOut"])
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/multi.py", line 100, in encodeIntoArray
    encoder.encodeIntoArray(self._getInputValue(obj, name), output[offset:])
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 250, in encodeIntoArray
    output[self.mapBucketIndexToNonZeroBits(bucketIdx)] = 1
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 233, in mapBucketIndexToNonZeroBits
    self._createBucket(index)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  <... many copies of this line ...>
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 267, in _createBucket
    self._createBucket(index+1)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 263, in _createBucket
    index)
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 299, in _newRepresentation
    not self._newRepresentationOK(newRepresentation, newIndex):
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 358, in _newRepresentationOK
    if not self._overlapOK(i, newIndex, overlap=runningOverlap):
  File "/usr/local/lib/python2.7/site-packages/nupic/encoders/random_distributed_scalar.py", line 402, in _overlapOK
    if overlap <= self._maxOverlap:
AttributeError: 'RandomDistributedScalarEncoder' object has no attribute '_maxOverlap'
```

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [March 21, 2018, 10:26pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/2 "2018-03-21T22:26:38Z")

</div>

I don’t understand how this error could be happening. The RDSE initializes its `_maxOverlap` attribute inside its ` __init__ ` function. I don’t understand how this attribute could be removed unless it was manually deleted.

Can you share `/opt/anomaly/anomalymodel.py`?

---

<div class="post-metadata">

**Author:** ![JustAnEngineerO](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/justanengineero/32/3323_2.png) [@JustAnEngineerO](https://discourse.numenta.org/u/JustAnEngineerO)\
**Post date:** [March 21, 2018, 10:27pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/3 "2018-03-21T22:27:17Z")

</div>

Yeah…I looked at the code too. \_maxOverlap is hardcoded to 2 in the constructor! We duplicated what you guys did in the NAB implementation. Sorry about some of this messy code!

Here is `anomalymodel.py`

[https://pastebin.com/d6x21rWQ](https://pastebin.com/d6x21rWQ)

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [March 21, 2018, 10:41pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/4 "2018-03-21T22:41:21Z")

</div>

Thanks. Does this script choke immediately on the first row of data processed? If so, can you share a portion of your data so I can try to replicate the error?

---

<div class="post-metadata">

**Author:** ![JustAnEngineerO](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/justanengineero/32/3323_2.png) [@JustAnEngineerO](https://discourse.numenta.org/u/JustAnEngineerO)\
**Post date:** [March 21, 2018, 10:44pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/5 "2018-03-21T22:44:14Z")

</div>

It happens way down the line, it doesn’t choke on the first row. We will try to duplicate it again and get you the data. Perhaps a serialized model was corrupted.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [March 21, 2018, 10:47pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/6 "2018-03-21T22:47:08Z")

</div>

> [@JustAnEngineerO](#):
>
> Perhaps a serialized model was corrupted.

We have received a couple of reports about this recently, including:

> <https://github.com/numenta/nupic-legacy/issues/3820>
>
> Hi!
> 
> I'm feeding data to a model in small batches, saving the model to disk at… the end of each batch, and loading it again for the next one. After a few batches, the model stops working and throws the following error when calling \`model.run(input)\`:
> \`\`\`
> Traceback (most recent call last):
> File "./anomalies.py", line 63, in \<module\>
> result = model.run(input)
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/frameworks/opf/htm\_prediction\_model.py", line 448, in run
> inferences = self.\_anomalyCompute()
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/frameworks/opf/htm\_prediction\_model.py", line 696, in \_anomalyCompute
> self.\_getAnomalyClassifier().compute()
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/engine/\_\_init\_\_.py", line 433, in compute
> return self.\_region.compute()
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/bindings/engine\_internal.py", line 1499, in compute
> return \_engine\_internal.Region\_compute(self)
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/bindings/regions/PyRegion.py", line 184, in guardedCompute
> return self.compute(inputs, DictReadOnlyWrapper(outputs))
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/regions/knn\_anomaly\_classifier\_region.py", line 326, in compute
> self.\_classifyState(record)
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/regions/knn\_anomaly\_classifier\_region.py", line 405, in \_classifyState
> self.\_addRecordToKNN(state)
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/regions/knn\_anomaly\_classifier\_region.py", line 490, in \_addRecordToKNN
> knn.learn(pattern, category, rowID=rowID)
> File "/home/dani/.local/lib/python2.7/site-packages/nupic/algorithms/knn\_classifier.py", line 537, in learn
> inputPattern = numpy.dot(self.\_vt, inputPattern - self.\_mean)
> ValueError: operands could not be broadcast together with shapes (65536,) (0,)
> \`\`\`
> 
> Here's the code used to load and store the model:
> \`\`\`python
> with open(model\_file, 'r') as f:
> model = HTMPredictionModel.readFromFile(f)
> \`\`\`
> \`\`\`python
> with open(model\_file, 'w') as f:
> model.writeToFile(f)
> \`\`\`
> 
> I've tried using a model generated from a previous batch and skipping some batches of data, to find out if it was the data that was somehow generating a bad model, but after the same number of batches, no matter their contents, I get to a broken model again. Thus, I suspect a bug is being triggered at \`readFromFile\` or \`writeToFile\` (or maybe I'm just doing it wrong).
> 
> This is with Python 2.7.9, and nupic 1.0.3 from pypi.

Does it _only_ happen after model de-serialization? It could be there is a bug that does not create the RDSE’s `_maxOverlap` attribute.

---

<div class="post-metadata">

**Author:** ![white](https://avatars.discourse-cdn.com/v4/letter/w/eb8c5e/32.png) [@white](https://discourse.numenta.org/u/white)\
**Post date:** [March 22, 2018, 2:40am UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/7 "2018-03-22T02:40:17Z")

</div>

The method `writeToCheckpoint/loadFromCheckpoint` of model is used in the source code. Why not use `save/load` of model?

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [April 13, 2018, 3:45pm UTC](https://discourse.numenta.org/t/getting-an-error-about-randomdistributedscalarencoder/3669/8 "2018-04-13T15:45:41Z")

</div>

This issue should now be solved in NuPIC 1.0.4. Please update and try again?