I looked into MTJ and it does seem to be a good choice. As far as documentation goes, this javadoc from the project’s GitHub README seems to be about all that is available - which is fine. After reading through it, I was unable to identify any methods by which to concatenate matrices, or expand the size of an existing one, short of 1) Turning matrixA into a double[][] 2) creating a new, larger matrix, matrixB, and initializing it with zeros, and 3) copying the values from the double[][] into matrixB “by hand” (via loops and matrixB.set() method).
Concatenating the matrices by hand is fine, but I am concerned that doing so might do away with some of the built in efficiency of MTJ. Did you happen to run into this issue with your port of the KNNClassifier?
Yes I was going to send you some references to resources I found regarding MTJ yesterday but I got caught up running errands and couldn’t organize over mobile.
So I’m working out the Matrix library issue now. I had started using the Sandia wrapper around MTJ and I had made a jar file with only the classes needed to do lin alg ops and not the other “extraneous” stuff. Give me some time this morning to make sure I can’t just use the MTJ library proper? Sorry for the confusion, I hadn’t thought I’d need to resolve this in any prompt fashion - but if you’re going to use a lib - I think we should definitely “vet” it…
One of the issues I had also was the expansion of an existing matrix, I think I had resolved something in that regard but let me work with it this morning and get back to you with my conclusions?
I still can’t get the native BLAS stuff to load but it will work anyway (just not in the most optimized way). Anyway, this is the pared down (culled) jar file which you can use to interact with the matrix classes. The example uses the Factory to create a SparseMatrix - I added a method that demonstrates that additions are possible… Not sure about merging yet though…
Thanks @cogmission! I should have a chance to work with your culled jar tomorrow afternoon, and maybe later tonight. I’ll let you know how it works out for me.
No worries about confusion or delay (not that there was any). I don’t mean to rush or inconvenience you in any way!
I was able to set up your jar file without incident. The example was very helpful, and ran perfectly. Your addRow() method is a fantastic addition, and solves 1/2 of my matrix concatenation problem. Of course the main thing that the lib is now lacking is an addColumn() method. That would completely solve the problem of concatenating matrices, as I could just add however many columns or rows I needed.
Again, though, I don’t mean to intrude upon you in any way. I’ll just continue working on the SDRClassifier port and work around the matrix-specific logic for now, until the problem of merging (adding columns) is solved.
Thanks again for the assistance on the matrix library!
I haven’t formalized tests for these because they haven’t been “officially” added to the repo yet. Whoever merges their PR first (you for the SDRClassifier and me for the KNNClassifier) could also create a formalized test for this functionality and/or we can share the responsibility and collaborate. Or maybe that could be a separate (and easier) task for someone else who wants to contribute? I’m open…
Thanks for the fast response David! I’m currently working the Matrix lib into the SDRClassifier code. I see no reason why it shouldn’t work, but I can’t technically confirm that it does just yet.
I’d love to help work on formal tests for the Matrix lib functionality once I get some more work done on the SDRClassifier.
Hey @cogmission! I’ve been hard at work on the SDRClassifier port, and I can now “technically” say that the matrix lib works brilliantly! Thanks again
Thus far I have written 4 classes: SDRClassifier, SDRClassifierSerializer, SDRClassifierDeserializer, and SDRClassifierTest. I now have 21/21 unit tests for the classifier passing. I just need to write one more when I get a chance, probably this evening. Then, all of the unit tests will have been ported and passed in my Java port.
What else do I need to add/implement for full integration of the SDRClassifier into HTM.Java? Once I know, I’ll go ahead and make a pull request for my fork’s sdr-classifier-dev branch, and include one of those nifty checklist deals showing what’s left until a merge can occur
I don’t believe it yet has the unit tests committed to it. I’ll need to commit those this evening (I’m on my mobile atm). There are also a few bug fixes that need committing as well. But, you should be able to get a sense of the code style and what not, to see if it fits your standard. If not, please let me know how to improve it!
Congratulations! I’ll take a look at it once you get the PR up. I’m glad the matrix lib is working for you, you are actually the first to test it (in HTM.Java)!
I’ll wait for the PR before taking a look at the “style”, but I do have to mention one thing. All classes now are serialized/deserialized via the Persistence framework which basically only requires that the class implement Persistable and the framework will handle the serialization for you. There are Persistable.preSerialize() and Persistable.postDeSerialize() which are there in case objects of your class need special setup or teardown prior to or after persistence.
You will need to write tests that show that the SDRClassifier can be serialized and deserialized correctly. What this means is:
1.) Have 2 SDRClassifiers. (SDRC 1 and SDRC 2)
2.) Have inputs A, B, C (for SDRC 1), and A, B, C (for SDRC 2), which have outputs X, Y, Z and X’, Y’, Z’.
3.) Send the input A into both Classifiers and compare outputs X and X’. to show they are the same.
4.) Serialize SDRC 2; then deserialize it.
5.) Send the input B into both Classifiers and compare outputs Y and Y’. to show they are the same.
6.) Send the input C into both Classifiers and compare outputs Z and Z’. to show they are the same.
If that isn’t clear for you, then let me know and we talk about it more, ok?
Anyway, the Serializer and DeSerializer of the old CLAClassifier were remnants of serialization that took place before we added the Persistence framework, and should be removed. Sorry for the confusion on that. I meant to get around to it, but didn’t somehow…
Great work! Let me know if what I described isn’t quite clear…
I’ve written a unit test within SDRClassifierTest that performs the test you described. Unfortunately, the test fails with the following error:
java.lang.RuntimeException: java.lang.RuntimeException:
Class no.uib.cipr.matrix.sparse.FlexCompRowMatrix does not implement Serializable or externalizable
It seems to me the way to fix this would be to modify the FlexCompRowMatrix class in the matrix lib to implement Serializable? If you like, I can do that myself, if you send me the source for the lib (all I currently have is the JAR file).
Or, if you’d prefer to remedy the error a different way, I’m completely amenable to other suggestions!
Also, since you mentioned the Serializer and Deserializer classes are outdated, I take it there is no need to include them in my PR? I’ll go ahead and take them out, but just wanted to be sure…
Sorry for the delay, I was doing some research because of a trick I learned while implementing persistence which is that due to Java’s new “Intersection Types”, you can dynamically assign the marker type Serializable to a Lambda, like this:
Also, I didn’t state explicitly that the serialization has to be done through the PersistenceAPI as such:
SerialConfig config = new SerialConfig("some name", SerialConfig.SERIAL_TEST_DIR);
PersistenceAPI api = Persistence.get(config);
SDRClassifier sdrc = ...
// 1. serialize
byte[] data = api.write(sdrc);
// 2. deserialize
SDRClassifier serialized = api.read(data);
assertTrue(DeepEquals.deepEquals(sdrc, serialized)); // This test should be included too.
// Do the rest of your testing ...
But alas, not usable in this case because it must be assigned at construction time.
Though I must say, 
