Relationship between JDMP and WEKA?

[DrGary]

How do JDMP and Weka relate to each other? Does JDMP duplicate any WEKA functionality, or does it pass through its data structures to WEKA for the ML algorithms? I see a partial example in the forums that show WEKA being used to create a classifier. But the next step, 'use JDMP' to read a file, isn't documented. Does JDMP translate or copy from JDMP data structures to WEKA data structures? Or does WEKA use interfaces that JDMP supports?

Thanks!

[arndt]

JDMP duplicates some of Weka's functionality. E.g. we have our own algorithms for a Naive Bayes Classifier and neural networks, and other algorithms will follow. However, our intention is not to rebuild the machine learning packages out there, but rather provide a framework to combine them efficiently.
Data structures are converted from JDMP into a format that another package such as Weka can handle. If possible, wrappers are used and data is not copied. This works well for Weka's algorithms but not for its data structures, since Weka does not provide a clear interface/class division for that.

[jay]

Hi,

We don't really need another Weka, RapidMiner, etc... The biggest gap most open source&commericial products and all but a select few (far too few) tools is scalability around memory requirements. From your descriptions of JDMP and the underlying matrix library it looks like this is the or one of the main gaps you see this software filling. That is excellent! WE don't need another Weka or another matrix library which is simply a substitute to existing offerings asides for a few select features here and there.

The scalability hole is a large one. I am very excited to see this project progress. Having the underlying matrix toolskit (also scalable) should aid in community development.

At present are the modeling methods in the package also capable of work with larger than memory datsets?

Best regards,

Jay

[arndt]

When the basis of a data set is a matrix, more than memory data sets can be handled easily through UJMP.

In addition to that, it is possible to use a list of samples which can be swapped to disk, e.g. using the EhCache plugin.

More options will follow soon...