Add dataset loading capabilities

From what I gather, there’s no go-to utility to load a matrix / dataset from disk without resorting to the fromJSON function of DenseMatrix and SparseMatrix. It would be great if we had a clean dataset loading interface supporting .csvs, the MatrixMarket format (very popular in R), and others alike.

Thanks! I would be happy to discuss this further and/or lend y’all a hand

[josdejong]

Thanks for your offer 👍

Before implementing anything I think it will be good to write out the various use cases we see. That way we can see what kind of API makes sense. What pops in my mind:

• fromCSV and toCSV functionality.
• Is CSV format the same as MatrixMarket data type or do we need a separate fromMTX and toMTX?
• the functions can import/export data as string and as a ReadableStream (that’s great for .

We should be careful not to reinvent the wheel here, and embrace API’s and libraries that are already out there. The idea for a “simple” fromCSV(src) may turn out not to be so simple: you need to built in support for nodejs and the browser, people will need to be able to send authentication headers and CORS headers etc etc.

It would be nice if we can allow people to use their rest client of choice and embrace that instead of creating a full blown rest client ourselves. So usage can look like:

// in the browser (user can add credentials, tokens, headers, whatever):
const dataString1 = await fetch(url).text() // <-- people use their loading mechanism of choice
const matrix1 = fromCSV(dataString1)

// in the browser, streaming for large amounts of data:
const dataStream2 = fetch(url).body // <-- people use their loading mechanism of choice
const matrix2 = fromCSV(dataStream2)

// in nodejs:
const dataStream3 = fs.createReadStream(file) // <-- people use their loading mechanism of choice
const matrix3 = fromCSV(dataStream3)

[davidmrdavid]

@danielruss , I actually never got around it due to my year getting a little crazy. If you want to give it a stab, I’d say go ahead 😃