Add sgkit-tskit repo?

Anyone object to us adding tskit as an import format, so we have an sgkit-tskit repo? I’m happy to do the coding here, and I think it’ll be a useful way to crystallise our general data import strategy.

We could also do export for tskit, in principle, using tsinfer but I’m not sure there’s much point.

[tomwhite]

In terms of parallelism by variants, this is definitely a weakness in the tskit API at the moment, we want to add some way of doing this well.

BTW I pushed another commit to https://github.com/pystatgen/sgkit/compare/main...tomwhite:ts_to_zarr for reading genotypes from a TreeSequence in parallel using the approach I sketched out above. It passes tests, but I don’t know how well it performs for larger datasets. It’s probably worth having both the sequential and parallel versions for the moment.

[jeromekelleher]

Thanks a million for this @tomwhite, it’s super helpful. I’ll take a good look at the code tomorrow.

In the short term, I think the simplest thing is for me to create a standalone “ga4gh-variant-sim” repo or something, where we just put the code to do this. We’ll want to add extra fields and stuff that are extrinsic to the tree sequence, so it’s as easy just put all the code in one place for now. I might make a start on this tomorrow, and maybe we could add your code for doing the translation to sgkit in there? Over time, we can see what a more mature interface might look like, and perhaps add it to tsconvert.

In terms of parallelism by variants, this is definitely a weakness in the tskit API at the moment, we want to add some way of doing this well.