Is there a way to see the dataset size before starting the download

Hi,

Is there a way to get information about the dataset size (num sentences, etc) before downloading it? Is this available through the API somehow? Is this what cols is https://github.com/thammegowda/mtdata/blob/master/mtdata/entry.py#L101 ?

Thanks, Nick

[kpu]

There’s two ways to do this: a HEAD request to the URLs on the fly and a cached set of statistics about each corpus, to include number of segments etc. To have the cached version without downloading, effectively what you are asking for is a continuous release system that downloads stuff then puts metadata in the release. This continuous release system could also cache things and be branded OPUS…

[thammegowda]

I like the HEAD request approach and we can easily do that. An edge case I am concerned is, we have indexed many zip/tarball files that gets mapped to multiple datasets. I could show the overall tarball file size, though it is inaccurate, it would be a good start.

Also, as shown in my previous comment, mtdata stats outputs character counts. I will revise it to output byte count, and also as human readable size (kB, MB, GB etc). This will be more accurate but costly (we have to download dataset for once)

Also +1 for caching these stats and distributing as part of release. OPUS is the large source and fortunately precomputed stats are available from their API. Then for the remaining datasets, I have to run on one of our servers and collect stats overnight. For any new additions, we can rerun a script to update the cached stats and make it available in the next release.

---

So, to summarize, the action items (for me, for the next release):

• Show byte count and total size in the output of mtdata stats <DataID>
  • Cache the stats once we compute them
• Add mtdata stats --quick <DataID> option to perform HEAD request and show content-length header
• Preserve stats from OPUS – put them in cache. https://github.com/thammegowda/mtdata/blob/b1c0b21d3b58c0053b3a6fa669158f71b0c7f0a7/mtdata/index/opus/opus_index.py#L14-L17
• A script to update the stats that are missing in cache;
  • run this on a server with sufficient delays between requests
• modify release process to include the stats cache and automatically update stats for newly added datasets
• (Optional/good to have) stats in search and visualizations http://gowda.ai/mtdata/search