Questions wrt training on TPU Pod

Hi Accelerate Team,

I’m looking to use run_mlm_no_trainer.py on TPU v3-128 pod. I have few questions before I want to get started with the process.

1. Can I stream the data directly GCP bucket, or should I have to download the data to the VM where I’m training?
2. Does accelerate library support training on TPU pod or is it limited to 8 cores? based on this #471
3. Should I be using TPU node or TPU VM for better performance with accelerate library?
4. Is there a notebook or blog to help setup environment and run small tests on GCP VM for TPU training?
5. I want to train with dataset of order 1TB, will hf datasets be able to handle this data on a machine wth 256GB RAM? (possibly a Q for datasets repo, but just trying here as well)

Thanks in Advance

cc : @sgugger @muellerzr

[muellerzr]

To help us properly gauge the need for this feature, if you are actively trying to train on a TPU pod with PyTorch could you react with a 👍 to this message? 😄

Thanks!

[jianguoz]

@Ontopic @sumanthd17 Hi there, please react above message with a 👍🏻 if you want to train models on more than 8 TPU cores in future