[BUG] getting started `03-Training-with-TF` nb gives OOM on a 16 GB GPU

Describe the bug

I am getting OOM issue when I train 03-Training-with-TF nb on a 16 GB GPU, and this can be problematic for users who are running these notebooks in the cloud with GPU memory sizes.

I can avoid OOM if I comment out os.environ["TF_MEMORY_ALLOCATION"] = "0.7" line. It works fine then.

Steps/Code to reproduce bug

Run 03-Training-with-TF to repro.

Expected behavior Should run without OOM. I recommend to remove os.environ["TF_MEMORY_ALLOCATION"] = "0.7" line from the example nb.

Environment details (please complete the following information):

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
• Method of NVTabular install: [conda, Docker, or from source]
  • If method of install is [Docker], provide docker pull & docker run commands used

I am using merlin-tensorflow-training:22.04.

[rnyak]

@EvenOldridge and @karlhigley My understanding is thatos.environ["TF_MEMORY_ALLOCATION"] = "0.5" is currently the default behavior under the hood (because of configure_tensorflow()) if we use NVT Keras dataloader. So I dont think we need to define it again in the notebook.

[karlhigley]

Which example is this for? Let me see how it goes on an 11GB GPU. 😺