[Discussion] Saving the field object

Usage: The field object is critical to checkpointing as it provides:

• tokenization
• padding
• numericalize

Having the ability to save the field object allows the user, given arbitrary text, to preprocess the text. The preprocessed text is then used with a checkpointed model. Then the output is predicted and interpreted without the output dictionary.

Problem: torch.save is implemented with pickle. The field object accepts lambdas for tokenization, preprocessing and postprocessing; therefore, cannot be pickled.

Key Points:

• The vocab object needs to be pickled because the output of the model is uninterpretable without it.

Discussion: What is the right abstraction here? Should the vocab object be saved and the field object discarded? Is it appropriate to have the field object and the vocab object closely bound?

[leimao]

It is nearly 2020, so what is the official solution now? Thanks.

[PetrochukM]

This repository has stagnated since the main contributors @jekbradbury and @nelson-liu left the project. Due to the lack of contribution and stagnation in torchtext, the NLP open source community fragmented. There are now multiple NLP tools that are competing with each other:

• https://github.com/PetrochukM/PyTorch-NLP
• https://github.com/facebookresearch/pytext
• https://github.com/allenai/allennlp

With regard to me, I split off two years ago and started my own project: https://github.com/PetrochukM/PyTorch-NLP. I wanted to simplify the coupled torchtext objects into something more functional like PyTorch. This has greatly helped my personal productivity as a Deep Learning engineer at Apple, University of Washington, AI2, and WellSaid Labs.

Here is a comparison of the various tools: https://github.com/facebookresearch/pytext/issues/110