[New Model] DocFormer: End-to-End Transformer for Document Understanding

🌟 New model addition

Model description

See “DocFormer: End-to-End Transformer for Document Understanding”, Appalaraju et al (ICCV 2021) on CVF and arXiv

DocFormer is a multi-modal transformer model for 2D/visual documents from Amazon (where, fair disclosure, I also currently work but not in research) - which I would characterize at a high level as being broadly along the same use cases as LayoutLMv2 (already in transformers), but achieving better (state-of-the-art) results with smaller datasets per the benchmarks in the paper.

I’ve found this kind of multi-modal, spatial/linguistic model very useful in the past (actually released an AWS sample and blog post with Hugging Face LayoutLMv1 earlier this year) and would love the improvements from DocFormer could be available through HF Transformers.

Open source status

• the model implementation is available: (give details)
  • Looks like there’s an (MIT-0) implementation at https://github.com/shabie/docformer
• the model weights are available: (give details)
  • Not currently as far as I can tell?
• who are the authors: (mention them, if possible by @gh-username)
  • Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha - all of AWS AI. Not sure of GitHub usernames
  • @shabie for the currently available implementation

[pzdkn]

Any updates on this?

[uakarsh]

As an update, the authors would be sharing the Textract OCR for the RVL CDIP Dataset, and as soon as they release it, we would try to achieve the benchmark performance as mentioned in the paper. However, we are also trying from our end, to make our own OCR part, and then perfrom pre train and fine tune