parser.apply does not return for a long time even though the progress bar indicates it finishes parsing

Description of the bug

This is not a bug, but a performance issue. This is not noticeable when parsing a small number of documents, but parser.apply does not return even though the progress bar indicates it finishes parsing a long time ago (1 hour or more ago).

To Reproduce

Steps to reproduce the behavior:

1. Parse many documents (my case: ~2500)

Expected behavior

parser.apply returns when the progress bar indicates it finished parsing all the documents.

Error Logs/Screenshots

If applicable, add error logs or screenshots to help explain your problem.

Environment (please complete the following information)

• OS: Debian Buster
• PostgreSQL Version: 12.1
• Poppler Utils Version: N/A
• Fonduer Version: 0.8.3+dev (01e0d9319b523aff7aa7f5c583a9f330b0705ecc)

Additional context

Add any other context about the problem here.

[robbieculkin]

I’ve tested master on ~200 docs and can confirm that these changes fix OOM errors and slow performance. Many thanks for the fix.

[lukehsiao]

Your description sounds correct to me, and this is definitely a real bottleneck several have run into. We would love to try and resolve this, but I suspect it wouldn’t be a quick fix.

I’m going to reopen this issue.