Very high memory usage with write_html.py

On my system running write_html.py without arguments requires too much memory and takes too long. After more than 30 minutes I had to manually stop it because my system became unresponsive. Memory usage increased slowly but relentlessly, until write_html.py used all 8 gigabytes of RAM plus 5 gigabytes of swap.

My data directory is currently 3.1 gigabytes. It will continue to expand in the future because I’m always fetching new subreddits.

How can I help debug this?

P.S. My knowledge of Python is still very small…

[ghost]

To give you a better idea, I have already archived 16 subreddits:

$ ./write_html.py
...
all done. 581830 links filtered to 581830

$ du -sm data
4715    data

[ghost]

After commenting out those lines, write_html.py no longer uses too much memory. But I noticed index.html files for each subreddit are now missing, so I can’t display the archives with a browser.