Tika 1.24.1 and gzip compression

Hello, Tika released 1.24.1 which allows gzip compression of input and output streams for tika-server.

What do you think of making it the default for the output stream? Since requests automatically decodes gzip and deflate transfer-encodings it’s just adding the header Accept-Encoding: gzip, deflate to services rmeta, tika, rmeta/text.

I can provide a PR.

Cheers, Carina

[carantunes]

Hi,

After some tests and benchmarks I’ve reconsidered if it should be changed by default. Gzip compression has the upside of improving transfer speed and bandwidth utilisation (~75%), at the cost of some cpu utilisation. For large files may be an improvement.

Another difference is that files sent to Tika with compression will have a different Content-Type returned (ie, from ‘application/pdf’ to [‘application/gzip’, ‘application/pdf’])

Instead I believe it would be sufficient to support sending/receiving gzip format by releasing 1.24.1

Input compression can be achieved with gzip or zlib:

    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(zlib.compress(file_obj.read()))

...

    with open(file, 'rb') as file_obj:
        return tika.parser.from_buffer(gzip.compress(file_obj.read()))

And output with the header:

    with open(file, 'rb') as file_obj:
        return tika.parser.from_file(file_obj, headers={'Accept-Encoding': 'gzip, deflate'})

A sample of benchmark (using pytest-benchmark) results using a ppt (100MB), run with the default timer first and lastly with --benchmark-timer=time.process_time which doesn’t include sleeping time or waiting for I/O:

[carantunes]

@tballison Pardon my delay, I was on vacation. Thanks for the input, I had not notice there was a difference.

After some debugging it looks like if I send -H "Content-Encoding: application/gzip" to rmeta I get a different result that if I send with -H "Content-Encoding: gzip" or with no header at all. I’ve created a ticket with more details if you want to further investigate/explain it https://issues.apache.org/jira/browse/TIKA-3169.