Improved AsyncParser API

Version

4.6.0-SNAPSHOT

Issues

• Closing the iterator’s returned by AsyncParser does not abort the parsing process. In fact, repeatedly abandoning iterators will cause parsing threads to silently pile up.
• AsyncParser’s default chunk size of 100K tuples introduces a long delay unsuitable for content probing. Though the 100k chunk size gives me 5-10% better throughput than significantly lower values.
• The EltStreamRDF is private. As mentioned in JENA-2309 those events would be useful in an hadoop/spark setting to scan for prefixes, thereby stopping the parser once only data is seen anymore.

Improvements

PR https://github.com/apache/jena/pull/1478 adds the following improvements:

• Changed AsyncParser API to return IteratorCloseables whose close() method actually cancels parsing.
• Added a public EvtStreamRDF interface for the parsing events. The existing private EltStreamRDF class remains as the internal data object. The naming is up for discussion 😃
• Added a Builder that gives control over chunk and queue sizes. The builder can create “low-level” IteratorCloseables as well as java Streams. The latter allows for convenient use with try-with-resources. The following snippet is from a new test case:

try (Stream<Triple> s = AsyncParser.of(Channels.newInputStream(channel), Lang.TURTLE, null)
        .setChunkSize(100).streamTriples().limit(expectedLimit)) {
    // ...
}

• If a parser fails then all remaining parsers are still started with a destination in ‘aborted state’ in order for them to close their resources.

Are you interested in contributing a solution yourself?

Yes

[Aklakan]

I updated the tests. I just started running the windows and mac tests on my github account and I am now waiting for the results.

Update: Both green for me 🎉

[afs]

There are a couple of test cases that spawn many (100) threads and subsequently test that all have terminated. This might be a problem in github CI pipelines which AFAIK only use 2 cores - it would be possible to decrease the number e.g. to 20 or 10.

Yes - probably a good idea to reduce the number. Many threads should not be a problem though they may go away in a very patchy manner. But if the test is good when using 10-20, it makes the suite a little bit more general.