Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Lucene wrapped in SailRepository with multithreading gives error

See original GitHub issue

Affected: v 2.0-SNAPSHOT, Java: openjdk-8 os: Ubuntu 16.04

I use LuceneSailRepository wrapped in SailsRepository. MemoryStore with dump file is used as a storage.

I load data in multithreads managed by Executors.newFixedThreadPoolpool. When I do .invokeAll I got error about closed call when new request is done. It seems Lucene index in not thread safety.

The full stacktrace is:

23:29:47,677 [pool-3-thread-2] ERROR LuceneSailConnection: Rolling back org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/tmp/rdf4j-loader-s5A6SCKMe.34724407296800170144873/lucenedir/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:780)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getIndexWriter(LuceneIndex.java:291)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.invalidateReaders(LuceneIndex.java:581)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.commit(LuceneIndex.java:651)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.addRemoveStatements(LuceneSailConnection.java:252)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:217)
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:169)
    at triple.loader$eval939$fn__940$fn__941.invoke(loader.clj:122)
    at triple.loader$eval939$fn__940.invoke(loader.clj:98)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at triple.loader$eval935$fn__936.invoke(loader.clj:88)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at sparql$load_multidata$fn__1094$fn__1095.invoke(sparql.clj:102)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

23:29:48,694 [pool-3-thread-2] ERROR LuceneSailConnection: Committing operations in lucenesail, encountered exception org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/tmp/rdf4j-loader-s5A6SCKMe.34724407296800170144873/lucenedir/write.lock. Only some operations were stored, 1 operations are discarded. Lucene Index is now corrupt. org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/tmp/rdf4j-loader-s5A6SCKMe.34724407296800170144873/lucenedir/write.lock
    at org.apache.lucene.store.Lock.obtain(Lock.java:89)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:780)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.getIndexWriter(LuceneIndex.java:291)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.rollback(LuceneIndex.java:658)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.addRemoveStatements(LuceneSailConnection.java:256)
    at org.eclipse.rdf4j.sail.lucene.LuceneSailConnection.commit(LuceneSailConnection.java:217)
    at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:169)
    at triple.loader$eval939$fn__940$fn__941.invoke(loader.clj:122)
    at triple.loader$eval939$fn__940.invoke(loader.clj:98)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at triple.loader$eval935$fn__936.invoke(loader.clj:88)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at sparql$load_multidata$fn__1094$fn__1095.invoke(sparql.clj:102)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

23:29:48,709 [clojure-agent-send-off-pool-0] ERROR sparql: There is error:  java.util.concurrent.ExecutionException
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: cannot close: prepareCommit was already called with no corresponding call to commit
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313)
    at sparql$load_multidata.invokeStatic(sparql.clj:110)
    at sparql$load_multidata.invoke(sparql.clj:90)
    at triple.multiload$fn__1114.invokeStatic(multiload.clj:15)
    at triple.multiload$fn__1114.invoke(multiload.clj:13)
    at clojure.test$test_var$fn__7983.invoke(test.clj:716)
    at clojure.test$test_var.invokeStatic(test.clj:716)
    at clojure.test$test_var.invoke(test.clj:707)
    at boot.user$eval133$fn__134.doInvoke(boot.user3412006909439801406.clj:23)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:646)
    at clojure.core$apply.invoke(core.clj:641)
    at boot.core$construct_tasks.doInvoke(core.clj:911)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:646)
    at clojure.core$apply.invoke(core.clj:641)
    at boot.core$boot$fn__933.invoke(core.clj:949)
    at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: cannot close: prepareCommit was already called with no corresponding call to commit
    at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:960)
    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1015)
    at org.eclipse.rdf4j.sail.lucene.LuceneIndex.shutDown(LuceneIndex.java:322)
    at org.eclipse.rdf4j.sail.lucene.LuceneSail.shutDown(LuceneSail.java:291)
    at org.eclipse.rdf4j.repository.sail.SailRepository.shutDownInternal(SailRepository.java:169)
    at org.eclipse.rdf4j.repository.base.AbstractRepository.shutDown(AbstractRepository.java:45)
    at triple.loader$eval939$fn__940.invoke(loader.clj:98)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at triple.loader$eval935$fn__936.invoke(loader.clj:88)
    at clojure.lang.MultiFn.invoke(MultiFn.java:238)
    at sparql$load_multidata$fn__1094$fn__1095.invoke(sparql.clj:102)
    ... 5 more

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
pulquerocommented, Jun 30, 2016

@jgrzebyta PR with fix now available. Please take a look and confirm if you believe the newly added test covers your use case.

0reactions
jgrzebytacommented, Jul 4, 2016

@pulquero IMO The test code covers my case. Unfortunately I do not have the original code so I cannot check the solution - I went into single thread solution.

Ps. I am not sure if your pull request was added to the main project if so please could you close this issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Lucene NIOFSDirectory and SimpleFSDirectory with ...
My basic question is: what's the proper way to create/use instances of NIOFSDirectory and SimpleFSDirectory when there's multiple threads that ...
Read more >
IndexWriter (Lucene 6.6.6 API)
NOTE: IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently. If your application requires ...
Read more >
Getting "Error committing changes_closed Lucene index ...
Hi,. I am using Gerrit 2.14.6, I restarted the Gerrit once and not able to see Merged list. And getting the following error....
Read more >
Top 20 C++ multithreading mistakes and how to avoid them
Mistake # 1: Not using join() to wait for background threads before terminating an application. If we forgot to join a thread or...
Read more >
There is an error in using multithreading to write ...
I use multiple threads for insert operation, but an error will occur at some time. Is it because all threads share a SqlConnection?...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

SPIN Construct Rule Fails To Run Upon Instantiation when Same SPARQL Query Runs in Workbench

Next page

Support BIND in SparqlQueryRenderer