Out of memory - low result query

I think I’m in the right project to try and get an answer for this problem, so here goes.

I’m using two drivers in my java project to talk to Neo4j 3.3: OGM and the straight java driver:

• neo4j-ogm-bolt-driver (3.1.0
• neo4j-java-driver (1.5.2)

OGM performs as expected, and now I’m translating the project to use the straight java driver as well. I create a Driver object as follows: Driver driver = GraphDatabase.driver("bolt://" + hostname + ":7687", AuthTokens.basic(username, password));

The driver object will then provide session objects on which I execute parameterized queries. However, after 5 queries (the same one, different parameters), the sixth one (different one) maxes out my CPU and eventually crashes with a memory exception (probably due to neo4j getting a lot of memory, so there’s not that much room for java to play with… after reducing neo4j’s memory it’s just the 100% cpu load):

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 875036672 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /hs_err_pid378.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000063b400000, 875036672, 0) failed; error='Cannot allocate memory' (errno=12)

The query I’m running is the following: MATCH (w0:Word) , (w0:Word)<-[:CONTAINS_WORD]-(:Sentence)-[:SENTENCE_OF]-(:Paragraph)<-[:CONTAINS]-(d:Document) WHERE w0.originalTextLower = {wT0} AND id(d) = {docid} RETURN w0 parameters = {wT0=abo, docid=2681032}

When I run the query out of context, in a test class, both drivers execute this query perfectly (it does not generate results, and executes in milliseconds). My main program runs fine on OGM, but the java-driver version fails after a while. I did get it running by re-initialising the Driver object each time I do a call, but I guess that’s not really the way to go, overhead wise? Using the same Driver object and manually closing Session object after I execute a query does nothing to the problem.

I don’t really know where to look for now, the query is fine, it executes on both drivers (out of context), I use the same wrapper code to call the drivers… At first I posted on stackoverflow, but after finding out I could get the system to work by rebooting the connection pool each time I’m leaning towards a deeper problem.

[joachimnielandt]

Hi @lutovich, using separate sessions was indeed the answer, I’ve already started doing that. At least now I know it is intended behaviour! And now I also know that Sessions have to be explicitly closed, I’ll keep that in mind 😃.

Thanks for the help, Zu

[lutovich]

Hi @CountZukula,

I think this problem happens because the same session is used for both queries. A session can only be backed by a single network connection at a time. Such connection is acquired from the connection pool to execute a query or when a transaction is started. It is released back to the pool when query result is fully fetched or transaction is committed/rolled back.

In your code same connection is used for both queries because result of the first query is not fully fetched when second query is executed. Database processes all commands for the single connection synchronously and in FIFO order. Second query has to wait for all records of the first query to arrive before it gets executed. So driver has to first buffer all 33M of Property nodes before fetching Resource nodes. This buffering is what causes application to use a lot of memory and CPU.

I’ve added an integration test to verify that nested queries work: https://github.com/neo4j/neo4j-java-driver/commit/05c451a179881e2a08fbb7de145974b7a5da1c11. It is green as expected, however it uses a much smaller dataset 😃

In order to make your code work I’d recommend changing it to smth like this:

try ( Neo4JBoltSessionFactory neo4JBoltSessionFactory = new Neo4JBoltSessionFactory() )
{
    try ( Session session1 = neo4JBoltSessionFactory.getSession() )
    {
        StatementResult properties = session1.run( "MATCH (p:Property) RETURN p" );
        while ( properties.hasNext() )
        {
            Record property = properties.next();
            System.out.println( "property = " + property );
            try ( Session session2 = neo4JBoltSessionFactory.getSession() )
            {
                StatementResult resources = session2.run( "MATCH (r:Resource) RETURN r" );
                while ( resources.hasNext() )
                {
                    System.out.println( "resources.next() = " + resources.next() );
                }
            }
        }
    }
}

here separate session is used to execute the second query. It is recommended to close sessions explicitly. That is why try-with-resources blocks were added.

Hope this helps! Please let us know if your problem is solved.