OConcurrentModificationException while adding edges

We’re developing a system that we’re basing on OrientDB graphs (OrientDB 2.1.3). In the application, we have a thin pojo->graph persistence layer that should do the work properly, but I get OConcurrentModificationException when having multiple threads updating the database.

Here’s an example scenario:

1. Create a Product vertex with edge to Color “Blue”
2. Simultaneously (while the transaction for creating Product 1 is open) create another Product vertex is created and also adds an edge to Color “Blue”.
3. OConcurrentModificationException is thrown since the version of Color “Blue” vertex is updated. Note that I’m not trying to save or modify the Color “Blue” vertex itself.

As I understood the docs at http://orientdb.com/docs/2.1/Concurrency.html#concurrency-on-adding-edges setting -DridBag.embeddedToSbtreeBonsaiThreshold=-1 should help me avoid my problem, although it still doesn’t work.

I think this is a perfectly valid case for a database like OrientDB, and according to the documentation, it should work.

Stacktrace of the exception:

Error on releasing database 'infogileorientdatabasetest' in pool
com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #40:1 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v34 your=v33)
at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:55)
at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:42)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:2279)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:1911)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:2364)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1111)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:609)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:156)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2582)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2551)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.commit(ONetworkProtocolBinary.java:1221)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:400)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:223)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

Test case I have reproduced the error using this test case (I would be delighted if there’s something else I’ve done wrong to cause the problem… 😃

OGlobalConfiguration.RID_BAG_EMBEDDED_TO_SBTREEBONSAI_THRESHOLD.setValue(-1) in a static block.

package se.infogile.persistence.orientdb;

import com.orientechnologies.orient.client.remote.OServerAdmin;
import com.orientechnologies.orient.core.config.OGlobalConfiguration;
import com.orientechnologies.orient.core.db.OPartitionedDatabasePool;
import com.orientechnologies.orient.core.db.OPartitionedDatabasePoolFactory;
import com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;
import com.orientechnologies.orient.core.exception.OConcurrentModificationException;
import com.orientechnologies.orient.core.exception.OConfigurationException;
import com.orientechnologies.orient.core.exception.OStorageException;
import com.orientechnologies.orient.core.tx.OTransaction;
import com.orientechnologies.orient.enterprise.channel.binary.OResponseProcessingException;
import com.orientechnologies.orient.server.OServer;
import com.orientechnologies.orient.server.OServerMain;
import com.orientechnologies.orient.server.config.OServerConfiguration;
import com.orientechnologies.orient.server.config.OServerConfigurationLoaderXml;
import com.orientechnologies.orient.server.config.OServerNetworkListenerConfiguration;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.impls.orient.OrientGraph;
import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testng.Assert;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;


/**
 * Created by heintz on 14/10/15.
 */
public class OrientDBEdgeProblemTest {
    static {
        OGlobalConfiguration.RID_BAG_EMBEDDED_TO_SBTREEBONSAI_THRESHOLD.setValue(-1);
    }
    private static OPartitionedDatabasePoolFactory dbPoolFactory = new OPartitionedDatabasePoolFactory(100);
    private static Logger logger = LoggerFactory.getLogger(OrientDBEdgeProblemTest.class);
    private OServer server = null;
    private static ExecutorService executorService = Executors.newFixedThreadPool(10);
    private static final String dbName = "edgeproblemtest";

    @Test
    public void testVersionIncrementError() throws Throwable {
        OrientGraph graph = getGraph(dbName);
        graph.getRawGraph().setDefaultTransactionMode();
        graph.createVertexType("Product");
        graph.createVertexType("Color");
        graph.createEdgeType("HasColor");
        graph.getRawGraph().begin(OTransaction.TXTYPE.OPTIMISTIC);

//        graph.begin();
        Vertex v1 = graph.addVertex("Color", "name", "Blue");
        graph.commit();
        graph.shutdown();

        char[] alphabet = new char[] {'A','B','C','D','E','F','G'};

        List<Future> futures = new ArrayList<>();
        for (int i = 0; i < 2; i++) {
            int pos = i;
            futures.add(executorService.submit(new Callable<Object>() {
                @Override
                public Object call() throws Exception {
                    OrientGraph g = getGraph(dbName);
                    try {
                        g.begin();
                        Vertex v2 = g.addVertex("Product", "name", "Product "+alphabet[pos]);
                        g.addEdge(null, v2, v1, "HasColor");
                        Thread.sleep(200);
                        g.commit();
                    } catch (OConcurrentModificationException ocme) {
                        logger.error("Exception while saving: ", ocme);
                        Assert.fail("OConcurrentModificationException");
                    } finally {
                        g.shutdown();
                    }
                    return null;
                }
            }));
        }
        for (Future f : futures) {
            f.get();
        }

        executorService.shutdown();
        executorService.awaitTermination(5, TimeUnit.SECONDS);
    }

    @AfterSuite
    public void tearDown() throws Exception {
        logger.info("Shutting down OrientDB");
        if (server != null) {
            server.shutdown();
        }
    }


    private OrientGraph getGraph(String dbName) {
        String _db = "remote:localhost:3424";
        String url = _db + "/" + dbName;
        ODatabaseDocumentTx db = null;
        try {
            OPartitionedDatabasePool pool = dbPoolFactory.get(url,
                    "root",
                    "admin");
            db = pool.acquire();
        } catch (OResponseProcessingException | OConfigurationException | OStorageException oce) {
            try {
                logger.info("creating new database named " + dbName);
                System.err.println("Before DB creation");
                OServerAdmin serverAdmin = new OServerAdmin(_db).connect(
                        "root",
                        "admin"
                );
                serverAdmin.createDatabase(dbName, "document", "plocal");
                serverAdmin.close();
                System.err.println("After DB creation");
            } catch (IOException ex) {
                logger.error("Unable to create database " + dbName, ex);
            }

            OPartitionedDatabasePool pool = dbPoolFactory.get(url,
                    "root",
                    "admin");
            db = pool.acquire();
        }

        return new OrientGraph(db);
    }

    @BeforeSuite
    public void setUpDatabase() throws Exception {
        File f = new File(".");
        InputStream is = GraphPersistenceServiceTest.class.getResourceAsStream("/orientdb.config");
        Assert.assertNotNull(is);
        logger.info("Starting OrientDB");
        server = OServerMain.create();

        OServerConfigurationLoaderXml loaderXml = new OServerConfigurationLoaderXml(OServerConfiguration.class, GraphPersistenceServiceTest.class.getResourceAsStream("/orientdb.config"));
        OServerConfiguration oServerConfiguration = new OServerConfiguration(loaderXml);
        System.setProperty("ORIENTDB_ROOT_PASSWORD", "admin");
        System.setProperty("RUNMODE", "UNITTEST");

        OServerNetworkListenerConfiguration networkConfig = oServerConfiguration.network.listeners.iterator().next();
        networkConfig.portRange = "3424-3430";

        server.setServerRootDirectory("./target/orientdb");

        server.startup(oServerConfiguration);

        File serverDir = new File("./target/orientdb");
        if (serverDir.exists()) {
            FileUtils.deleteDirectory(serverDir);
        }
        serverDir.mkdirs();
        File dbDir = new File(serverDir, "databases");
        dbDir.mkdirs();

        server.activate();
        OGlobalConfiguration.dumpConfiguration(System.out);
        Thread.sleep(2000);
    }
}

Running this test case, I also occasionally get a NullPointerException, but mostly the same OConcurrentModificationException as above. The test case I have provided never succeeds, I still feel it’s a valid case where multiple threads have an open transaction and adding an edge to the same vertex…

Caused by: java.lang.NullPointerException at com.orientechnologies.orient.core.db.record.ridbag.sbtree.OSBTreeRidBag.serializ‌​e(OSBTreeRidBag.java:816) at com.orientechnologies.orient.core.db.record.ridbag.ORidBag.toStream(ORidBag.java‌​:277)

[laa]

Hi , Have you seen fixed test case ? Does it add vledges concurrently ? 😃 You can not share the same Java objects between threads, because they like a POJOs mapped to database structure. About presence of single edge , once you get CME you should reload vertex and continue add edges . It is disadvantage of optmistic locking approach . Users can not overwrite entity content if they do see this content before change .

On Tue, Nov 17, 2015, 19:48 Anders Heintz [e-mail] wrote:

Could you please elaborate? I can buy the workaround that you need at least one edge before starting any concurrent operation on vertices, but do you mean there is a limitation in OrientDB that two different threads cannot add edges to the same vertex simultanously? What about a distributed scenario, where there are multiple application instances?

— Reply to this email directly or view it on GitHub https://github.com/orientechnologies/orientdb/issues/5152#issuecomment-157450662 .

Best regards, Andrey Lomakin, R&D lead. OrientDB Ltd

twitter:@Andrey_Lomakin linkedin:https://ua.linkedin.com/in/andreylomakin

[gtadudeps]

Hi @laa @luigidellaquila I am able to handle CME on a single node setup by setting RID_BAG_EMBEDDED_TO_SBTREEBONSAI_THRESHOLD=-1 but since SBTrees are not supported in distributed mode http://orientdb.com/docs/2.2.x/RidBag.html how can we handle CME while adding the edges from multiple clients in a distributed mode setup?