Error on Importing a list of xml-files

Good morning,

I have the task to import a long list of around 9000 xml files, all in the size range of 1 to 150MB.

I have encountered some errors while importing, and I can’t find a solution for them.

The most common error is the following:

[10:13:07 ERROR] Aborting import due to errors. [10:13:07 ERROR] Caused by: java.lang.NoClassDefFoundError: org/citydb/core/operation/common/cache/BranchCacheTable [10:13:07 ERROR] Caused by: java.lang.ClassNotFoundException: org.citydb.core.operation.common.cache.BranchCacheTable [10:13:07 WARN] Database import aborted. Exception in thread "main" java.lang.NoClassDefFoundError: picocli/CommandLine$IExitCodeGenerator at picocli.CommandLine$RunAll.recursivelyExtractExitCodeGenerators(CommandLine.java:2441) at picocli.CommandLine$RunAll.extractExitCodeGenerators(CommandLine.java:2438) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2180) at picocli.CommandLine$RunAll.execute(CommandLine.java:2391) at org.citydb.cli.ImpExpCli.process(ImpExpCli.java:305) at org.citydb.cli.ImpExpCli.start(ImpExpCli.java:157) at org.citydb.cli.ImpExpCli.main(ImpExpCli.java:149) Caused by: java.lang.ClassNotFoundException: picocli.CommandLine$IExitCodeGenerator at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 7 more

The error occurs mostly on the bigger files of the list (110MB+), but if I import them as a single file, everything runs fine. Apparently this error comes up when the Importer/Exporter is already running some time on other files, and then finally gets somehow overwhelmed from those bigger files. It is hard to investigate in this, as the errors only occur when the Importer is running for some time and on a bigger list, but the BranchCacheTable Keyword, has always been in the description of the error.

The error also happend while using the GUI, where it looked like this: [01:27:05 ERROR] Aborting import due to errors. [01:27:05 ERROR] Caused by: java.lang.NoClassDefFoundError: org/citydb/core/operation/common/cache/BranchCacheTable [01:27:05 ERROR] Caused by: java.lang.ClassNotFoundException: org.citydb.core.operation.common.cache.BranchCacheTable [01:27:05 WARN] Database import aborted. Exception in thread "AWT-EventQueue-0" java.lang.NoClassDefFoundError: com/formdev/flatlaf/MnemonicHandler$1 at com.formdev.flatlaf.MnemonicHandler.showMnemonics(MnemonicHandler.java:205) at com.formdev.flatlaf.MnemonicHandler.processKeyEventOnWindows(MnemonicHandler.java:125) at com.formdev.flatlaf.MnemonicHandler.postProcessKeyEvent(MnemonicHandler.java:82) at java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(Unknown Source) at java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(Unknown Source) at java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(Unknown Source) at java.awt.DefaultKeyboardFocusManager.dispatchEvent(Unknown Source) at java.awt.Component.dispatchEventImpl(Unknown Source) at java.awt.Container.dispatchEventImpl(Unknown Source) at java.awt.Window.dispatchEventImpl(Unknown Source) at java.awt.Component.dispatchEvent(Unknown Source) at java.awt.EventQueue.dispatchEventImpl(Unknown Source) at java.awt.EventQueue.access$500(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.awt.EventQueue$3.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source) at java.awt.EventQueue$4.run(Unknown Source) at java.awt.EventQueue$4.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source) at java.awt.EventQueue.dispatchEvent(Unknown Source) at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.run(Unknown Source) Caused by: java.lang.ClassNotFoundException: com.formdev.flatlaf.MnemonicHandler$1 at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 29 more

Both error logs look somehow different on the paths, but happened on the same files and around the “expected” time (in the later part of a list, where bigger files were located.

Unfortunately I cannot provide the files, that are producing this error due to privacy reasons. But as the error is not occurring when I import the files one by one (and also the validation function returns that the files are valid), I hope to find a solution to this without the files.

If you need further information, I’ll respond as soon as possible. Thanks! I’m curious about your answer!

Little remark: It is somehow a little bit annoying, that the Import/Exporter does not follow the order of the files in the folder, that is selected. For me it seems like the order in which the Importer works through the list is somehow random, which makes it impossible to continue an import, that was interrupted by an error. If the Import had an option to import in an size-order or alphabetical-order, it would be easy to find the list of files, that are already imported or otherwise that are still remaining.

My setup: Importer/Exporter 5.1.0 used in CLI mode OS: Windows Server

[clausnagel]

I guess your Out of Shared Memory error is thrown by your PostgreSQL database? You might need to increase the max_locks_per_transaction paramemter in your PostgreSQL configuration (see, for example, here). Alternatively, you can reduce the number of parallel workers and connections that are used by the Importer/Exporter. See the online manual and a similar discussion here.

[clausnagel]

Your issue looks strange because your JVM cannot find classes like BranchCacheTable which are contained in the JARs of the Importer/Exporter. These JARs are located in the lib folder inside the installation folder of the Importer/Exporter and the filenames start with impexp-*.jar.

Cache tables like BranchCacheTable store temporary information when importing/exporting datasets. The Importer/Exporter first holds this temporary information in main memory but writes it to the database (or a local cache) as soon as a predefined limit is reached to avoid out-of-memory situations. That’s why your issue only occurs after some time because only then these limits are reached. And obviously the limits are not reached when you only import single files.

You can set these limits under “Preferences -> Import -> Resources”. Check the “Cache settings” paragraph in this chapter of the online user manual. So, by reducing the numbers for the geometry and feature cache (e.g., to 20 or 50), you can force the Importer/Exporter to write the temporary information much earlier. And thus, your issue will occur much earlier. Just an idea to speed up your tests.

To resume an import operation, read this.