JDOObjectNotFoundException: No such database row: Part 2

Current Behavior:

Dependency-Track 4.3.3 was released with persistence fixes, including for the JDOObjectNotFoundException reported in #1059. This exception is still occurring after upgrade to 4.3.3, albeit with teeny differences in the exception output (highlighted below).

Logging this whilst understanding that it might be related to datanucleus and unresolvable without good steps to reproduce. Hopefully, another DT user can provide some help in that area.

Anyway, here’s the Exception:

03:02:49.828 ERROR [VulnerabilityAnalysisTask] An unexpected error occurred performing a vulnerability analysis task
javax.jdo.JDOObjectNotFoundException: No such database row
	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:563)
	at org.datanucleus.api.jdo.JDOPersistenceManager.getObjectById(JDOPersistenceManager.java:1747)
	at alpine.persistence.AbstractAlpineQueryManager.getObjectById(AbstractAlpineQueryManager.java:522)
	at org.dependencytrack.tasks.scanners.InternalAnalysisTask.analyze(InternalAnalysisTask.java:84)
	at org.dependencytrack.tasks.scanners.InternalAnalysisTask.inform(InternalAnalysisTask.java:59)
	at org.dependencytrack.tasks.VulnerabilityAnalysisTask.performAnalysis(VulnerabilityAnalysisTask.java:146)  <=====
	at org.dependencytrack.tasks.VulnerabilityAnalysisTask.analyzeComponents(VulnerabilityAnalysisTask.java:110) <=====
	at org.dependencytrack.tasks.VulnerabilityAnalysisTask.inform(VulnerabilityAnalysisTask.java:79) <=====
	at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:99)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row
	at org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:439)
	at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:316)
	at org.datanucleus.state.StateManagerImpl.loadFieldsFromDatastore(StateManagerImpl.java:1542)
	at org.datanucleus.state.StateManagerImpl.validate(StateManagerImpl.java:5521)
	at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3562)
	at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3021)
	at org.datanucleus.api.jdo.JDOPersistenceManager.getObjectById(JDOPersistenceManager.java:1742)
	... 10 common frames omitted

Steps to Reproduce:

Not sure… but the exceptions come in bursts. The timestamps below coincide with our nightly builds. We have multiple Jenkins servers and some have many nodes.

23:35:00.210 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:35:10.019 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:35:20.061 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:35:29.974 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:35:40.047 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:35:49.995 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:35:59.912 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:36:10.298 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:36:20.909 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:36:30.953 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:36:40.988 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:36:51.105 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:37:01.172 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:37:11.088 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:37:21.197 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:37:31.569 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:37:41.604 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:37:51.087 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:38:01.120 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:38:11.051 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:38:20.876 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:38:30.854 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:38:40.924 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:38:51.034 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:39:00.460 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:39:10.131 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:39:19.773 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:39:29.491 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:39:39.086 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:39:48.679 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:39:58.311 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:40:07.734 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:40:17.380 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:40:27.076 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:40:36.764 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:40:46.069 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:40:55.125 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:41:05.027 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:41:15.154 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:41:25.156 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:41:34.884 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:41:44.455 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:41:54.341 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:42:04.289 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:42:14.442 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:42:24.417 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:42:34.028 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:42:43.693 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:42:53.386 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:43:03.357 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
23:43:12.924 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:01:17.180 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:01:36.249 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:01:54.559 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:02:12.914 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:02:31.491 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:02:49.828 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:03:07.997 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:03:26.326 ERROR [VulnerabilityAnalysisTask] An unexpected error ...
03:03:44.597 ERROR [VulnerabilityAnalysisTask] An unexpected error ...

The last 9 events are 28 hours later, not 4 hours later (So I also need to fix my logging!).

Expected Behavior:

No JDOObjectNotFoundException events in log for VulnerabilityAnalysisTask (or, hopefully, anywhere).

Environment:

• Dependency-Track Version: 4.3.3
• Distribution: [ Docker ]
• BOM Format & Version: Cyclonedx: 4 Projects with 1.0. 8 with 1.1, 134 with 1.2, 3 with 1.3.
• Database Server: [PostgreSQL ]
• Browser: Firefox

[bugbouncer]

I encounter the same issue in our instance.

The API Server is running for 3 days and the error messages are issued by internal and sonatype tasks.

The internal tasks logs the following error message, containing the same id again and again:

Object with id "720831" not found !

The ids change after a few hours.

I looked a little at the source code:

1. The missing id refers to a component
2. The task collects all to-be analyzed components at task-creation time
3. When the task is executed, the component cannot be found anymore
4. The task seems to complete, though the final message e.g. “Internal analysis complete” is not in the log (there is no more code after that message. It is just not written because the Exception is thrown and catch in a calling class)

So I assume we are dealing with a problem that components are deleted after the tasks have been created. As the tasks have a component-id the component must have been persisted before (or at least a transaction was created and might have been rolled back, creating the id, but not persisting the component).

If you have a busy Dependency Track server (many projects, many BOM uploads) the deletion of components can happen quite frequently.

I hope that is of help to a developer who can fix this.

[msymons]

Is VulnerabilityAnalysisTask always associated with a specific project? If so, then would it be possible to log project uuid for such errors? That might help some when it comes to generating steps to reproduce. ie, proving that the problem is not restricted to a particular BOM schema, etc.