OpenCTI - Front end crashes - Out of memory

Description

OpenCTI index.js crashes after a couple of days of parsing messages in queue

Environment

1. OS (where OpenCTI server runs): Ubuntu 18.04
2. OpenCTI version: 3.1.0
3. OpenCTI client: Frontend
4. Other environment details: OpenCTI Setup :

• OpenCTI + RabbitMQ + Minio : 16vCPU , 64GB RAM [ On a single server]
• ElasticSearch Cluster : 2 Servers ( 4vCPU , 16GB RAM) [ separate cluster]
• Grakn Server - 16vCPU , 128GB RAM [ Separate server]

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Frontend is manually stood up using node build/index.js
2. Main connectors running : CrowdStrike and MISP. CS connect has report fetch date of April 20th, 2020
3. Queue Size : 269.25K Messages , 3 Workers

Observations : I have noticed that while there are 3 python3 workers in the backend , the front end only recognizes 2. However all 3 are pulling from RabbitMQ .

Expected Output

Actual Output

Last Output from openCTI index.js process :

{"message":"[OPENCTI] Technical error > A database error has occured!","locations":[{"line":4,"column":25}],"path":["reportEdit","relationAdd"],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"name":"DatabaseError","_error":{},"_showLocations":false,"_showPath":false,"time_thrown":"2020-04-21T18:36:14.309Z","data":{"type":"technical","details":"There is more than one thing of type [stix_relation_embedded] that owns the key [e3caf292-1dab-521e-b791-503e0fdbc100] of type [internal_id_key]. "},"internalData":{},"_stack":"DatabaseError: A database error has occured!\n    at /opt/opencti/opencti/build/index.js:1:4389\n    at async R (/opt/opencti/opencti/build/index.js:1:4554)\n    at async Te (/opt/opencti/opencti/build/index.js:1:26068)\n    at async je (/opt/opencti/opencti/build/index.js:1:28669)\n    at async h (/opt/opencti/opencti/build/index.js:1:57699)","stacktrace":["DatabaseError: A database error has occured!","    at /opt/opencti/opencti/build/index.js:1:4389","    at async R (/opt/opencti/opencti/build/index.js:1:4554)","    at async Te (/opt/opencti/opencti/build/index.js:1:26068)","    at async je (/opt/opencti/opencti/build/index.js:1:28669)","    at async h (/opt/opencti/opencti/build/index.js:1:57699)"]}},"level":"error"}
{"code":3,"metadata":{"_internal_repr":{},"flags":0},"details":"InvalidKBException-A structural validation error has occurred. Please correct the [`1`] errors found. \nThere is more than one thing of type [stix_relation_embedded] that owns the key [e3caf292-1dab-521e-b791-503e0fdbc100] of type [internal_id_key]. \n. Please check server logs for the stack trace.","level":"error","message":"[GRAKN] CommitWriteTx error > 3 INVALID_ARGUMENT: InvalidKBException-A structural validation error has occurred. Please correct the [`1`] errors found. \nThere is more than one thing of type [stix_relation_embedded] that owns the key [e3caf292-1dab-521e-b791-503e0fdbc100] of type [internal_id_key]. \n. Please check server logs for the stack trace.","stack":"Error: 3 INVALID_ARGUMENT: InvalidKBException-A structural validation error has occurred. Please correct the [`1`] errors found. \nThere is more than one thing of type [stix_relation_embedded] that owns the key [e3caf292-1dab-521e-b791-503e0fdbc100] of type [internal_id_key]. \n. Please check server logs for the stack trace.\n    at Object.exports.createStatusError (/opt/opencti/opencti/node_modules/grpc/src/common.js:91:15)\n    at ClientDuplexStream._emitStatusIfDone (/opt/opencti/opencti/node_modules/grpc/src/client.js:233:26)\n    at ClientDuplexStream._receiveStatus (/opt/opencti/opencti/node_modules/grpc/src/client.js:211:8)\n    at Object.onReceiveStatus (/opt/opencti/opencti/node_modules/grpc/src/client_interceptors.js:1311:15)\n    at InterceptingListener._callNext (/opt/opencti/opencti/node_modules/grpc/src/client_interceptors.js:568:42)\n    at InterceptingListener.onReceiveStatus (/opt/opencti/opencti/node_modules/grpc/src/client_interceptors.js:618:8)\n    at /opt/opencti/opencti/node_modules/grpc/src/client_interceptors.js:1127:18"}
{"name":"DatabaseError","_error":{},"_showLocations":false,"_showPath":false,"time_thrown":"2020-04-21T18:36:14.615Z","data":{"type":"technical","details":"There is more than one thing of type [stix_relation_embedded] that owns the key [e3caf292-1dab-521e-b791-503e0fdbc100] of type [internal_id_key]. "},"internalData":{},"level":"error","message":"[GRAKN] executeWrite error > A database error has occured!","stack":"DatabaseError: A database error has occured!\n    at /opt/opencti/opencti/build/index.js:1:4389\n    at async R (/opt/opencti/opencti/build/index.js:1:4554)\n    at async Te (/opt/opencti/opencti/build/index.js:1:26068)\n    at async je (/opt/opencti/opencti/build/index.js:1:28669)\n    at async h (/opt/opencti/opencti/build/index.js:1:57699)"}
{"message":"[OPENCTI] Technical error > A database error has occured!","locations":[{"line":4,"column":25}],"path":["reportEdit","relationAdd"],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"name":"DatabaseError","_error":{},"_showLocations":false,"_showPath":false,"time_thrown":"2020-04-21T18:36:14.615Z","data":{"type":"technical","details":"There is more than one thing of type [stix_relation_embedded] that owns the key [e3caf292-1dab-521e-b791-503e0fdbc100] of type [internal_id_key]. "},"internalData":{},"_stack":"DatabaseError: A database error has occured!\n    at /opt/opencti/opencti/build/index.js:1:4389\n    at async R (/opt/opencti/opencti/build/index.js:1:4554)\n    at async Te (/opt/opencti/opencti/build/index.js:1:26068)\n    at async je (/opt/opencti/opencti/build/index.js:1:28669)\n    at async h (/opt/opencti/opencti/build/index.js:1:57699)","stacktrace":["DatabaseError: A database error has occured!","    at /opt/opencti/opencti/build/index.js:1:4389","    at async R (/opt/opencti/opencti/build/index.js:1:4554)","    at async Te (/opt/opencti/opencti/build/index.js:1:26068)","    at async je (/opt/opencti/opencti/build/index.js:1:28669)","    at async h (/opt/opencti/opencti/build/index.js:1:57699)"]}},"level":"error"}

<--- Last few GCs --->

[16404:0x2cde7a0] 194095778 ms: Mark-sweep 1969.5 (2076.7) -> 1956.6 (2080.2) MB, 3356.2 / 0.2 ms  (average mu = 0.105, current mu = 0.032) allocation failure scavenge might not succeed
[16404:0x2cde7a0] 194099786 ms: Mark-sweep 1971.7 (2080.2) -> 1956.7 (2077.0) MB, 3895.6 / 0.4 ms  (average mu = 0.065, current mu = 0.028) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x13c03d9]
    1: StubFrame [pc: 0x13fd410]
Security context: 0x31901bf008d1 <JSObject>
    2: replace [0x31901bf163c1](this=0x233e59c24f39 <String[#1]: ->,0x3ccccc89c4f1 <JSRegExp <String[#21]: [-\/\\^$*+?.()|[\]{}]>>,0x233e59c27901 <String[#3]: \$&>)
    3: unescapeFormat(aka unescapeFormat) [0x17ffb4d0f571] [/opt/opencti/opencti/node_modules/moment/moment.js:~677] [pc=0x304ae9bdac05](this=0x161f5b1c04b1 <undefined...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa02f90 node::Abort() [node]
 2: 0xa033b5 node::OnFatalError(char const*, char const*) [node]
 3: 0xb76ffe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb77379 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd23ad5  [node]
 6: 0xd24166 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd309e5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd31895 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd3434c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd021fc v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [node]
11: 0x10bb17d v8::internal::ReplacementStringBuilder::ToString() [node]
12: 0x106a30b  [node]
13: 0x106b4ca v8::internal::Runtime_RegExpReplaceRT(int, unsigned long*, v8::internal::Isolate*) [node]
14: 0x13c03d9  [node]

Grakn Output :

2020-04-21 18:36:46,272 [grpc-request-handler-28] ERROR g.c.s.r.SessionService$TransactionListener - Runtime Exception in RPC TransactionListener:
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
        at io.grpc.Status.asRuntimeException(Status.java:524)
        at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
        at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:864)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-04-21 18:36:46,273 [grpc-request-handler-31] ERROR g.c.s.r.SessionService$TransactionListener - Runtime Exception in RPC TransactionListener:
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
        at io.grpc.Status.asRuntimeException(Status.java:524)
        at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
        at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:864)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Wo

Additional information

I’ve not encountered this error before. Usually it is Grakn that runs out of memory, however this time based on the logs, it seems the opencti index.js ran out of memory. Any suggestions on what I can do for better optimization?

[SamuelHassine]

I close this issue for the moment. Please re-open it if the problem persists on the V4.

[richard-julien]

The report is auto generated when the process crash.