Deletion of labels does not affect label references on labeled reports

Description

Deleting a label that is used will not remove its references from reports, causing errors when reports are listed in the frontend.
Statistics on the Dashboard are also affected as referenced label does not exist

Environment

1. OS: Ubuntu 18.04.5 LTS
2. OpenCTI version: 4.0.7
3. OpenCTI client: frontend
4. Other environment details: Running in Docker Stack on Docker 20.10.2, build 2291f61

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Go to “Settings"→"Labels & Attributes” in the web frontend
2. Click the dots and click “delete”
3. Go to “Activities"→"Analysis”
4. Go to “Dashboard”

Expected Output

Label should be deleted along with references to it.
Reports should be visible under Analysis, but now without the deleted tag.
Dashboard widget for “Top labels” should reflect labels except those deleted.

Actual Output

Step 3 is giving error message in the frontend: “An unknown error occurred. Please contact your administrator or the OpenCTI maintainers.”
Step 4: Dashboard struggles to load widget for “Top labels” (and possibly “Top Active Entities”). The page keeps reloading every second.

Additional information

Docker log entry from opencti_opencti container after attempting to load the “Analysis” page in the frontend:

opencti_opencti.1.yww3ckzxusp1@hostname | {"user":{"ip":"::****:*.*.*.*","user_id":"************"},"type":"READ_ERROR","operation_query":"query ReportsLinesPaginationQuery($search:String$count:Int!$cursor:ID$orderBy:ReportsOrdering$orderMode:OrderingMode$filters:[ReportsFiltering]){...ReportsLines_data_2wN0PW}fragment ReportLine_node on Report{id name published x_opencti_report_status createdBy{__typename ...on Identity{__isIdentity:__typename id name entity_type}}objectMarking{edges{node{id definition x_opencti_color}}}objectLabel{edges{node{id value color}}}}fragment ReportsLines_data_2wN0PW on Query{reports(search:$search first:$count after:$cursor orderBy:$orderBy orderMode:$orderMode filters:$filters){edges{node{id name published createdBy{__typename ...on Identity{__isIdentity:__typename id name entity_type}}objectMarking{edges{node{id definition}}}...ReportLine_node __typename}cursor}pageInfo{endCursor hasNextPage globalCount}}}","inner_relation_creation":0,"operation":"ReportsLinesPaginationQuery","time":53,"variables":{"search":"","count":25,"cursor":null,"orderBy":"published","orderMode":"desc","filters":[{"key":"report_types","values":["MISP Event"],"operator":"eq"}]},"size":154,"error":{"stacktrace":["Error: Cannot return null for non-nullable field LabelEdge.node.","at completeValue (/opt/opencti/node_modules/graphql/execution/execute.js:561:13)","at resolveField (/opt/opencti/node_modules/graphql/execution/execute.js:474:19)","at executeFields (/opt/opencti/node_modules/graphql/execution/execute.js:294:18)","at collectAndExecuteSubfields (/opt/opencti/node_modules/graphql/execution/execute.js:749:10)","at completeObjectValue (/opt/opencti/node_modules/graphql/execution/execute.js:739:10)","at completeValue (/opt/opencti/node_modules/graphql/execution/execute.js:592:12)","at /opt/opencti/node_modules/graphql/execution/execute.js:626:25","at from (<anonymous>)","at completeListValue (/opt/opencti/node_modules/graphql/execution/execute.js:613:49)","at completeValue (/opt/opencti/node_modules/graphql/execution/execute.js:574:12)","at resolveField (/opt/opencti/node_modules/graphql/execution/execute.js:474:19)","at executeFields (/opt/opencti/node_modules/graphql/execution/execute.js:294:18)","at collectAndExecuteSubfields (/opt/opencti/node_modules/graphql/execution/execute.js:749:10)","at completeObjectValue (/opt/opencti/node_modules/graphql/execution/execute.js:739:10)","at completeValue (/opt/opencti/node_modules/graphql/execution/execute.js:592:12)","at /opt/opencti/node_modules/graphql/execution/execute.js:471:16"]},"level":"error","message":"API Call","timestamp":"2021-01-19T14:33:43.528Z"}

[richard-julien]

Finally the implementation will iterate and remove everything. So if you have a lot of elements it can take some times. The API is not designed to do massive update or delete for now, so if your front timeout waiting for the answer, be careful to not click again on the button 😃

For the data you have, the release will have a specific yarn command to clear the broken data if you want to.

[richard-julien]

@crym ,

The code had a limitation and only delete 1000 related elements connected to the element before deleting it, with the effect to let some relations linked to deleted data.

I change the behavior to delete the target only of all related elements are deleted, with the hard limit of 5000 to prevent locking the platform with very large delete.

So if you want to delete a label linked to 10 000 reports, you will have to ask 2 times for the deletion. (After the first one the element will be display again after refresh because only 5000 elements will be deleted the first time)

Now for your current platform that is “broken”. Can you start from scratch or do you need to cleanup the data?