App takes long to end after work is done

I created this custom script based on the samples:

const {detectClones} = require("jscpd");
const {MemoryStore} = require("@jscpd/core");

const BASE_APPS = [
//several source folders here
];
const DEFAULT_OPTIONS = {
    path: BASE_APPS,
    silent: true,
    minLines: 15,
    minTokens: 60,
    output: './master-report-js',
    reporters: ['json'],
    format: 'javascript,css,less,markup'
};

(async () => {
    const store = new MemoryStore();
    await detectClones(DEFAULT_OPTIONS, store);
})();

For me it takes about 6 seconds to output:

JSON report saved to master-report-js\0\jscpd-report.json Detection time:: 5916.966ms

After that it takes about 20 seconds more… and it is unclear to me what it is doing. It seems to happen after the await returns, because adding console.logs in a .then() clause also makes them also print out directly after the detection time, but then it takes 20 seconds more.

My quick and dirty solution is to add this to the detectClones… but it feels really horrible

await detectClones(DEFAULT_OPTIONS, store).then(()=>process.exit(0));

Could it be that the detectClones creates lots of events / async calls that need to be cleaned up?

The following turned out to be caused by a mistake I made in the file paths with the above program,so please ignore the remark below. Running the same command using the jscpd main app takes significantly longer:

npx jscpd --output ./master-report-double/0 --format javascript,css,less,markup --min-tokens 60 --min-lines 15 --ignore *.stories.js --silent --reporters json [...lots of source dirs]
JSON report saved to master-report-double\0\jscpd-report.json
Duplications detection: Found 86 exact clones with 3507(4.59%) duplicated lines in 596 (4 formats) files.
Detection time:: 261734.781ms

[sonacy]

when you changed memory store to sth like leveldb-store, the problem will be fixed. the key problem is about using promise in recursive maybe causing memory leak. I tried several ways to fix that.

1. you can just changed the file rabin-karp.ts, changes the loop function.

.finally(() => {
    if (!iteration.done) {
      if (clone) {
        clone = RabinKarp.enlargeClone(
          clone,
          iteration.value as IMapFrame,
          mapFrameInStore
        );
      }
      // loop
      setImmediate(loop);
    } else {
      resolve(clones);
    }
  });

i just change loop to setImmediate(loop), in that way, the event loop will not create a new task, but just execute the loop as a micro task immediate. but it will cause more time and memory to handle.
2. I just changed store to plain function， without using promise。

for same test code, first way causing 20s, but second causing 9s.

[jonim8or]

I created a simple way to reproduce it. It checks out moment.js and tries to find duplicates there. jscpd-411.zip

const {detectClones} = require("jscpd");

(async () => {
    console.time('Total time');
    await detectClones({path:['moment/src'], silent: true, output:'jscpd-output'});
    console.timeEnd('Total time');
	//This is where the ugly hack would come in 
	//process.exit(0);
})();

#!/bin/sh
if [ ! -d "/home/node/moment" ]
then 
	git clone https://github.com/moment/moment.git
fi
npm install
echo "Starting execution of main script"
START_TIME=$(date +%s)
node main.js
END_TIME=$(date +%s)
echo "Execution took $(($END_TIME - $START_TIME)) seconds"

It gives this output (when run with node 15)

Cloning into 'moment'...
remote: Enumerating objects: 28136, done.
remote: Total 28136 (delta 0), reused 0 (delta 0), pack-reused 28136
Receiving objects: 100% (28136/28136), 19.51 MiB | 5.51 MiB/s, done.
Resolving deltas: 100% (19477/19477), done.
Checking out files: 100% (786/786), done.
npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.

added 130 packages, and audited 130 packages in 16s

10 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npm notice
npm notice New patch version of npm available! 7.0.8 -> 7.0.13
npm notice Changelog: https://github.com/npm/cli/releases/tag/v7.0.13
npm notice Run npm install -g npm@7.0.13 to update!
npm notice
Starting execution of main script
Found 1302 clones.
Detection time:: 24.333s
Total time: 26.398s
Execution took 102 seconds