App takes long to end after work is done
See original GitHub issueI created this custom script based on the samples:
const {detectClones} = require("jscpd");
const {MemoryStore} = require("@jscpd/core");
const BASE_APPS = [
//several source folders here
];
const DEFAULT_OPTIONS = {
path: BASE_APPS,
silent: true,
minLines: 15,
minTokens: 60,
output: './master-report-js',
reporters: ['json'],
format: 'javascript,css,less,markup'
};
(async () => {
const store = new MemoryStore();
await detectClones(DEFAULT_OPTIONS, store);
})();
For me it takes about 6 seconds to output:
JSON report saved to master-report-js\0\jscpd-report.json Detection time:: 5916.966ms
After that it takes about 20 seconds more… and it is unclear to me what it is doing. It seems to happen after the await returns, because adding console.logs in a .then() clause also makes them also print out directly after the detection time, but then it takes 20 seconds more.
My quick and dirty solution is to add this to the detectClones… but it feels really horrible
await detectClones(DEFAULT_OPTIONS, store).then(()=>process.exit(0));
Could it be that the detectClones creates lots of events / async calls that need to be cleaned up?
The following turned out to be caused by a mistake I made in the file paths with the above program,so please ignore the remark below.
Running the same command using the jscpd main app takes significantly longer:
npx jscpd --output ./master-report-double/0 --format javascript,css,less,markup --min-tokens 60 --min-lines 15 --ignore *.stories.js --silent --reporters json [...lots of source dirs]
JSON report saved to master-report-double\0\jscpd-report.json
Duplications detection: Found 86 exact clones with 3507(4.59%) duplicated lines in 596 (4 formats) files.
Detection time:: 261734.781ms
Issue Analytics
- State:
- Created 3 years ago
- Comments:9 (3 by maintainers)
when you changed memory store to sth like leveldb-store, the problem will be fixed. the key problem is about using promise in recursive maybe causing memory leak. I tried several ways to fix that.
i just change loop to setImmediate(loop), in that way, the event loop will not create a new task, but just execute the loop as a micro task immediate. but it will cause more time and memory to handle.
2. I just changed store to plain function, without using promise。
for same test code, first way causing 20s, but second causing 9s.
I created a simple way to reproduce it. It checks out moment.js and tries to find duplicates there. jscpd-411.zip
It gives this output (when run with node 15)