Need help in optimization

Hi,

Thanks for this great library.

I have a use case related to multiple domains where I will have to do nlp.clone() multiple times in order not to overlap lexicon information related to one domain with another.

Currently, each nlp.clone() call is taking approx. 160ms. So, for 40 domains it is taking approximately 6 seconds. Any better way of doing this?

var nlp = require('compromise');
const performance = require('perf_hooks').performance;
var totalTime = 0;
var numOfDomains = 40;
for(var i = 0; i < numOfDomains; i++){
  var t0 = performance.now();
  var _nlp = nlp.clone();
  var t1 = performance.now();
  var diff = (t1 - t0); 
  totalTime += diff;
}
console.log("Time taken for " + numOfDomains + " apps is approx. " + Math.round(totalTime / 1000) + " Seconds");

Time taken for 40 apps is approx. 6 Seconds

[sheikirfanbasha]

@spencermountain Yes. Replaced nlp.clone() with below code

function doClone(text, _nlp, lexicon){
	var nlp = _nlp.bind({});
	var doc = nlp(text);
	doc.world().plugin(lexicon);
	doc.tagger();
	return doc;
}

[sheikirfanbasha]

hey @sheikirfanbasha thanks, that’s an interesting problem. I think you’re doing everything right.

The nlp.clone() is a bit of a mess - but the main task is to run a deep-clone on the World object. All of the linguistic data is on that thing, so it’s understandable to take a few-dozen milliseconds to do it each time.

The good part is that you probably don’t need to completely clone the whole dataset each time. If you’re making small changes to the lexicon, you may be able to get around it with some clever hacks.

this is rough, but maybe something like this?

let nlp2 = nlp.bind({});

let doc1 = nlp('it was slow');
let doc2 = nlp2('it was slow', { slow: 'NewTag' });

doc1.debug();
//default behaviour
doc2.debug();
//has #NewTag

to be honest, now that I think about it, that’s probably how nlp.clone() should work 😉

let me know how it comes along. You could also do nlp.tokenize() (which doesn’t touch the word-data), then set World values directly, then run .tagger().

cheers

@spencermountain Thanks for the prompt reply! I used your suggestion of setting the world values directly and then running .tagger(). Performance is much better now! Thank you!