`--concurrency` in `tilelive-copy` doesn't appear to have any affect

It appears (to me) that the --concurrency argument to tilelive-copy doesn’t have any affect. No matter what value I put in top shows that 4 node processes are running, regardless if I’m using a machine with 4 CPU cores (my desktop) or a server with 8. Regardless of the --concurrency value, I get approximately the same throughput when doing a copy. Changing --concurrency to ten times it’s old value doesn’t appear to increase (or decrease) the throughput.

Is there something I’m missing to get increased concurrency?

[ianthetechie]

@mapsam thanks so much for going out of your way to look into this. I’ll give this a shot this AM (Korean time) and report back!

[mapsam]

Thanks for the info- it’s helpful to know which tilelive plugins you’re using since this module only connects the interfaces. Upon quick glance it looks like this could be fixed in the tilelive-bridge & tilelive-mapnik modules, which both use mapnik-pool (which sets threadpool size based on number of cpus here). Each module sets the size differently:

• tilelive-mapnik allows the user to set via UV_THREADPOOL_SIZE in the environment, otherwise defaults to require('os').cpus().length
• tilelive-bridge hard codes to cpus() * 2 and is not configurable it seems. Would gladly accept a PR that first tries process.env.UV_THREADPOOL_SIZE and backs up to making a best guess.

My suggestion would be to test changing that number in whichever way makes sense for you to see if that helps speed things up for you. @ianthetechie Glad to hear you’re working on something better, feel free to share your work or backport fixes in the tilelive modules!