Image.WriteToFile multithreading performance on windows
See original GitHub issueHello.
I’m using WriteToFile
method to write multiple tiles in multithreading application.
While going through some tests I’ve noticed something weird and decided to ask you about it. The thing is, that WriteToFile
multithreading performance is working kind of weird on Windows-x64. Here’s the test application, that would help you see what I’m talking about (it’s pretty much the same I sent in the previous issue).
When I use one thread tiling, overall performance is good, as you can see in log:
Windows 1 thread log
WriteToFile took 220 ms to run on zoom 0
WriteToFile took 217 ms to run on zoom 1
WriteToFile took 218 ms to run on zoom 2
WriteToFile took 229 ms to run on zoom 3
WriteToFile took 218 ms to run on zoom 4
WriteToFile took 223 ms to run on zoom 5
WriteToFile took 222 ms to run on zoom 6
WriteToFile took 218 ms to run on zoom 7
WriteToFile took 218 ms to run on zoom 8
WriteToFile took 224 ms to run on zoom 9
WriteToFile took 227 ms to run on zoom 10
WriteToFile took 1295 ms to run on zoom 11
WriteToFile took 66 ms to run on zoom 12
WriteToFile took 66 ms to run on zoom 12
WriteToFile took 92 ms to run on zoom 12
WriteToFile took 98 ms to run on zoom 12
WriteToFile took 39 ms to run on zoom 13
WriteToFile took 56 ms to run on zoom 13
WriteToFile took 56 ms to run on zoom 13
WriteToFile took 41 ms to run on zoom 13
WriteToFile took 50 ms to run on zoom 13
WriteToFile took 80 ms to run on zoom 13
WriteToFile took 80 ms to run on zoom 13
WriteToFile took 54 ms to run on zoom 13
WriteToFile took 13 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
WriteToFile took 18 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 18 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 18 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 39 ms to run on zoom 14
WriteToFile took 45 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 41 ms to run on zoom 14
WriteToFile took 40 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 37 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 39 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 37 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 16 ms to run on zoom 14
WriteToFile took 34 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 36 ms to run on zoom 14
WriteToFile took 36 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
Elapsed time: 5953
There’s not much time spent for one tile. But when I change it to 10 threads there are some problems coming up. The performance in the beginning is OK, then it suddenly becomes very slow, and then becomes quick as one-threaded again. The log for this run is here too:
Windows 10 threads log
WriteToFile took 218 ms to run on zoom 0
WriteToFile took 215 ms to run on zoom 1
WriteToFile took 214 ms to run on zoom 2
WriteToFile took 212 ms to run on zoom 3
WriteToFile took 219 ms to run on zoom 4
WriteToFile took 220 ms to run on zoom 5
WriteToFile took 219 ms to run on zoom 6
WriteToFile took 217 ms to run on zoom 7
WriteToFile took 217 ms to run on zoom 8
WriteToFile took 219 ms to run on zoom 9
WriteToFile took 225 ms to run on zoom 10
WriteToFile took 727 ms to run on zoom 11
WriteToFile took 1424 ms to run on zoom 12
WriteToFile took 1425 ms to run on zoom 12
WriteToFile took 1503 ms to run on zoom 12
WriteToFile took 1505 ms to run on zoom 12
WriteToFile took 8114 ms to run on zoom 13
WriteToFile took 8226 ms to run on zoom 13
WriteToFile took 8252 ms to run on zoom 13
WriteToFile took 8280 ms to run on zoom 13
WriteToFile took 8304 ms to run on zoom 13
WriteToFile took 8311 ms to run on zoom 13
WriteToFile took 8311 ms to run on zoom 13
WriteToFile took 8307 ms to run on zoom 13
WriteToFile took 54 ms to run on zoom 14
WriteToFile took 63 ms to run on zoom 14
WriteToFile took 101 ms to run on zoom 14
WriteToFile took 126 ms to run on zoom 14
WriteToFile took 164 ms to run on zoom 14
WriteToFile took 252 ms to run on zoom 14
WriteToFile took 275 ms to run on zoom 14
WriteToFile took 339 ms to run on zoom 14
WriteToFile took 1575 ms to run on zoom 14
WriteToFile took 2081 ms to run on zoom 14
WriteToFile took 2355 ms to run on zoom 14
WriteToFile took 2257 ms to run on zoom 14
WriteToFile took 2219 ms to run on zoom 14
WriteToFile took 2491 ms to run on zoom 14
WriteToFile took 2540 ms to run on zoom 14
WriteToFile took 26 ms to run on zoom 14
WriteToFile took 2525 ms to run on zoom 14
WriteToFile took 2666 ms to run on zoom 14
WriteToFile took 1711 ms to run on zoom 14
WriteToFile took 3663 ms to run on zoom 14
WriteToFile took 2905 ms to run on zoom 14
WriteToFile took 3352 ms to run on zoom 14
WriteToFile took 3143 ms to run on zoom 14
WriteToFile took 3367 ms to run on zoom 14
WriteToFile took 2878 ms to run on zoom 14
WriteToFile took 3357 ms to run on zoom 14
WriteToFile took 3264 ms to run on zoom 14
WriteToFile took 3369 ms to run on zoom 14
WriteToFile took 3119 ms to run on zoom 14
WriteToFile took 284 ms to run on zoom 14
WriteToFile took 509 ms to run on zoom 14
WriteToFile took 750 ms to run on zoom 14
Elapsed time: 19274
I should also mention, that if I set the zoom for 18 or around so, zooms 16-18 are written really fast on 10 threads mode with very low time for each tile. I also tried to do the same things with streams and NetVips.Native 8.9.0-rc4, but it’s the same thing. That’s the results on Windows-x64. I also run this program on Ubuntu and got good results for both 1 and 10 threads mode.
Ubuntu 1 thread log
WriteToFile took 228 ms to run on zoom 0
WriteToFile took 263 ms to run on zoom 1
WriteToFile took 263 ms to run on zoom 2
WriteToFile took 221 ms to run on zoom 3
WriteToFile took 216 ms to run on zoom 4
WriteToFile took 214 ms to run on zoom 5
WriteToFile took 215 ms to run on zoom 6
WriteToFile took 237 ms to run on zoom 7
WriteToFile took 222 ms to run on zoom 8
WriteToFile took 217 ms to run on zoom 9
WriteToFile took 231 ms to run on zoom 10
WriteToFile took 1382 ms to run on zoom 11
WriteToFile took 57 ms to run on zoom 12
WriteToFile took 60 ms to run on zoom 12
WriteToFile took 91 ms to run on zoom 12
WriteToFile took 92 ms to run on zoom 12
WriteToFile took 33 ms to run on zoom 13
WriteToFile took 53 ms to run on zoom 13
WriteToFile took 56 ms to run on zoom 13
WriteToFile took 37 ms to run on zoom 13
WriteToFile took 49 ms to run on zoom 13
WriteToFile took 451 ms to run on zoom 13
WriteToFile took 79 ms to run on zoom 13
WriteToFile took 479 ms to run on zoom 13
WriteToFile took 6 ms to run on zoom 14
WriteToFile took 11 ms to run on zoom 14
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 16 ms to run on zoom 14
WriteToFile took 15 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 10 ms to run on zoom 14
WriteToFile took 15 ms to run on zoom 14
WriteToFile took 41 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 40 ms to run on zoom 14
WriteToFile took 34 ms to run on zoom 14
WriteToFile took 51 ms to run on zoom 14
WriteToFile took 528 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 12 ms to run on zoom 14
WriteToFile took 34 ms to run on zoom 14
WriteToFile took 46 ms to run on zoom 14
WriteToFile took 313 ms to run on zoom 14
WriteToFile took 41 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 15 ms to run on zoom 14
WriteToFile took 9 ms to run on zoom 14
WriteToFile took 32 ms to run on zoom 14
WriteToFile took 31 ms to run on zoom 14
WriteToFile took 40 ms to run on zoom 14
WriteToFile took 530 ms to run on zoom 14
WriteToFile took 29 ms to run on zoom 14
WriteToFile took 33 ms to run on zoom 14
WriteToFile took 12 ms to run on zoom 14
Elapsed time: 7926
Ubuntu 10 threads log
WriteToFile took 215 ms to run on zoom 0
WriteToFile took 215 ms to run on zoom 1
WriteToFile took 214 ms to run on zoom 2
WriteToFile took 215 ms to run on zoom 3
WriteToFile took 214 ms to run on zoom 4
WriteToFile took 229 ms to run on zoom 5
WriteToFile took 221 ms to run on zoom 6
WriteToFile took 221 ms to run on zoom 7
WriteToFile took 213 ms to run on zoom 8
WriteToFile took 217 ms to run on zoom 9
WriteToFile took 262 ms to run on zoom 10
WriteToFile took 994 ms to run on zoom 11
WriteToFile took 345 ms to run on zoom 12
WriteToFile took 1029 ms to run on zoom 12
WriteToFile took 1037 ms to run on zoom 12
WriteToFile took 1030 ms to run on zoom 12
WriteToFile took 365 ms to run on zoom 13
WriteToFile took 376 ms to run on zoom 13
WriteToFile took 403 ms to run on zoom 13
WriteToFile took 422 ms to run on zoom 13
WriteToFile took 794 ms to run on zoom 13
WriteToFile took 485 ms to run on zoom 13
WriteToFile took 480 ms to run on zoom 13
WriteToFile took 457 ms to run on zoom 13
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 45 ms to run on zoom 14
WriteToFile took 117 ms to run on zoom 14
WriteToFile took 152 ms to run on zoom 14
WriteToFile took 213 ms to run on zoom 14
WriteToFile took 220 ms to run on zoom 14
WriteToFile took 268 ms to run on zoom 14
WriteToFile took 85 ms to run on zoom 14
WriteToFile took 134 ms to run on zoom 14
WriteToFile took 238 ms to run on zoom 14
WriteToFile took 225 ms to run on zoom 14
WriteToFile took 45 ms to run on zoom 14
WriteToFile took 26 ms to run on zoom 14
WriteToFile took 48 ms to run on zoom 14
WriteToFile took 106 ms to run on zoom 14
WriteToFile took 33 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 88 ms to run on zoom 14
WriteToFile took 117 ms to run on zoom 14
WriteToFile took 169 ms to run on zoom 14
WriteToFile took 196 ms to run on zoom 14
WriteToFile took 131 ms to run on zoom 14
WriteToFile took 46 ms to run on zoom 14
WriteToFile took 149 ms to run on zoom 14
WriteToFile took 128 ms to run on zoom 14
WriteToFile took 194 ms to run on zoom 14
WriteToFile took 66 ms to run on zoom 14
WriteToFile took 66 ms to run on zoom 14
WriteToFile took 57 ms to run on zoom 14
WriteToFile took 44 ms to run on zoom 14
WriteToFile took 205 ms to run on zoom 14
WriteToFile took 275 ms to run on zoom 14
Elapsed time: 6751
Issue Analytics
- State:
- Created 4 years ago
- Comments:10 (6 by maintainers)
That’s great to hear! MapTiler seems to use bilinear resampling by default1, which can cause aliasing. When comparing with other software, it is best to try to align the resize kernels (for example, you can use
Enums.Kernel.Linear
as a resize kernel).There are still 2 resize issues within libvips (see https://github.com/libvips/libvips/issues/1512 and https://github.com/libvips/libvips/issues/1518), which I’ll try to resolve soon. If you have problems with the second linked issue, you could disable the vector path with the
VIPS_NOVECTOR=1
environment variable or use this in C#:This fell off my radar, sorry. If forcing the image into memory is not desired, you could use a tile cache (as mentioned in https://github.com/libvips/libvips/pull/1551). For example:
I see now this (on my Windows PC):
Details
(Tested with a slightly modified program)
So it’s a bit slower than the 920 ms that forced the whole image into memory, but much faster than the timing you saw.
BTW, libvips 8.10 should fix the pixel shifts you noticed in https://github.com/kleisauke/net-vips/issues/25#issuecomment-453680434, see: https://libvips.github.io/libvips/2020/06/18/What’s-new-in-8.10.html#image-resize
I’ve adapted the above program to always use
vips_resize
. The 8.10.0-beta1 version of libvips will soon be published on NuGet.