Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Image.WriteToFile multithreading performance on windows

See original GitHub issue

Hello. I’m using WriteToFile method to write multiple tiles in multithreading application. While going through some tests I’ve noticed something weird and decided to ask you about it. The thing is, that WriteToFile multithreading performance is working kind of weird on Windows-x64. Here’s the test application, that would help you see what I’m talking about (it’s pretty much the same I sent in the previous issue). When I use one thread tiling, overall performance is good, as you can see in log:

Windows 1 thread log 
WriteToFile took 220 ms to run on zoom 0
WriteToFile took 217 ms to run on zoom 1
WriteToFile took 218 ms to run on zoom 2
WriteToFile took 229 ms to run on zoom 3
WriteToFile took 218 ms to run on zoom 4
WriteToFile took 223 ms to run on zoom 5
WriteToFile took 222 ms to run on zoom 6
WriteToFile took 218 ms to run on zoom 7
WriteToFile took 218 ms to run on zoom 8
WriteToFile took 224 ms to run on zoom 9
WriteToFile took 227 ms to run on zoom 10
WriteToFile took 1295 ms to run on zoom 11
WriteToFile took 66 ms to run on zoom 12
WriteToFile took 66 ms to run on zoom 12
WriteToFile took 92 ms to run on zoom 12
WriteToFile took 98 ms to run on zoom 12
WriteToFile took 39 ms to run on zoom 13
WriteToFile took 56 ms to run on zoom 13
WriteToFile took 56 ms to run on zoom 13
WriteToFile took 41 ms to run on zoom 13
WriteToFile took 50 ms to run on zoom 13
WriteToFile took 80 ms to run on zoom 13
WriteToFile took 80 ms to run on zoom 13
WriteToFile took 54 ms to run on zoom 13
WriteToFile took 13 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
WriteToFile took 18 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 18 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 18 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 39 ms to run on zoom 14
WriteToFile took 45 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 41 ms to run on zoom 14
WriteToFile took 40 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 37 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 39 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 37 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 16 ms to run on zoom 14
WriteToFile took 34 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 36 ms to run on zoom 14
WriteToFile took 36 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 19 ms to run on zoom 14
Elapsed time: 5953

There’s not much time spent for one tile. But when I change it to 10 threads there are some problems coming up. The performance in the beginning is OK, then it suddenly becomes very slow, and then becomes quick as one-threaded again. The log for this run is here too:

Windows 10 threads log 
WriteToFile took 218 ms to run on zoom 0
WriteToFile took 215 ms to run on zoom 1
WriteToFile took 214 ms to run on zoom 2
WriteToFile took 212 ms to run on zoom 3
WriteToFile took 219 ms to run on zoom 4
WriteToFile took 220 ms to run on zoom 5
WriteToFile took 219 ms to run on zoom 6
WriteToFile took 217 ms to run on zoom 7
WriteToFile took 217 ms to run on zoom 8
WriteToFile took 219 ms to run on zoom 9
WriteToFile took 225 ms to run on zoom 10
WriteToFile took 727 ms to run on zoom 11
WriteToFile took 1424 ms to run on zoom 12
WriteToFile took 1425 ms to run on zoom 12
WriteToFile took 1503 ms to run on zoom 12
WriteToFile took 1505 ms to run on zoom 12
WriteToFile took 8114 ms to run on zoom 13
WriteToFile took 8226 ms to run on zoom 13
WriteToFile took 8252 ms to run on zoom 13
WriteToFile took 8280 ms to run on zoom 13
WriteToFile took 8304 ms to run on zoom 13
WriteToFile took 8311 ms to run on zoom 13
WriteToFile took 8311 ms to run on zoom 13
WriteToFile took 8307 ms to run on zoom 13
WriteToFile took 54 ms to run on zoom 14
WriteToFile took 63 ms to run on zoom 14
WriteToFile took 101 ms to run on zoom 14
WriteToFile took 126 ms to run on zoom 14
WriteToFile took 164 ms to run on zoom 14
WriteToFile took 252 ms to run on zoom 14
WriteToFile took 275 ms to run on zoom 14
WriteToFile took 339 ms to run on zoom 14
WriteToFile took 1575 ms to run on zoom 14
WriteToFile took 2081 ms to run on zoom 14
WriteToFile took 2355 ms to run on zoom 14
WriteToFile took 2257 ms to run on zoom 14
WriteToFile took 2219 ms to run on zoom 14
WriteToFile took 2491 ms to run on zoom 14
WriteToFile took 2540 ms to run on zoom 14
WriteToFile took 26 ms to run on zoom 14
WriteToFile took 2525 ms to run on zoom 14
WriteToFile took 2666 ms to run on zoom 14
WriteToFile took 1711 ms to run on zoom 14
WriteToFile took 3663 ms to run on zoom 14
WriteToFile took 2905 ms to run on zoom 14
WriteToFile took 3352 ms to run on zoom 14
WriteToFile took 3143 ms to run on zoom 14
WriteToFile took 3367 ms to run on zoom 14
WriteToFile took 2878 ms to run on zoom 14
WriteToFile took 3357 ms to run on zoom 14
WriteToFile took 3264 ms to run on zoom 14
WriteToFile took 3369 ms to run on zoom 14
WriteToFile took 3119 ms to run on zoom 14
WriteToFile took 284 ms to run on zoom 14
WriteToFile took 509 ms to run on zoom 14
WriteToFile took 750 ms to run on zoom 14
Elapsed time: 19274

I should also mention, that if I set the zoom for 18 or around so, zooms 16-18 are written really fast on 10 threads mode with very low time for each tile. I also tried to do the same things with streams and NetVips.Native 8.9.0-rc4, but it’s the same thing. That’s the results on Windows-x64. I also run this program on Ubuntu and got good results for both 1 and 10 threads mode.

Ubuntu 1 thread log 
WriteToFile took 228 ms to run on zoom 0
WriteToFile took 263 ms to run on zoom 1
WriteToFile took 263 ms to run on zoom 2
WriteToFile took 221 ms to run on zoom 3
WriteToFile took 216 ms to run on zoom 4
WriteToFile took 214 ms to run on zoom 5
WriteToFile took 215 ms to run on zoom 6
WriteToFile took 237 ms to run on zoom 7
WriteToFile took 222 ms to run on zoom 8
WriteToFile took 217 ms to run on zoom 9
WriteToFile took 231 ms to run on zoom 10
WriteToFile took 1382 ms to run on zoom 11
WriteToFile took 57 ms to run on zoom 12
WriteToFile took 60 ms to run on zoom 12
WriteToFile took 91 ms to run on zoom 12
WriteToFile took 92 ms to run on zoom 12
WriteToFile took 33 ms to run on zoom 13
WriteToFile took 53 ms to run on zoom 13
WriteToFile took 56 ms to run on zoom 13
WriteToFile took 37 ms to run on zoom 13
WriteToFile took 49 ms to run on zoom 13
WriteToFile took 451 ms to run on zoom 13
WriteToFile took 79 ms to run on zoom 13
WriteToFile took 479 ms to run on zoom 13
WriteToFile took 6 ms to run on zoom 14
WriteToFile took 11 ms to run on zoom 14
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 16 ms to run on zoom 14
WriteToFile took 15 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 10 ms to run on zoom 14
WriteToFile took 15 ms to run on zoom 14
WriteToFile took 41 ms to run on zoom 14
WriteToFile took 38 ms to run on zoom 14
WriteToFile took 40 ms to run on zoom 14
WriteToFile took 34 ms to run on zoom 14
WriteToFile took 51 ms to run on zoom 14
WriteToFile took 528 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 12 ms to run on zoom 14
WriteToFile took 34 ms to run on zoom 14
WriteToFile took 46 ms to run on zoom 14
WriteToFile took 313 ms to run on zoom 14
WriteToFile took 41 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 35 ms to run on zoom 14
WriteToFile took 15 ms to run on zoom 14
WriteToFile took 9 ms to run on zoom 14
WriteToFile took 32 ms to run on zoom 14
WriteToFile took 31 ms to run on zoom 14
WriteToFile took 40 ms to run on zoom 14
WriteToFile took 530 ms to run on zoom 14
WriteToFile took 29 ms to run on zoom 14
WriteToFile took 33 ms to run on zoom 14
WriteToFile took 12 ms to run on zoom 14
Elapsed time: 7926
Ubuntu 10 threads log 
WriteToFile took 215 ms to run on zoom 0
WriteToFile took 215 ms to run on zoom 1
WriteToFile took 214 ms to run on zoom 2
WriteToFile took 215 ms to run on zoom 3
WriteToFile took 214 ms to run on zoom 4
WriteToFile took 229 ms to run on zoom 5
WriteToFile took 221 ms to run on zoom 6
WriteToFile took 221 ms to run on zoom 7
WriteToFile took 213 ms to run on zoom 8
WriteToFile took 217 ms to run on zoom 9
WriteToFile took 262 ms to run on zoom 10
WriteToFile took 994 ms to run on zoom 11
WriteToFile took 345 ms to run on zoom 12
WriteToFile took 1029 ms to run on zoom 12
WriteToFile took 1037 ms to run on zoom 12
WriteToFile took 1030 ms to run on zoom 12
WriteToFile took 365 ms to run on zoom 13
WriteToFile took 376 ms to run on zoom 13
WriteToFile took 403 ms to run on zoom 13
WriteToFile took 422 ms to run on zoom 13
WriteToFile took 794 ms to run on zoom 13
WriteToFile took 485 ms to run on zoom 13
WriteToFile took 480 ms to run on zoom 13
WriteToFile took 457 ms to run on zoom 13
WriteToFile took 17 ms to run on zoom 14
WriteToFile took 45 ms to run on zoom 14
WriteToFile took 117 ms to run on zoom 14
WriteToFile took 152 ms to run on zoom 14
WriteToFile took 213 ms to run on zoom 14
WriteToFile took 220 ms to run on zoom 14
WriteToFile took 268 ms to run on zoom 14
WriteToFile took 85 ms to run on zoom 14
WriteToFile took 134 ms to run on zoom 14
WriteToFile took 238 ms to run on zoom 14
WriteToFile took 225 ms to run on zoom 14
WriteToFile took 45 ms to run on zoom 14
WriteToFile took 26 ms to run on zoom 14
WriteToFile took 48 ms to run on zoom 14
WriteToFile took 106 ms to run on zoom 14
WriteToFile took 33 ms to run on zoom 14
WriteToFile took 20 ms to run on zoom 14
WriteToFile took 88 ms to run on zoom 14
WriteToFile took 117 ms to run on zoom 14
WriteToFile took 169 ms to run on zoom 14
WriteToFile took 196 ms to run on zoom 14
WriteToFile took 131 ms to run on zoom 14
WriteToFile took 46 ms to run on zoom 14
WriteToFile took 149 ms to run on zoom 14
WriteToFile took 128 ms to run on zoom 14
WriteToFile took 194 ms to run on zoom 14
WriteToFile took 66 ms to run on zoom 14
WriteToFile took 66 ms to run on zoom 14
WriteToFile took 57 ms to run on zoom 14
WriteToFile took 44 ms to run on zoom 14
WriteToFile took 205 ms to run on zoom 14
WriteToFile took 275 ms to run on zoom 14
Elapsed time: 6751

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:10 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
kleisaukecommented, Jul 17, 2020

That’s great to hear! MapTiler seems to use bilinear resampling by default1, which can cause aliasing. When comparing with other software, it is best to try to align the resize kernels (for example, you can use Enums.Kernel.Linear as a resize kernel).

There are still 2 resize issues within libvips (see https://github.com/libvips/libvips/issues/1512 and https://github.com/libvips/libvips/issues/1518), which I’ll try to resolve soon. If you have problems with the second linked issue, you could disable the vector path with the VIPS_NOVECTOR=1 environment variable or use this in C#:

NetVips.VectorSet(false);
1reaction
kleisaukecommented, Jul 8, 2020

This fell off my radar, sorry. If forcing the image into memory is not desired, you could use a tile cache (as mentioned in https://github.com/libvips/libvips/pull/1551). For example:

--- a/Program.cs
+++ b/Program.cs
@@ -88,6 +88,11 @@ namespace TestNetVips
         /// </summary>
         private static int TileSize { get; } = 256;
 
+        /// <summary>
+        /// Max tiles to cache.
+        /// </summary>
+        private static int MaxTiles { get; } = 1000;
+
         #endregion
 
         private static async Task Main(string[] args)
@@ -108,10 +113,12 @@ namespace TestNetVips
             RasterXSize = inputImage.Width;
             RasterYSize = inputImage.Height;
 
+            using Image image = inputImage.Tilecache(tileWidth: TileSize, tileHeight: TileSize, maxTiles: MaxTiles, threaded: true);
+
             //Create progress-reporter.
             ConsoleProgress<double> consoleProgress = new ConsoleProgress<double>(Console.WriteLine);
 
-            await GenerateTilesAsync(inputImage, OutputDirectoryInfo, consoleProgress).ConfigureAwait(false);
+            await GenerateTilesAsync(image, OutputDirectoryInfo, consoleProgress).ConfigureAwait(false);
 
             Console.WriteLine("Done");
             Console.WriteLine($"Elapsed time: {stopwatch.ElapsedMilliseconds}");

I see now this (on my Windows PC):

Details 
WriteToFile took 259 ms to run on zoom 0
WriteToFile took 257 ms to run on zoom 1
WriteToFile took 256 ms to run on zoom 2
WriteToFile took 254 ms to run on zoom 3
WriteToFile took 259 ms to run on zoom 4
WriteToFile took 254 ms to run on zoom 5
WriteToFile took 256 ms to run on zoom 6
WriteToFile took 257 ms to run on zoom 7
WriteToFile took 259 ms to run on zoom 8
WriteToFile took 257 ms to run on zoom 9
WriteToFile took 261 ms to run on zoom 10
WriteToFile took 236 ms to run on zoom 11
WriteToFile took 63 ms to run on zoom 12
WriteToFile took 68 ms to run on zoom 12
WriteToFile took 91 ms to run on zoom 12
WriteToFile took 94 ms to run on zoom 12
WriteToFile took 35 ms to run on zoom 13
WriteToFile took 50 ms to run on zoom 13
WriteToFile took 51 ms to run on zoom 13
WriteToFile took 35 ms to run on zoom 13
WriteToFile took 42 ms to run on zoom 13
WriteToFile took 72 ms to run on zoom 13
WriteToFile took 65 ms to run on zoom 13
WriteToFile took 47 ms to run on zoom 13
WriteToFile took 8 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 13 ms to run on zoom 14
WriteToFile took 13 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 10 ms to run on zoom 14
WriteToFile took 12 ms to run on zoom 14
WriteToFile took 32 ms to run on zoom 14
WriteToFile took 32 ms to run on zoom 14
WriteToFile took 32 ms to run on zoom 14
WriteToFile took 31 ms to run on zoom 14
WriteToFile took 30 ms to run on zoom 14
WriteToFile took 31 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
WriteToFile took 11 ms to run on zoom 14
WriteToFile took 29 ms to run on zoom 14
WriteToFile took 31 ms to run on zoom 14
WriteToFile took 30 ms to run on zoom 14
WriteToFile took 27 ms to run on zoom 14
WriteToFile took 29 ms to run on zoom 14
WriteToFile took 27 ms to run on zoom 14
WriteToFile took 13 ms to run on zoom 14
WriteToFile took 10 ms to run on zoom 14
WriteToFile took 24 ms to run on zoom 14
WriteToFile took 26 ms to run on zoom 14
WriteToFile took 26 ms to run on zoom 14
WriteToFile took 28 ms to run on zoom 14
WriteToFile took 27 ms to run on zoom 14
WriteToFile took 25 ms to run on zoom 14
WriteToFile took 14 ms to run on zoom 14
Done
Elapsed time: 4712

(Tested with a slightly modified program)

So it’s a bit slower than the 920 ms that forced the whole image into memory, but much faster than the timing you saw.

BTW, libvips 8.10 should fix the pixel shifts you noticed in https://github.com/kleisauke/net-vips/issues/25#issuecomment-453680434, see: https://libvips.github.io/libvips/2020/06/18/What’s-new-in-8.10.html#image-resize

I’ve adapted the above program to always use vips_resize. The 8.10.0-beta1 version of libvips will soon be published on NuGet.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Write to a file from multiple threads asynchronously c# - ...
This is done by specifying O_APPEND under Posix(Linux,Unix), and FILE_APPEND_DATA under Windows. In C# you don't call the OS 'open', or ' ...
Read more >
WriteFile() crashes when trying to carry on multithreading
I have added a multi-threading component to it and am trying to use the WriteFile() function to write to my serial port, but...
Read more >
Multithread support - Usage & Issues
Any way to encourage ImageJ to multithread calculations? Having a problem with performance issues on my PC. my images aren't terribly large, ...
Read more >
Performance Improvements in .NET 7
NET 7 is fast. Really fast. This post deep-dives into hundreds of performance improvements that contributed to that reality.
Read more >
Understanding the Windows I/O System
Within the operating system, mapped file I/O is used for important operations such as file caching and image activation (loading and running ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Possible TargetCustom Leak

Next page

Image.WriteToFile crashes with NetVips.Native 8.9.0-rc4