Import from a virtual disk in an external SSD is slow at roughly 5-9MBps inside a VM.

Observed behavior

Inside an Ubuntu 16.04.6 VM - Importing channel content (Khan Academy English about 19GB) from a local virtual disk is slow at roughly 5-9MBps whereas copying a folder of the same size yields >100MBps inside the same VM.

This local disk is a “virtual disk” that is saved in an external Seagate Fast SSD 500GB that’s supposed to have 540MBps read/write theoretical speeds. It’s connected via my USB Type-C to get the best performance.

Progress so far:

• 35 minutes = 30%
• 50 minutes = 37%
• 55 minutes = 40%

At this rate, it’ll take about 135+ minutes to finish importing the 19GB of channel content.

Is this normal behaviour or is there a bottleneck somewhere when importing in Kolibri?

Here’s the .gif of the Ubuntu VM with iotop while importing

Here’s the .gif of the Ubuntu VM with iotop while copying the folder inside the same VM

Expected behavior

I expected a faster than 9MBps read/write speed when importing content from the local virtual disk (saved in an external SSD) even when inside an Ubuntu VM.

Context

• Kolibri version: v0.13.2-beta1
• Operating system: Ubuntu 16.04.6 inside VMWare Fusion on a macOS Mojave 10.14.6
• Virtual RAM: 8GB
• Virtual Processors: 2
• Shared Graphics Memory: 768MB (as recommended by VMWare)
• Browser: Firefox v74.0

Related Issue/s:

This seems related to this issue Khan Academy (English) Import Metrics.

Note/s

• I just used the results from iotop for the 5-9MBps speed. I have no other means (yet) to determine why the import process is slower than I expected. 😸

[rtibbles]

It’s related, but a little different - seems like we should be coercing the total value to an int as well. Could you open a separate issue for this error, as I don’t think we should release with it.

[rtibbles]

In this case, you can just skip to the second step, if the database is already there.