Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Plugin loading time

See original GitHub issue

Issue Type: Bug Report

Write the following environmental information.
- OS version: Windows 2016 8vCPU, 32G Memory
- Java version: 1.8
- Embulk version: 0.9.16
- Your Embulk configuration (YAML)
- Plugin versions
Write all what you did, e.g. your commands executed
Write what you expected and observed actually

Is there a way to reduce the bundle loading time? Below is the execution log from digdag

Task start : 3 sec Plugin bundle loading(BUNDLE_GEMFILE is being set --> Started Embulk): 9 sec – I looking for a solution to minimize this.

2019-03-20 22:10:18.059 +0000 [INFO] (6153@[0:edw_pipeline_streaming]+mssql_to_snowflake+execute_pipeline^sub+for-0=i=0=0+step1) io.digdag.core.agent.OperatorManager: sh>: embulk run .\scripts\config.yml.liquid -b c:/utils/embulk_bundle
2019-03-20 22:10:21.246 +0000: Embulk v0.9.16
2019-03-20 22:10:21.824 +0000 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2019-03-20 22:10:25.105 +0000 [INFO] (main): BUNDLE_GEMFILE is being set: "c:\utils\embulk_bundle\Gemfile"
2019-03-20 22:10:25.121 +0000 [INFO] (main): Gem's home and path are being cleared.
2019-03-20 22:10:30.371 +0000 [INFO] (main): Started Embulk v0.9.16
2019-03-20 22:10:30.527 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-sqlserver (0.9.3)
2019-03-20 22:10:30.558 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-azure_blob_storage (0.1.7)
2019-03-20 22:10:30.699 +0000 [INFO] (0001:transaction): Connecting to jdbc:sqlserver:

If I ran 7 embulk jobs at the same time, CPU spikes up to 100% and plugin loading time takes more than 30 seconds.

Issue Analytics

State:
Created 5 years ago
Comments:8 (3 by maintainers)

Top GitHub Comments

1reaction

kulmam92commented, Mar 22, 2019

I’m adding this for other people. I was able bring down the loading time in half by applying below parameters.

embulk “-R—dev” “-J-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Djruby.compile.mode=OFF” run myproject.yml.liquid

0reactions

nickolaycommented, Mar 31, 2019

Simple embulk -J-O run ... halved the startup time for me. I was confused by the -J-O Disable JVM optimizations to speed up startup time (enabled by default if command is 'run') note in embulk --help – it’s the JVM optimizations and slow startup are the default if the command is ‘run’, not the -J-O behavior…

[edit] forgot to mention that running java -Xshare:dump as suggested on the jruby wiki also shaved a few seconds.