Running with ruby system

Hi

I’m trying exec embulk using the system ruby command, but it is not working. If I run directly in terminal, it works.

The command is:

path_prefix='FILENAME' /Users/myuser/.embulk/bin/embulk run lib/embulk/repository.yml.liquid

The output is:

Ignoring bcrypt-3.1.11 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.11
Ignoring bcrypt-3.1.10 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.10
Ignoring binding_of_caller-0.7.2 because its extensions are not built.  Try: gem pristine binding_of_caller --version 0.7.2
Ignoring byebug-9.0.6 because its extensions are not built.  Try: gem pristine byebug --version 9.0.6
Ignoring byebug-9.0.5 because its extensions are not built.  Try: gem pristine byebug --version 9.0.5
Ignoring byebug-9.0.4 because its extensions are not built.  Try: gem pristine byebug --version 9.0.4
Ignoring byebug-9.0.0 because its extensions are not built.  Try: gem pristine byebug --version 9.0.0
Ignoring byebug-8.2.5 because its extensions are not built.  Try: gem pristine byebug --version 8.2.5
Ignoring byebug-8.2.4 because its extensions are not built.  Try: gem pristine byebug --version 8.2.4
Ignoring byebug-8.2.2 because its extensions are not built.  Try: gem pristine byebug --version 8.2.2
Ignoring byebug-8.2.1 because its extensions are not built.  Try: gem pristine byebug --version 8.2.1
Ignoring byebug-4.0.5 because its extensions are not built.  Try: gem pristine byebug --version 4.0.5
Ignoring debug_inspector-0.0.2 because its extensions are not built.  Try: gem pristine debug_inspector --version 0.0.2
Ignoring eventmachine-1.0.9.1 because its extensions are not built.  Try: gem pristine eventmachine --version 1.0.9.1
Ignoring ffi-1.9.14 because its extensions are not built.  Try: gem pristine ffi --version 1.9.14
Ignoring ffi-1.9.13 because its extensions are not built.  Try: gem pristine ffi --version 1.9.13
Ignoring ffi-1.9.10 because its extensions are not built.  Try: gem pristine ffi --version 1.9.10
Ignoring ffi-1.9.8 because its extensions are not built.  Try: gem pristine ffi --version 1.9.8
Ignoring hitimes-1.2.3 because its extensions are not built.  Try: gem pristine hitimes --version 1.2.3
Ignoring http_parser.rb-0.6.0 because its extensions are not built.  Try: gem pristine http_parser.rb --version 0.6.0
Ignoring json-1.8.2 because its extensions are not built.  Try: gem pristine json --version 1.8.2
Ignoring kgio-2.10.0 because its extensions are not built.  Try: gem pristine kgio --version 2.10.0
Ignoring libv8-3.16.14.13 because its extensions are not built.  Try: gem pristine libv8 --version 3.16.14.13
Ignoring libv8-3.3.10.4 because its extensions are not built.  Try: gem pristine libv8 --version 3.3.10.4
Ignoring mysql2-0.4.4 because its extensions are not built.  Try: gem pristine mysql2 --version 0.4.4
Ignoring mysql2-0.4.3 because its extensions are not built.  Try: gem pristine mysql2 --version 0.4.3
Ignoring mysql2-0.4.2 because its extensions are not built.  Try: gem pristine mysql2 --version 0.4.2
Ignoring mysql2-0.3.21 because its extensions are not built.  Try: gem pristine mysql2 --version 0.3.21
Ignoring mysql2-0.3.20 because its extensions are not built.  Try: gem pristine mysql2 --version 0.3.20
Ignoring nio4r-1.2.1 because its extensions are not built.  Try: gem pristine nio4r --version 1.2.1
Ignoring nokogiri-1.6.8 because its extensions are not built.  Try: gem pristine nokogiri --version 1.6.8
Ignoring nokogiri-1.6.7.2 because its extensions are not built.  Try: gem pristine nokogiri --version 1.6.7.2
Ignoring nokogiri-1.6.7.1 because its extensions are not built.  Try: gem pristine nokogiri --version 1.6.7.1
Ignoring nokogiri-1.6.6.2 because its extensions are not built.  Try: gem pristine nokogiri --version 1.6.6.2
Ignoring oj-2.17.1 because its extensions are not built.  Try: gem pristine oj --version 2.17.1
Ignoring oj-2.15.0 because its extensions are not built.  Try: gem pristine oj --version 2.15.0
Ignoring oj-2.14.3 because its extensions are not built.  Try: gem pristine oj --version 2.14.3
Ignoring oj-2.12.14 because its extensions are not built.  Try: gem pristine oj --version 2.12.14
Ignoring passenger-5.0.6 because its extensions are not built.  Try: gem pristine passenger --version 5.0.6
Ignoring pg-0.19.0 because its extensions are not built.  Try: gem pristine pg --version 0.19.0
Ignoring pg-0.18.4 because its extensions are not built.  Try: gem pristine pg --version 0.18.4
Ignoring pg-0.18.1 because its extensions are not built.  Try: gem pristine pg --version 0.18.1
Ignoring puma-3.6.0 because its extensions are not built.  Try: gem pristine puma --version 3.6.0
Ignoring puma-3.4.0 because its extensions are not built.  Try: gem pristine puma --version 3.4.0
Ignoring puma-2.16.0 because its extensions are not built.  Try: gem pristine puma --version 2.16.0
Ignoring puma-2.8.2 because its extensions are not built.  Try: gem pristine puma --version 2.8.2
Ignoring raindrops-0.16.0 because its extensions are not built.  Try: gem pristine raindrops --version 0.16.0
Ignoring raindrops-0.15.0 because its extensions are not built.  Try: gem pristine raindrops --version 0.15.0
Ignoring rmagick-2.15.4 because its extensions are not built.  Try: gem pristine rmagick --version 2.15.4
Ignoring sqlite3-1.3.11 because its extensions are not built.  Try: gem pristine sqlite3 --version 1.3.11
Ignoring therubyracer-0.12.2 because its extensions are not built.  Try: gem pristine therubyracer --version 0.12.2
Ignoring thin-1.5.1 because its extensions are not built.  Try: gem pristine thin --version 1.5.1
Ignoring tiny_tds-1.0.3 because its extensions are not built.  Try: gem pristine tiny_tds --version 1.0.3
Ignoring tiny_tds-0.7.0 because its extensions are not built.  Try: gem pristine tiny_tds --version 0.7.0
Ignoring unf_ext-0.0.7.2 because its extensions are not built.  Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicorn-5.1.0 because its extensions are not built.  Try: gem pristine unicorn --version 5.1.0
Ignoring unicorn-5.0.1 because its extensions are not built.  Try: gem pristine unicorn --version 5.0.1
Ignoring websocket-driver-0.6.4 because its extensions are not built.  Try: gem pristine websocket-driver --version 0.6.4
Ignoring websocket-driver-0.6.3 because its extensions are not built.  Try: gem pristine websocket-driver --version 0.6.3
Could not find gem 'tzinfo-data java' in any of the gem sources listed in your Gemfile or available on this machine.
Run `bundle install` to install missing gems.

Is there a limitation about it?

[thiagogsr]

It works perfectly, thank you @hiroyuki-sato 😃

[hiroyuki-sato]

@thiagogsr

I asked embulk users who is familiar about Bundler. Please try the following step.

Prepare embulk_bundle directory

embulk mkbundle embulk_bundle

Directory layout is the following.

.
|-- Gemfile
|-- Gemfile.lock
|-- Rakefile
|-- config.yml.liquid
|-- embulk_bundle # <-- NEW
`-- hoge
    |-- csv
    |   `-- sample_01.csv.gz
    `-- seed.yml

Edit embulk_bundle/Gemfile like the following.

source 'https://rubygems.org/'
gem 'embulk', '~> 0.8.0'
gem 'embulk-filter-row' # <-- your plugin

Install your plugins into embulk_bundle path with embulk bundle install

cd embulk_bundle
embulk bundle install

Edit Rake file

Use Bundler.with_clean_env and embulk -b embulk_bundle

task :run_embulk do
  filename = "./hoge/csv/sample"
  Bundler.with_clean_env do
    sh "path_prefix=#{filename} embulk run -b embulk_bundle config.yml.liquid"
  end
end

bundle exec rake run_embulk
path_prefix=./hoge/csv/sample embulk run -b embulk_bundle config.yml.liquid
2016-10-04 10:15:31.378 +0900: Embulk v0.8.13
2016-10-04 10:15:34.110 +0900 [INFO] (0001:transaction): Listing local files at directory 'hoge/csv' filtering filename by prefix 'sample'
2016-10-04 10:15:34.124 +0900 [INFO] (0001:transaction): Loading files [hoge/csv/sample_01.csv.gz]
2016-10-04 10:15:34.224 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=8 / output tasks 4 = input tasks 1 * 4
2016-10-04 10:15:34.237 +0900 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
1,32864,2015-01-27 19:23:49,20150127,embulk
2,14824,2015-01-27 19:01:23,20150127,embulk jruby
3,27559,2015-01-28 02:20:02,20150128,Embulk "csv" parser plugin
4,11270,2015-01-29 11:54:36,20150129,
2016-10-04 10:15:34.487 +0900 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
2016-10-04 10:15:34.495 +0900 [INFO] (main): Committed.
2016-10-04 10:15:34.495 +0900 [INFO] (main): Next config diff: {"in":{"last_path":"hoge/csv/sample_01.csv.gz"},"out":{}}