What decides loading files order at file input plugin?

When I use embulk-input-file(embedded plugin) with following files

imitation_matter__c_load1_4_all.csv
imitation_matter__c_load1_4_minimal_null.csv
imitation_matter__c_load2_10000_minimal_null.csv
imitation_matter__c_load3_11000_minimal_null.csv

then “Loading files” order is different in server A and B. (both environments are CentOS 6.4 / Java 1.8.0_112 / embulk 0.8.15)

• Server A

imitation_matter__c_load1_4_all.csv
imitation_matter__c_load1_4_minimal_null.csv
imitation_matter__c_load2_10000_minimal_null.csv
imitation_matter__c_load3_11000_minimal_null.csv

• Server B

imitation_matter__c_load1_4_all.csv
imitation_matter__c_load2_10000_minimal_null.csv
imitation_matter__c_load3_11000_minimal_null.csv
imitation_matter__c_load1_4_minimal_null.csv

“Loading files” order seems to be the same find command(find <path> -name "imitation_matter__c_*") result. And find command result is different in server A and B.

I expected “Loading files” order is decided by filename order but it’ not. so what decides loading files order?(or order differs just in the logs?)

[noissefnoc]

• locale settings: same (checked by locale command)
• max_threads: 1: same result (it’s not multi-thread case)
• filesystem: same (ext4)

I also check different servers and I find filesystem mount options are different with different sort order.

• ? : Server A(I don’t have permission to check) and Windows7

imitation_matter__c_load1_4_all.csv
imitation_matter__c_load1_4_minimal_null.csv
imitation_matter__c_load2_10000_minimal_null.csv
imitation_matter__c_load3_11000_minimal_null.csv

• default : Server B order

imitation_matter__c_load1_4_all.csv
imitation_matter__c_load2_10000_minimal_null.csv
imitation_matter__c_load3_11000_minimal_null.csv
imitation_matter__c_load1_4_minimal_null.csv

• default,nobarrier :

imitation_matter__c_load3_10004_minimal_null.csv
imitation_matter__c_load2_10000_minimal_null.csv
imitation_matter__c_load1_4_minimal_null.csv
imitation_matter__c_load3_11000_minimal_null.csv
imitation_matter__c_load1_4_all.csv

• default,relatime,nobarrier :

imitation_matter__c_load2_10000_minimal_null.csv
imitation_matter__c_load3_11000_minimal_null.csv
imitation_matter__c_load1_4_all.csv
imitation_matter__c_load1_4_minimal_null.csv
imitation_matter__c_load3_10004_minimal_null.csv

I think this may affect sort order.

Anyway

• This may be not embulk issue
• “Loading files” order with single-thread follows find (also ls -U) order.

So I close this issue. Thanks @hiroyuki-sato @uu59

[hiroyuki-sato]

@noissefnoc

Thank you for sharing the information.

Could you clarify the “Loading files” meanings?

It is the following in embulk run? Correct?

Loading files [/private/tmp/hoge/csv/sample_01.csv.gz, /private/tmp/hoge/csv/sample_02.csv.gz]

I think, this list will sort after output the logs.

“Loading files” output part. https://github.com/embulk/embulk/blob/master/embulk-standards/src/main/java/org/embulk/standards/LocalFileInputPlugin.java#L64

sort part. https://github.com/embulk/embulk/blob/master/embulk-standards/src/main/java/org/embulk/standards/LocalFileInputPlugin.java#L92