FileInputPlugin causes InvalidPathException by using Japanese filename

FileInputPlugin causes InvalidPathException by using Japanese filename (e.g. ほげ.csv). It seems that it’s influenced by the locale LANG.

vagrant@debian:~/docker-embulk$ docker exec test-embulk cat /var/log/cron.log
2017-03-31 13:56:05.856 +0000: Embulk v0.8.18
java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: ??????
        at sun.nio.fs.UnixPath.encode(sun/nio/fs/UnixPath.java:147)
        at sun.nio.fs.UnixPath.<init>(sun/nio/fs/UnixPath.java:71)
        at sun.nio.fs.UnixFileSystem.getPath(sun/nio/fs/UnixFileSystem.java:281)
        at java.nio.file.Paths.get(java/nio/file/Paths.java:84)
        at org.embulk.standards.LocalFileInputPlugin.listFiles(org/embulk/standards/LocalFileInputPlugin.java:107)
        at org.embulk.standards.LocalFileInputPlugin.transaction(org/embulk/standards/LocalFileInputPlugin.java:63)
        at org.embulk.spi.FileInputRunner.transaction(org/embulk/spi/FileInputRunner.java:65)
        at org.embulk.exec.SamplingParserPlugin.runFileInputSampling(org/embulk/exec/SamplingParserPlugin.java:34)
        at org.embulk.exec.PreviewExecutor.doPreview(org/embulk/exec/PreviewExecutor.java:96)
        at org.embulk.exec.PreviewExecutor.access$000(org/embulk/exec/PreviewExecutor.java:30)
        at org.embulk.exec.PreviewExecutor$1.run(org/embulk/exec/PreviewExecutor.java:70)
        at org.embulk.exec.PreviewExecutor$1.run(org/embulk/exec/PreviewExecutor.java:66)
        at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
...

I’m assuming to use this on docker container. As a workaround, I am setting locale to docker. But I don’t want platfrom settings comprecated.

Paths#get seems to cause Exception, do you have some way to fix it?

    public List<String> listFiles(PluginTask task)
    {
     Path pathPrefix = Paths.get(task.getPathPrefix()).normalize();

[koooge]

This problem was caused by cron. 😅 cf. http://www.mk-mode.com/octopress/2013/11/26/linux-cron-locale-behavior/ (japenese)

This issue is also same behavior.

(I wrote "locale >> /var/log/cron.log" in docker-entrypoint.sh)
vagrant@vagrant-debian-jessie:~/docker-recipes/embulk$ docker exec test-embulk cat /var/log/cron.log
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

As a result, I have to set locale in Dockerfile… Thank you for your help! 🙇‍♂️

[hiroyuki-sato]

Thank you for sharing your knowledge!