Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Exception occurs in v0.9.1 using embulk-filter-add_time

See original GitHub issue

In v.0.9.1 using embulk-filter-add_time, requiring TimeZone Parameter but it could not find a configuration point. And this does not occur in v.0.9.0 so it may have some breaking changes. Is this bug?

https://github.com/treasure-data/embulk-filter-add_time

config.yml

in:
  type: file
  path_prefix: csv/sample_
  decoders:
  - {type: gzip}
  parser:
    charset: UTF-8
    newline: LF
    type: csv
    delimiter: ','
    quote: '"'
    escape: '"'
    null_string: 'NULL'
    trim_if_not_quoted: false
    skip_header_lines: 1
    allow_extra_columns: false
    allow_optional_columns: false
    columns:
    - {name: id, type: long}
    - {name: account, type: long}
    - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
    - {name: purchase, type: timestamp, format: '%Y%m%d'}
    - {name: comment, type: string}
filters:
- type: add_time
  to_column:
    name: time
    type: timestamp
  from_value:
    value: "2016-01-01 00:00:00 UTC"
  TimeZone: UTC
out: {type: stdout}

StackTrace

bash-3.2$ embulk preview config.yml
2018-02-10 22:38:11.129 +0900: Embulk v0.9.1

********************************** INFORMATION **********************************
  Join us! Embulk-announce mailing list is up for IMPORTANT announcement such as
    compatibility-breaking changes and key feature updates.
  https://groups.google.com/forum/#!forum/embulk-announce
*********************************************************************************

2018-02-10 22:38:12.917 +0900 [INFO] (main): Gem's home and path are set by default: "/Users/smdmts/.embulk/lib/gems"
2018-02-10 22:38:13.397 +0900 [INFO] (main): Started Embulk v0.9.1
2018-02-10 22:38:13.460 +0900 [INFO] (0001:preview): Loaded plugin embulk-filter-add_time (0.2.0)
2018-02-10 22:38:13.488 +0900 [INFO] (0001:preview): Listing local files at directory 'csv' filtering filename by prefix 'sample_'
2018-02-10 22:38:13.489 +0900 [INFO] (0001:preview): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.
2018-02-10 22:38:13.493 +0900 [INFO] (0001:preview): Loading files [csv/sample_01.csv.gz]
2018-02-10 22:38:13.501 +0900 [INFO] (0001:preview): Try to read 32,768 bytes from input source
2018-02-10 22:38:13.576 +0900 [WARN] (0001:preview): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
java.lang.IllegalArgumentException: Field 'TimeZone' is required but not set
 at [Source: N/A; line: -1, column: -1]
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3459)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3400)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3388)
    at org.embulk.config.TaskSerDe$TaskDeserializer.deserialize(TaskSerDe.java:152)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3708)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2005)
    at org.embulk.config.ModelManager.readObject(ModelManager.java:45)
    at org.embulk.config.DataSourceImpl.loadTask(DataSourceImpl.java:192)
    at org.embulk.filter.add_time.AddTimeFilterPlugin.open(AddTimeFilterPlugin.java:168)
    at org.embulk.spi.util.Filters.open(Filters.java:50)
    at org.embulk.exec.PreviewExecutor$2$1.run(PreviewExecutor.java:127)
    at org.embulk.spi.util.Filters$RecursiveControl.transaction(Filters.java:84)
    at org.embulk.spi.util.Filters$RecursiveControl$1.run(Filters.java:80)
    at org.embulk.filter.add_time.AddTimeFilterPlugin.transaction(AddTimeFilterPlugin.java:161)
    at org.embulk.spi.util.Filters$RecursiveControl.transaction(Filters.java:76)
    at org.embulk.spi.util.Filters.transaction(Filters.java:42)
    at org.embulk.exec.PreviewExecutor$2.run(PreviewExecutor.java:118)
    at org.embulk.spi.FileInputRunner$RunnerControl$1$1.run(FileInputRunner.java:112)
    at org.embulk.standards.CsvParserPlugin.transaction(CsvParserPlugin.java:228)
    at org.embulk.spi.FileInputRunner$RunnerControl$1.run(FileInputRunner.java:107)
    at org.embulk.spi.util.Decoders$RecursiveControl.transaction(Decoders.java:68)
    at org.embulk.spi.util.Decoders$RecursiveControl$1.run(Decoders.java:64)
    at org.embulk.standards.GzipFileDecoderPlugin.transaction(GzipFileDecoderPlugin.java:25)
    at org.embulk.spi.util.Decoders$RecursiveControl.transaction(Decoders.java:60)
    at org.embulk.spi.util.Decoders.transaction(Decoders.java:29)
    at org.embulk.spi.FileInputRunner$RunnerControl.run(FileInputRunner.java:105)
    at org.embulk.exec.BufferFileInputPlugin.transaction(BufferFileInputPlugin.java:21)
    at org.embulk.spi.FileInputRunner.transaction(FileInputRunner.java:62)
    at org.embulk.exec.PreviewExecutor.doPreview(PreviewExecutor.java:116)
    at org.embulk.exec.PreviewExecutor.doPreview(PreviewExecutor.java:102)
    at org.embulk.exec.PreviewExecutor.access$000(PreviewExecutor.java:30)
    at org.embulk.exec.PreviewExecutor$1.run(PreviewExecutor.java:74)
    at org.embulk.exec.PreviewExecutor$1.run(PreviewExecutor.java:71)
    at org.embulk.spi.Exec.doWith(Exec.java:22)
    at org.embulk.exec.PreviewExecutor.preview(PreviewExecutor.java:71)
    at org.embulk.EmbulkEmbed.preview(EmbulkEmbed.java:152)
    at org.embulk.EmbulkRunner.previewInternal(EmbulkRunner.java:214)
    at org.embulk.EmbulkRunner.preview(EmbulkRunner.java:106)
    at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:433)
    at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:91)
    at org.embulk.cli.Main.main(Main.java:26)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Field 'TimeZone' is required but not set
 at [Source: N/A; line: -1, column: -1]
    at org.embulk.config.TaskSerDe$TaskDeserializer.deserialize(TaskSerDe.java:180)
    at com.fasterxml.jackson.datatype.guava.deser.GuavaOptionalDeserializer.deserialize(GuavaOptionalDeserializer.java:118)
    at com.fasterxml.jackson.datatype.guava.deser.GuavaOptionalDeserializer.deserialize(GuavaOptionalDeserializer.java:16)
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3454)
    ... 40 more

Error: Field 'TimeZone' is required but not set
 at [Source: N/A; line: -1, column: -1]
bash-3.2$ embulk preview config.yml
2018-02-10 22:39:04.763 +0900: Embulk v0.9.1

2018-02-10 22:47:36.437 +0900: Embulk v0.9.0

********************************** INFORMATION **********************************
  Join us! Embulk-announce mailing list is up for IMPORTANT announcement such as
    compatibility-breaking changes and key feature updates.
  https://groups.google.com/forum/#!forum/embulk-announce
*********************************************************************************

2018-02-10 22:47:38.277 +0900 [INFO] (main): Gem's home and path are set by default: "/Users/smdmts/.embulk/lib/gems"
2018-02-10 22:47:38.863 +0900 [INFO] (main): Started Embulk v0.9.0
2018-02-10 22:47:38.926 +0900 [INFO] (0001:preview): Loaded plugin embulk-filter-add_time (0.2.0)
2018-02-10 22:47:38.950 +0900 [INFO] (0001:preview): Listing local files at directory 'csv' filtering filename by prefix 'sample_'
2018-02-10 22:47:38.951 +0900 [INFO] (0001:preview): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.
2018-02-10 22:47:38.956 +0900 [INFO] (0001:preview): Loading files [csv/sample_01.csv.gz]
2018-02-10 22:47:38.963 +0900 [INFO] (0001:preview): Try to read 32,768 bytes from input source
2018-02-10 22:47:39.044 +0900 [WARN] (0001:preview): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-10 22:47:39.058 +0900 [WARN] (0001:preview): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
+---------+--------------+-------------------------+-------------------------+----------------------------+-------------------------+
| id:long | account:long |         time_:timestamp |      purchase:timestamp |             comment:string |          time:timestamp |
+---------+--------------+-------------------------+-------------------------+----------------------------+-------------------------+
|       1 |       32,864 | 2015-01-27 19:23:49 UTC | 2015-01-27 00:00:00 UTC |                     embulk | 2016-01-01 00:00:00 UTC |
|       2 |       14,824 | 2015-01-27 19:01:23 UTC | 2015-01-27 00:00:00 UTC |               embulk jruby | 2016-01-01 00:00:00 UTC |
|       3 |       27,559 | 2015-01-28 02:20:02 UTC | 2015-01-28 00:00:00 UTC | Embulk "csv" parser plugin | 2016-01-01 00:00:00 UTC |
|       4 |       11,270 | 2015-01-29 11:54:36 UTC | 2015-01-29 00:00:00 UTC |                            | 2016-01-01 00:00:00 UTC |
+---------+--------------+-------------------------+-------------------------+----------------------------+-------------------------+

Issue Analytics

State:
Created 6 years ago
Comments:6 (6 by maintainers)

Top GitHub Comments

1reaction

smdmtscommented, Feb 14, 2018

@dmikurube It seems to fix the latest version and M/R executor as well.Thanks a lot.

2018-02-14 11:50:17.886 +0900: Embulk v0.9.3

********************************** INFORMATION **********************************
  Join us! Embulk-announce mailing list is up for IMPORTANT announcement such as
    compatibility-breaking changes and key feature updates.
  https://groups.google.com/forum/#!forum/embulk-announce
*********************************************************************************

2018-02-14 11:50:18.880 +0900 [INFO] (main): Started Embulk v0.9.3
2018-02-14 11:50:21.479 +0900 [INFO] (0001:transaction): Gem's home and path are set by default: "/Users/smdmts/.embulk/lib/gems"
2018-02-14 11:50:22.348 +0900 [INFO] (0001:transaction): Loaded plugin embulk-filter-add_time (0.2.0)
2018-02-14 11:50:22.383 +0900 [INFO] (0001:transaction): Listing local files at directory '/Users/smdmts/work/embulk/csv' filtering filename by prefix 'sample_'
2018-02-14 11:50:22.385 +0900 [INFO] (0001:transaction): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.
2018-02-14 11:50:22.388 +0900 [INFO] (0001:transaction): Loading files [/Users/smdmts/work/embulk/csv/sample_01.csv.gz]
2018-02-14 11:50:22.447 +0900 [WARN] (0001:transaction): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.474 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=16 / output tasks 8 = input tasks 1 * 8
2018-02-14 11:50:22.480 +0900 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
2018-02-14 11:50:22.503 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.560 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.561 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.562 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.563 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.565 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.567 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
2018-02-14 11:50:22.569 +0900 [WARN] (0016:task-0000): to_column 'time' is set but 'time' column also exists. The existent 'time' column is renamed to 'time_'.
1,32864,2015-01-27 19:23:49,20150127,embulk,2016-01-01 00:00:00.000000 +0000
2,14824,2015-01-27 19:01:23,20150127,embulk jruby,2016-01-01 00:00:00.000000 +0000
3,27559,2015-01-28 02:20:02,20150128,Embulk "csv" parser plugin,2016-01-01 00:00:00.000000 +0000
4,11270,2015-01-29 11:54:36,20150129,,2016-01-01 00:00:00.000000 +0000
2018-02-14 11:50:22.604 +0900 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
2018-02-14 11:50:22.610 +0900 [INFO] (main): Committed.
2018-02-14 11:50:22.611 +0900 [INFO] (main): Next config diff: {"in":{"last_path":"/Users/smdmts/work/embulk/csv/sample_01.csv.gz"},"out":{}}

2018-02-14 03:28:52.287 +0000: Embulk v0.9.3

********************************** INFORMATION **********************************
  Join us! Embulk-announce mailing list is up for IMPORTANT announcement such as
    compatibility-breaking changes and key feature updates.
  https://groups.google.com/forum/#!forum/embulk-announce
*********************************************************************************

2018-02-14 03:28:55.888 +0000 [INFO] (main): Started Embulk v0.9.3
2018-02-14 03:29:02.769 +0000 [INFO] (0001:transaction): Gem's home and path are set by default: "/home/hadoop/.embulk/lib/gems"
2018-02-14 03:29:05.417 +0000 [INFO] (0001:transaction): Loaded plugin embulk-executor-mapreduce (0.3.0)
2018-02-14 03:29:05.590 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-gcs (0.2.5)
2018-02-14 03:29:05.771 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-s3 (1.5.0)
2018-02-14 03:29:05.867 +0000 [INFO] (0001:transaction): Loaded plugin embulk-filter-add_time (0.2.0)
2018-02-14 03:29:06.041 +0000 [INFO] (0001:transaction): Loaded plugin embulk-parser-firebase_avro (0.1.2)
2018-02-14 03:29:10.080 +0000 [INFO] (0001:transaction): Loaded plugin embulk-formatter-jsonl (0.1.4)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2018-02-14 03:29:29.669 +0000 [INFO] (0001:transaction): Starting job =xxxx
2018-02-14 03:29:29.692 +0000 [INFO] (0001:transaction): map 0.0% reduce 0.0%
2018-02-14 03:29:34.714 +0000 [INFO] (0001:transaction): map 0.0% reduce 0.0%
2018-02-14 03:29:39.773 +0000 [INFO] (0001:transaction): map 0.0% reduce 0.0%

1reaction

dmikurubecommented, Feb 10, 2018

@smdmts Got it. Thanks for reporting this.

It looks caused by #956 – removing a dirty hack introduced to relax deprecations. But @Config annotations are not inherited to sub-interfaces.

I will revert it, and release v0.9.2.

Top Results From Across the Web

Exception handling best practices call for secure code design

9.1 Using exceptions to deal with failure ... As part of this, an exception is thrown if no account is found or if...

8. Errors and Exceptions — Python 3.11.1 documentation

If an exception occurs which does not match the exception named in the except clause, it is passed on to outer try statements;...

What Is an Exception? (The Java™ Tutorials > Essential Java ...

Definition: An exception is an event, which occurs during the execution of a program, that disrupts the normal flow of the program's instructions....

Exceptions, HTTP Errors, and Redirects - The Pylons Project

The view is called with the exception object as its context, and whatever response the view returns will be sent to the browser....

9.1. Nios® II Exception Handling Overview - Intel

Miscellaneous exceptions. The Nios® II processor offers two distinct approaches to handling hardware interrupts: The internal interrupt controller (IIC); The ...