Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Should Evaluator run _before_ ModelValidator?
See original GitHub issue
Looking at how the pipeline is arranged, it seems like the Evaluator should run before the Model Validator. However, when I run the pipeline, the ModelValidator gets executed first, and consequently fails.
-
It seems like the
model_analyzer = Evaluator()is used only in the return value of_create_pipeline(). Unlike the other components where the inputs and outputs are plugged into other components. -
As previously mentioned, my ModelValidator is failing, though it doesn’t have a baseline to compare with, which I assume means that it would give it’s “blessing”. Since training and evaluation both succeeded (at least the Trainer component is green), I’m having trouble understanding the issue.
Here’s the log:
[2019-05-16 16:13:47,699] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: darkrai.ModelValidator.darkrai.ModelValidator.exec 2019-05-16T08:02:44.896105+00:00 [queued]>
[2019-05-16 16:13:47,712] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: darkrai.ModelValidator.darkrai.ModelValidator.exec 2019-05-16T08:02:44.896105+00:00 [queued]>
[2019-05-16 16:13:47,712] {__init__.py:1353} INFO -
--------------------------------------------------------------------------------
[2019-05-16 16:13:47,712] {__init__.py:1354} INFO - Starting attempt 1 of 1
[2019-05-16 16:13:47,712] {__init__.py:1355} INFO -
--------------------------------------------------------------------------------
[2019-05-16 16:13:47,732] {__init__.py:1374} INFO - Executing <Task(PythonOperator): darkrai.ModelValidator.exec> on 2019-05-16T08:02:44.896105+00:00
[2019-05-16 16:13:47,733] {base_task_runner.py:119} INFO - Running: ['airflow', 'run', 'darkrai.ModelValidator', 'darkrai.ModelValidator.exec', '2019-05-16T08:02:44.896105+00:00', '--job_id', '1910', '--raw', '-sd', 'DAGS_FOLDER/darkrai_pipeline.py', '--cfg_path', '/tmp/tmp5wchsjgx']
[2019-05-16 16:13:48,950] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [2019-05-16 16:13:48,948] {__init__.py:51} INFO - Using executor SequentialExecutor
[2019-05-16 16:13:49,382] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [2019-05-16 16:13:49,380] {__init__.py:305} INFO - Filling up the DagBag from /home/benjamintan/workspace/darkrai/pipeline/dags/darkrai_pipeline.py
[2019-05-16 16:13:54,670] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [2019-05-16 16:13:54,670] {settings.py:50} WARNING - Path for coco pretrained h5 could not be found...
[2019-05-16 16:13:54,670] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [2019-05-16 16:13:54,670] {settings.py:53} WARNING - DATA dirpath PATH_TO_THE_DATA_DIR could not be found !
[2019-05-16 16:13:54,670] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [2019-05-16 16:13:54,670] {settings.py:56} WARNING - DATA_OTHER dirpath PATH_TO_THE_DATA_DIR could not be found !
[2019-05-16 16:13:54,720] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [2019-05-16 16:13:54,720] {deprecation.py:323} WARNING - From /home/benjamintan/workspace/darkrai/pipeline/dags/darkrai_pipeline.py:53: PipelineDecorator (from tfx.orchestration.pipeline) is deprecated and will be removed in a future version.
[2019-05-16 16:13:54,720] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec Instructions for updating:
[2019-05-16 16:13:54,720] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec PipelineDecorator is no longer needed. Please construct a pipeline directly from a list of components using the constructor call to pipeline.Pipeline.
[2019-05-16 16:13:54,747] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec <tfx.components.base.base_component.ComponentOutputs object at 0x7f552f856390>
[2019-05-16 16:13:54,747] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [2019-05-16 16:13:54,746] {cli.py:517} INFO - Running <TaskInstance: darkrai.ModelValidator.darkrai.ModelValidator.exec 2019-05-16T08:02:44.896105+00:00 [running]> on host dnnserver2
[2019-05-16 16:13:54,758] {python_operator.py:104} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_ID=darkrai.ModelValidator
AIRFLOW_CTX_TASK_ID=darkrai.ModelValidator.exec
AIRFLOW_CTX_EXECUTION_DATE=2019-05-16T08:02:44.896105+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill_2019-05-16T08:02:44.896105+00:00
[2019-05-16 16:13:54,772] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,772] {base_executor.py:102} INFO - Starting Executor execution.
[2019-05-16 16:13:54,773] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,773] {base_executor.py:104} INFO - Inputs for Executor is: {"examples": [{"artifact": {"id": "24", "typeId": "2", "uri": "/home/benjamintan/tfx/pipelines/darkrai/ImportExampleGen/examples/35/train/", "properties": {"split": {"stringValue": "train"}, "state": {"stringValue": "published"}, "span": {"intValue": "1"}, "type_name": {"stringValue": "ExamplesPath"}}}, "artifact_type": {"name": "ExamplesPath", "properties": {"name": "STRING", "type_name": "STRING", "split": "STRING", "state": "STRING", "span": "INT"}}}, {"artifact": {"id": "25", "typeId": "2", "uri": "/home/benjamintan/tfx/pipelines/darkrai/ImportExampleGen/examples/35/eval/", "properties": {"state": {"stringValue": "published"}, "span": {"intValue": "1"}, "type_name": {"stringValue": "ExamplesPath"}, "split": {"stringValue": "eval"}}}, "artifact_type": {"name": "ExamplesPath", "properties": {"type_name": "STRING", "split": "STRING", "state": "STRING", "span": "INT", "name": "STRING"}}}], "model": [{"artifact": {"id": "34", "typeId": "14", "uri": "/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/", "properties": {"state": {"stringValue": "published"}, "span": {"intValue": "1"}, "type_name": {"stringValue": "ModelExportPath"}, "split": {"stringValue": ""}}}, "artifact_type": {"name": "ModelExportPath", "properties": {"state": "STRING", "span": "INT", "name": "STRING", "type_name": "STRING", "split": "STRING"}}}]}
[2019-05-16 16:13:54,773] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,773] {base_executor.py:106} INFO - Outputs for Executor is: {"blessing": [{"artifact": {"uri": "/home/benjamintan/tfx/pipelines/darkrai/ModelValidator/blessing/52/", "properties": {"split": {"stringValue": ""}, "span": {"intValue": "1"}, "type_name": {"stringValue": "ModelBlessingPath"}}}, "artifact_type": {"name": "ModelBlessingPath", "properties": {"name": "STRING", "type_name": "STRING", "split": "STRING", "state": "STRING", "span": "INT"}}}]}
[2019-05-16 16:13:54,774] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,773] {base_executor.py:108} INFO - Execution properties for Executor is: {"blessed_model": null, "blessed_model_id": null}
[2019-05-16 16:13:54,774] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,774] {base_executor.py:94} INFO - Creating temp directory at /home/benjamintan/tfx/pipelines/darkrai/.temp/52/
[2019-05-16 16:13:54,774] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,774] {executor.py:134} INFO - Using temp path /home/benjamintan/tfx/pipelines/darkrai/.temp/52/ for tft.beam
[2019-05-16 16:13:54,774] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,774] {executor.py:141} INFO - Using /home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/ as current model.
[2019-05-16 16:13:54,774] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,774] {executor.py:148} INFO - Using None as blessed model.
[2019-05-16 16:13:54,774] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,774] {executor.py:153} INFO - Validating model.
[2019-05-16 16:13:54,776] {logging_mixin.py:95} INFO - [2019-05-16 16:13:54,776] {pipeline.py:143} INFO - Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
[2019-05-16 16:13:54,866] {logging_mixin.py:95} WARNING - /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/slicer/slicer.py:407: BeamDeprecationWarning: RemoveDuplicates is deprecated since 2.12. Use Distinct instead.
| 'IncrementCounter' >> beam.Map(increment_counter))
[2019-05-16 16:13:55,703] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,703] {fn_api_runner_transforms.py:468} INFO - ==================== <function annotate_downstream_side_inputs at 0x7f552f332840> ====================
[2019-05-16 16:13:55,705] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,705] {fn_api_runner_transforms.py:468} INFO - ==================== <function fix_side_input_pcoll_coders at 0x7f552f332950> ====================
[2019-05-16 16:13:55,707] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,707] {fn_api_runner_transforms.py:468} INFO - ==================== <function lift_combiners at 0x7f552f3329d8> ====================
[2019-05-16 16:13:55,710] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,710] {fn_api_runner_transforms.py:468} INFO - ==================== <function expand_sdf at 0x7f552f332a60> ====================
[2019-05-16 16:13:55,712] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,711] {fn_api_runner_transforms.py:468} INFO - ==================== <function expand_gbk at 0x7f552f332ae8> ====================
[2019-05-16 16:13:55,714] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,714] {fn_api_runner_transforms.py:468} INFO - ==================== <function sink_flattens at 0x7f552f332bf8> ====================
[2019-05-16 16:13:55,714] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,714] {fn_api_runner_transforms.py:468} INFO - ==================== <function greedily_fuse at 0x7f552f332c80> ====================
[2019-05-16 16:13:55,722] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,722] {fn_api_runner_transforms.py:468} INFO - ==================== <function read_to_impulse at 0x7f552f332d08> ====================
[2019-05-16 16:13:55,723] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,723] {fn_api_runner_transforms.py:468} INFO - ==================== <function impulse_to_input at 0x7f552f332d90> ====================
[2019-05-16 16:13:55,724] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,724] {fn_api_runner_transforms.py:468} INFO - ==================== <function inject_timer_pcollections at 0x7f552f332f28> ====================
[2019-05-16 16:13:55,725] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,725] {fn_api_runner_transforms.py:468} INFO - ==================== <function sort_stages at 0x7f552f334048> ====================
[2019-05-16 16:13:55,726] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,726] {fn_api_runner_transforms.py:468} INFO - ==================== <function window_pcollection_coders at 0x7f552f3340d0> ====================
[2019-05-16 16:13:55,737] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,737] {fn_api_runner.py:427} INFO - Running (((ref_AppliedPTransform_EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/CreateEvalConfig/Read_97)+(ref_AppliedPTransform_EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/WriteEvalConfig/Write/WriteImpl/Map(<lambda at iobase.py:984>)_104))+(ref_AppliedPTransform_EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/WriteEvalConfig/Write/WriteImpl/WindowInto(WindowIntoFn)_105))+(EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/WriteEvalConfig/Write/WriteImpl/GroupByKey/Write)
[2019-05-16 16:13:55,771] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,771] {fn_api_runner.py:427} INFO - Running (((ref_AppliedPTransform_EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/WriteEvalConfig/Write/WriteImpl/DoOnce/Read_102)+(ref_AppliedPTransform_EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/WriteEvalConfig/Write/WriteImpl/InitializeWrite_103))+(ref_PCollection_PCollection_54/Write))+(ref_PCollection_PCollection_55/Write)
[2019-05-16 16:13:55,786] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,786] {fn_api_runner.py:427} INFO - Running ((EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/WriteEvalConfig/Write/WriteImpl/GroupByKey/Read)+(ref_AppliedPTransform_EvalCurrentModel/WriteEvalConfig(EvalConfig(model_location='/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527', data_location='<user provided PCollection>', slice_spec=[SingleSliceSpec(columns=frozenset(), features=frozenset())], example_weight_metric_key='post_export_metrics/example_count', num_bootstrap_samples=1))/WriteEvalConfig/Write/WriteImpl/WriteBundles_110))+(ref_PCollection_PCollection_61/Write)
[2019-05-16 16:13:55,796] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,796] {tfrecordio.py:54} WARNING - Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
[2019-05-16 16:13:55,803] {logging_mixin.py:95} INFO - [2019-05-16 16:13:55,803] {fn_api_runner.py:427} INFO - Running ((((((((((((ref_AppliedPTransform_ReadData/Read_3)+(ref_AppliedPTransform_EvalCurrentModel/InputsToExtracts/Map(<lambda at model_eval_lib.py:393>)_6))+(ref_AppliedPTransform_EvalCurrentModel/ExtractAndEvaluate/Predict/Batch/ParDo(_GlobalWindowsBatchingDoFn)_10))+(ref_AppliedPTransform_EvalCurrentModel/ExtractAndEvaluate/Predict/Predict_11))+(ref_AppliedPTransform_EvalCurrentModel/ExtractAndEvaluate/ExtractSliceKeys/ParDo(_ExtractSliceKeysFn)_13))+(ref_AppliedPTransform_EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/Filter/Map(filter_extracts)_16))+(ref_AppliedPTransform_EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots/FanoutSlices/DoSlicing_19))+(ref_AppliedPTransform_EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots/FanoutSlices/TrackDistinctSliceKeys/ExtractSliceKeys_21))+(EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots/ComputePerSliceMetrics/CombinePerSlice/Precombine))+(ref_AppliedPTransform_EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots/FanoutSlices/TrackDistinctSliceKeys/RemoveDuplicates/RemoveDuplicates/ToPairs_24))+(EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots/FanoutSlices/TrackDistinctSliceKeys/RemoveDuplicates/RemoveDuplicates/Group/Precombine))+(EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots/FanoutSlices/TrackDistinctSliceKeys/RemoveDuplicates/RemoveDuplicates/Group/Group/Write))+(EvalCurrentModel/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots/ComputePerSliceMetrics/CombinePerSlice/Group/Write)
[2019-05-16 16:13:56,680] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.680611: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55d663aa0b90 executing computations on platform CUDA. Devices:
[2019-05-16 16:13:56,681] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.680747: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce GTX 1060 6GB, Compute Capability 6.1
[2019-05-16 16:13:56,704] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.704328: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 1995350000 Hz
[2019-05-16 16:13:56,707] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.706898: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55d663b0c370 executing computations on platform Host. Devices:
[2019-05-16 16:13:56,707] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.706977: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
[2019-05-16 16:13:56,707] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.707530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
[2019-05-16 16:13:56,707] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7085
[2019-05-16 16:13:56,708] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec pciBusID: 0000:05:00.0
[2019-05-16 16:13:56,708] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec totalMemory: 5.93GiB freeMemory: 5.86GiB
[2019-05-16 16:13:56,708] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.707587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
[2019-05-16 16:13:56,709] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.709272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
[2019-05-16 16:13:56,709] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.709320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
[2019-05-16 16:13:56,709] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.709344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
[2019-05-16 16:13:56,710] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 2019-05-16 16:13:56.709767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5703 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:05:00.0, compute capability: 6.1)
[2019-05-16 16:13:56,713] {logging_mixin.py:95} INFO - [2019-05-16 16:13:56,713] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
[2019-05-16 16:13:58,570] {logging_mixin.py:95} INFO - [2019-05-16 16:13:58,570] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
[2019-05-16 16:13:58,571] {logging_mixin.py:95} INFO - [2019-05-16 16:13:58,571] {saver.py:1270} INFO - Restoring parameters from /home/benjamintan/tfx/pipelines/darkrai/Trainer/output/50/eval_model_dir/1557993527/variables/variables
[2019-05-16 16:14:00,168] {logging_mixin.py:95} INFO - [2019-05-16 16:14:00,168] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
[2019-05-16 16:14:00,251] {logging_mixin.py:95} INFO - [2019-05-16 16:14:00,251] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[2019-05-16 16:14:00,540] {logging_mixin.py:95} INFO - [2019-05-16 16:14:00,538] {post_export_metrics.py:478} INFO - ExampleCount post export metric: could not find any of the standard keys in predictions_dict (keys were: dict_keys([]))
[2019-05-16 16:14:00,540] {logging_mixin.py:95} INFO - [2019-05-16 16:14:00,540] {post_export_metrics.py:491} INFO - Using the first key from labels_dict: gt_boxes
[2019-05-16 16:14:01,789] {__init__.py:1580} ERROR - tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 2 values, but the requested shape has 1
[[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]]
Caused by op 'Reshape', defined at:
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/bin/airflow", line 32, in <module>
args.func(args)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
_run(args, dag, ti)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, in _run
pool=args.pool,
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 112, in execute
return_value = self.execute_callable()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 117, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/benjamintan/workspace/tfx/tfx/orchestration/airflow/airflow_adapter.py", line 155, in python_exec
executor.Do(self._input_dict, self._output_dict, self._exec_properties)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 159, in Do
blessed_model_dir=blessed_model_dir)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 89, in _generate_blessing_result
output_path=blessed_model_eval_result_path))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 426, in __exit__
self.run().wait_until_finish()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 419, in run
return self.runner.run_pipeline(self, self._options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 276, in run_pipeline
default_environment=self._default_environment))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 280, in run_via_runner_api
return self.run_stages(*self.create_stages(pipeline_proto))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 356, in run_stages
stage_context.safe_coders)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 511, in run_stage
data_input, data_output)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 1217, in process_bundle
result_future = self._controller.control_handler.push(process_bundle)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 832, in push
response = self.worker.do_instruction(request)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 312, in do_instruction
request.instruction_id)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 331, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 538, in process_bundle
op.start()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 73, in start_bundle
construct_fn(self._model_load_seconds)))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 221, in acquire
return _shared_map.acquire(self._key, constructor_fn)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 183, in acquire
result = control_block.acquire(constructor_fn)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 85, in acquire
result = constructor_fn()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 43, in construct
additional_fetches=additional_fetches)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 90, in __init__
super(EvalSavedModel, self).__init__()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_metrics_graph/eval_metrics_graph.py", line 106, in __init__
self._construct_graph()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 150, in _construct_graph
self._session, [constants.EVAL_TAG], self._path)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 269, in load
return loader.load(sess, tags, import_scope, **saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 420, in load
**saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 350, in load_graph
meta_graph_def, import_scope=import_scope, **saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
**kwargs))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
return_elements=return_elements)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
for c_op in c_api_util.new_tf_operations(self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 2 values, but the requested shape has 1
[[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]] [while running 'EvalCurrentModel/ExtractAndEvaluate/Predict/Predict']
Traceback (most recent call last):
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 2 values, but the requested shape has 1
[[{{node Reshape}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 846, in apache_beam.runners.common._OutputProcessor.process_outputs
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/extractors/predict_extractor.py", line 87, in process
for fetched in self._eval_saved_model.predict_list(serialized_examples):
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 308, in predict_list
additional_fetches) = self._predict_list_fn(*input_args)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1204, in _generic_run
return self.run(fetches, feed_dict=feed_dict, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 2 values, but the requested shape has 1
[[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]]
Caused by op 'Reshape', defined at:
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/bin/airflow", line 32, in <module>
args.func(args)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
_run(args, dag, ti)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, in _run
pool=args.pool,
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 112, in execute
return_value = self.execute_callable()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 117, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/benjamintan/workspace/tfx/tfx/orchestration/airflow/airflow_adapter.py", line 155, in python_exec
executor.Do(self._input_dict, self._output_dict, self._exec_properties)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 159, in Do
blessed_model_dir=blessed_model_dir)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 89, in _generate_blessing_result
output_path=blessed_model_eval_result_path))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 426, in __exit__
self.run().wait_until_finish()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 419, in run
return self.runner.run_pipeline(self, self._options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 276, in run_pipeline
default_environment=self._default_environment))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 280, in run_via_runner_api
return self.run_stages(*self.create_stages(pipeline_proto))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 356, in run_stages
stage_context.safe_coders)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 511, in run_stage
data_input, data_output)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 1217, in process_bundle
result_future = self._controller.control_handler.push(process_bundle)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 832, in push
response = self.worker.do_instruction(request)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 312, in do_instruction
request.instruction_id)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 331, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 538, in process_bundle
op.start()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 73, in start_bundle
construct_fn(self._model_load_seconds)))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 221, in acquire
return _shared_map.acquire(self._key, constructor_fn)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 183, in acquire
result = control_block.acquire(constructor_fn)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 85, in acquire
result = constructor_fn()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 43, in construct
additional_fetches=additional_fetches)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 90, in __init__
super(EvalSavedModel, self).__init__()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_metrics_graph/eval_metrics_graph.py", line 106, in __init__
self._construct_graph()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 150, in _construct_graph
self._session, [constants.EVAL_TAG], self._path)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 269, in load
return loader.load(sess, tags, import_scope, **saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 420, in load
**saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 350, in load_graph
meta_graph_def, import_scope=import_scope, **saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
**kwargs))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
return_elements=return_elements)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
for c_op in c_api_util.new_tf_operations(self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 2 values, but the requested shape has 1
[[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 112, in execute
return_value = self.execute_callable()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 117, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/benjamintan/workspace/tfx/tfx/orchestration/airflow/airflow_adapter.py", line 155, in python_exec
executor.Do(self._input_dict, self._output_dict, self._exec_properties)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 159, in Do
blessed_model_dir=blessed_model_dir)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 89, in _generate_blessing_result
output_path=blessed_model_eval_result_path))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 426, in __exit__
self.run().wait_until_finish()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 419, in run
return self.runner.run_pipeline(self, self._options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 276, in run_pipeline
default_environment=self._default_environment))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 280, in run_via_runner_api
return self.run_stages(*self.create_stages(pipeline_proto))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 356, in run_stages
stage_context.safe_coders)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 511, in run_stage
data_input, data_output)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 1217, in process_bundle
result_future = self._controller.control_handler.push(process_bundle)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 832, in push
response = self.worker.do_instruction(request)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 312, in do_instruction
request.instruction_id)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 331, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 554, in process_bundle
].process_encoded(data.data)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 140, in process_encoded
self.output(decoded_value)
File "apache_beam/runners/worker/operations.py", line 245, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 246, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 396, in apache_beam.runners.worker.operations.ImpulseReadOperation.process
File "apache_beam/runners/worker/operations.py", line 403, in apache_beam.runners.worker.operations.ImpulseReadOperation.process
File "apache_beam/runners/worker/operations.py", line 246, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 560, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 561, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 740, in apache_beam.runners.common.DoFnRunner.receive
File "apache_beam/runners/common.py", line 746, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 785, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 870, in apache_beam.runners.common._OutputProcessor.process_outputs
File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 560, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 561, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 740, in apache_beam.runners.common.DoFnRunner.receive
File "apache_beam/runners/common.py", line 746, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 785, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 870, in apache_beam.runners.common._OutputProcessor.process_outputs
File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 560, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 561, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 740, in apache_beam.runners.common.DoFnRunner.receive
File "apache_beam/runners/common.py", line 746, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 800, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/future/utils/__init__.py", line 419, in raise_with_traceback
raise exc.with_traceback(traceback)
File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 846, in apache_beam.runners.common._OutputProcessor.process_outputs
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/extractors/predict_extractor.py", line 87, in process
for fetched in self._eval_saved_model.predict_list(serialized_examples):
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 308, in predict_list
additional_fetches) = self._predict_list_fn(*input_args)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1204, in _generic_run
return self.run(fetches, feed_dict=feed_dict, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
RuntimeError: tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 2 values, but the requested shape has 1
[[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]]
Caused by op 'Reshape', defined at:
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/bin/airflow", line 32, in <module>
args.func(args)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
_run(args, dag, ti)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, in _run
pool=args.pool,
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 112, in execute
return_value = self.execute_callable()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 117, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/benjamintan/workspace/tfx/tfx/orchestration/airflow/airflow_adapter.py", line 155, in python_exec
executor.Do(self._input_dict, self._output_dict, self._exec_properties)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 159, in Do
blessed_model_dir=blessed_model_dir)
File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 89, in _generate_blessing_result
output_path=blessed_model_eval_result_path))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 426, in __exit__
self.run().wait_until_finish()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 419, in run
return self.runner.run_pipeline(self, self._options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 276, in run_pipeline
default_environment=self._default_environment))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 280, in run_via_runner_api
return self.run_stages(*self.create_stages(pipeline_proto))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 356, in run_stages
stage_context.safe_coders)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 511, in run_stage
data_input, data_output)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 1217, in process_bundle
result_future = self._controller.control_handler.push(process_bundle)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 832, in push
response = self.worker.do_instruction(request)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 312, in do_instruction
request.instruction_id)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 331, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 538, in process_bundle
op.start()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 73, in start_bundle
construct_fn(self._model_load_seconds)))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 221, in acquire
return _shared_map.acquire(self._key, constructor_fn)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 183, in acquire
result = control_block.acquire(constructor_fn)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 85, in acquire
result = constructor_fn()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 43, in construct
additional_fetches=additional_fetches)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 90, in __init__
super(EvalSavedModel, self).__init__()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_metrics_graph/eval_metrics_graph.py", line 106, in __init__
self._construct_graph()
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 150, in _construct_graph
self._session, [constants.EVAL_TAG], self._path)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 269, in load
return loader.load(sess, tags, import_scope, **saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 420, in load
**saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 350, in load_graph
meta_graph_def, import_scope=import_scope, **saver_kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
**kwargs))
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
return_elements=return_elements)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
for c_op in c_api_util.new_tf_operations(self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 2 values, but the requested shape has 1
[[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]] [while running 'EvalCurrentModel/ExtractAndEvaluate/Predict/Predict']
[2019-05-16 16:14:01,795] {__init__.py:1611} INFO - Marking task as FAILED.
[2019-05-16 16:14:01,821] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/__init__.py:84: UserWarning: Running the Apache Beam SDK on Python 3 is not yet fully supported. You may encounter buggy behavior or missing features.
[2019-05-16 16:14:01,821] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec 'Running the Apache Beam SDK on Python 3 is not yet fully supported. '
[2019-05-16 16:14:01,827] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec Traceback (most recent call last):
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return fn(*args)
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec options, feed_dict, fetch_list, target_list, run_metadata)
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec run_metadata)
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 2 values, but the requested shape has 1
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [[{{node Reshape}}]]
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,828] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec During handling of the above exception, another exception occurred:
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec Traceback (most recent call last):
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 846, in apache_beam.runners.common._OutputProcessor.process_outputs
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/extractors/predict_extractor.py", line 87, in process
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for fetched in self._eval_saved_model.predict_list(serialized_examples):
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 308, in predict_list
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec additional_fetches) = self._predict_list_fn(*input_args)
[2019-05-16 16:14:01,829] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1204, in _generic_run
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.run(fetches, feed_dict=feed_dict, **kwargs)
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec run_metadata_ptr)
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec feed_dict_tensor, options, run_metadata)
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec run_metadata)
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec raise type(e)(node_def, op, message)
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 2 values, but the requested shape has 1
[2019-05-16 16:14:01,830] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]]
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec Caused by op 'Reshape', defined at:
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/bin/airflow", line 32, in <module>
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec args.func(args)
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return f(*args, **kwargs)
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec _run(args, dag, ti)
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, in _run
[2019-05-16 16:14:01,831] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec pool=args.pool,
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return func(*args, **kwargs)
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result = task_copy.execute(context=context)
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 112, in execute
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return_value = self.execute_callable()
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 117, in execute_callable
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.python_callable(*self.op_args, **self.op_kwargs)
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/orchestration/airflow/airflow_adapter.py", line 155, in python_exec
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec executor.Do(self._input_dict, self._output_dict, self._exec_properties)
[2019-05-16 16:14:01,832] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 159, in Do
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec blessed_model_dir=blessed_model_dir)
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 89, in _generate_blessing_result
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec output_path=blessed_model_eval_result_path))
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 426, in __exit__
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self.run().wait_until_finish()
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 419, in run
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.runner.run_pipeline(self, self._options)
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return runner.run_pipeline(pipeline, options)
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 276, in run_pipeline
[2019-05-16 16:14:01,833] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec default_environment=self._default_environment))
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 280, in run_via_runner_api
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.run_stages(*self.create_stages(pipeline_proto))
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 356, in run_stages
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec stage_context.safe_coders)
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 511, in run_stage
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec data_input, data_output)
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 1217, in process_bundle
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result_future = self._controller.control_handler.push(process_bundle)
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 832, in push
[2019-05-16 16:14:01,834] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec response = self.worker.do_instruction(request)
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 312, in do_instruction
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec request.instruction_id)
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 331, in process_bundle
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec bundle_processor.process_bundle(instruction_id))
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 538, in process_bundle
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec op.start()
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 73, in start_bundle
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec construct_fn(self._model_load_seconds)))
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 221, in acquire
[2019-05-16 16:14:01,835] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return _shared_map.acquire(self._key, constructor_fn)
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 183, in acquire
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result = control_block.acquire(constructor_fn)
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 85, in acquire
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result = constructor_fn()
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 43, in construct
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec additional_fetches=additional_fetches)
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 90, in __init__
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec super(EvalSavedModel, self).__init__()
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_metrics_graph/eval_metrics_graph.py", line 106, in __init__
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self._construct_graph()
[2019-05-16 16:14:01,836] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 150, in _construct_graph
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self._session, [constants.EVAL_TAG], self._path)
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return func(*args, **kwargs)
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 269, in load
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return loader.load(sess, tags, import_scope, **saver_kwargs)
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 420, in load
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec **saver_kwargs)
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 350, in load_graph
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec meta_graph_def, import_scope=import_scope, **saver_kwargs)
[2019-05-16 16:14:01,837] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec **kwargs))
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return_elements=return_elements)
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return func(*args, **kwargs)
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec _ProcessNewOps(graph)
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
[2019-05-16 16:14:01,838] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for c_op in c_api_util.new_tf_operations(self)
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for c_op in c_api_util.new_tf_operations(self)
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec ret = Operation(c_op, self)
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self._traceback = tf_stack.extract_stack()
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 2 values, but the requested shape has 1
[2019-05-16 16:14:01,839] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]]
[2019-05-16 16:14:01,841] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,841] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec During handling of the above exception, another exception occurred:
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec Traceback (most recent call last):
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/bin/airflow", line 32, in <module>
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec args.func(args)
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return f(*args, **kwargs)
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec _run(args, dag, ti)
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, in _run
[2019-05-16 16:14:01,842] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec pool=args.pool,
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return func(*args, **kwargs)
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result = task_copy.execute(context=context)
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 112, in execute
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return_value = self.execute_callable()
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 117, in execute_callable
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.python_callable(*self.op_args, **self.op_kwargs)
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/orchestration/airflow/airflow_adapter.py", line 155, in python_exec
[2019-05-16 16:14:01,843] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec executor.Do(self._input_dict, self._output_dict, self._exec_properties)
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 159, in Do
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec blessed_model_dir=blessed_model_dir)
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 89, in _generate_blessing_result
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec output_path=blessed_model_eval_result_path))
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 426, in __exit__
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self.run().wait_until_finish()
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 419, in run
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.runner.run_pipeline(self, self._options)
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return runner.run_pipeline(pipeline, options)
[2019-05-16 16:14:01,844] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 276, in run_pipeline
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec default_environment=self._default_environment))
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 280, in run_via_runner_api
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.run_stages(*self.create_stages(pipeline_proto))
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 356, in run_stages
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec stage_context.safe_coders)
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 511, in run_stage
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec data_input, data_output)
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 1217, in process_bundle
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result_future = self._controller.control_handler.push(process_bundle)
[2019-05-16 16:14:01,845] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 832, in push
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec response = self.worker.do_instruction(request)
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 312, in do_instruction
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec request.instruction_id)
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 331, in process_bundle
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec bundle_processor.process_bundle(instruction_id))
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 554, in process_bundle
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec ].process_encoded(data.data)
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 140, in process_encoded
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self.output(decoded_value)
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 245, in apache_beam.runners.worker.operations.Operation.output
[2019-05-16 16:14:01,846] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 246, in apache_beam.runners.worker.operations.Operation.output
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 396, in apache_beam.runners.worker.operations.ImpulseReadOperation.process
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 403, in apache_beam.runners.worker.operations.ImpulseReadOperation.process
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 246, in apache_beam.runners.worker.operations.Operation.output
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 560, in apache_beam.runners.worker.operations.DoOperation.process
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 561, in apache_beam.runners.worker.operations.DoOperation.process
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 740, in apache_beam.runners.common.DoFnRunner.receive
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 746, in apache_beam.runners.common.DoFnRunner.process
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 785, in apache_beam.runners.common.DoFnRunner._reraise_augmented
[2019-05-16 16:14:01,847] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 870, in apache_beam.runners.common._OutputProcessor.process_outputs
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 560, in apache_beam.runners.worker.operations.DoOperation.process
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 561, in apache_beam.runners.worker.operations.DoOperation.process
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 740, in apache_beam.runners.common.DoFnRunner.receive
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 746, in apache_beam.runners.common.DoFnRunner.process
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 785, in apache_beam.runners.common.DoFnRunner._reraise_augmented
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
[2019-05-16 16:14:01,848] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 870, in apache_beam.runners.common._OutputProcessor.process_outputs
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 142, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 560, in apache_beam.runners.worker.operations.DoOperation.process
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/worker/operations.py", line 561, in apache_beam.runners.worker.operations.DoOperation.process
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 740, in apache_beam.runners.common.DoFnRunner.receive
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 746, in apache_beam.runners.common.DoFnRunner.process
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 800, in apache_beam.runners.common.DoFnRunner._reraise_augmented
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/future/utils/__init__.py", line 419, in raise_with_traceback
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec raise exc.with_traceback(traceback)
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 744, in apache_beam.runners.common.DoFnRunner.process
[2019-05-16 16:14:01,849] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 422, in apache_beam.runners.common.SimpleInvoker.invoke_process
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "apache_beam/runners/common.py", line 846, in apache_beam.runners.common._OutputProcessor.process_outputs
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/extractors/predict_extractor.py", line 87, in process
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for fetched in self._eval_saved_model.predict_list(serialized_examples):
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 308, in predict_list
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec additional_fetches) = self._predict_list_fn(*input_args)
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1204, in _generic_run
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.run(fetches, feed_dict=feed_dict, **kwargs)
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec run_metadata_ptr)
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
[2019-05-16 16:14:01,850] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec feed_dict_tensor, options, run_metadata)
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec run_metadata)
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec raise type(e)(node_def, op, message)
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec RuntimeError: tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 2 values, but the requested shape has 1
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]]
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec Caused by op 'Reshape', defined at:
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/bin/airflow", line 32, in <module>
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec args.func(args)
[2019-05-16 16:14:01,851] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return f(*args, **kwargs)
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 523, in run
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec _run(args, dag, ti)
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/bin/cli.py", line 442, in _run
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec pool=args.pool,
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return func(*args, **kwargs)
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/models/__init__.py", line 1441, in _run_raw_task
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result = task_copy.execute(context=context)
[2019-05-16 16:14:01,852] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 112, in execute
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return_value = self.execute_callable()
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 117, in execute_callable
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.python_callable(*self.op_args, **self.op_kwargs)
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/orchestration/airflow/airflow_adapter.py", line 155, in python_exec
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec executor.Do(self._input_dict, self._output_dict, self._exec_properties)
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 159, in Do
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec blessed_model_dir=blessed_model_dir)
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/workspace/tfx/tfx/components/model_validator/executor.py", line 89, in _generate_blessing_result
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec output_path=blessed_model_eval_result_path))
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 426, in __exit__
[2019-05-16 16:14:01,853] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self.run().wait_until_finish()
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/pipeline.py", line 419, in run
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.runner.run_pipeline(self, self._options)
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/direct/direct_runner.py", line 132, in run_pipeline
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return runner.run_pipeline(pipeline, options)
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 276, in run_pipeline
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec default_environment=self._default_environment))
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 280, in run_via_runner_api
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return self.run_stages(*self.create_stages(pipeline_proto))
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 356, in run_stages
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec stage_context.safe_coders)
[2019-05-16 16:14:01,854] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 511, in run_stage
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec data_input, data_output)
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 1217, in process_bundle
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result_future = self._controller.control_handler.push(process_bundle)
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/portability/fn_api_runner.py", line 832, in push
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec response = self.worker.do_instruction(request)
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 312, in do_instruction
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec request.instruction_id)
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/sdk_worker.py", line 331, in process_bundle
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec bundle_processor.process_bundle(instruction_id))
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/apache_beam-2.12.0-py3.6-linux-x86_64.egg/apache_beam/runners/worker/bundle_processor.py", line 538, in process_bundle
[2019-05-16 16:14:01,855] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec op.start()
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 73, in start_bundle
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec construct_fn(self._model_load_seconds)))
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 221, in acquire
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return _shared_map.acquire(self._key, constructor_fn)
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 183, in acquire
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result = control_block.acquire(constructor_fn)
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_transform-0.13.0-py3.6.egg/tensorflow_transform/beam/shared.py", line 85, in acquire
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec result = constructor_fn()
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/dofn.py", line 43, in construct
[2019-05-16 16:14:01,856] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec additional_fetches=additional_fetches)
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 90, in __init__
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec super(EvalSavedModel, self).__init__()
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_metrics_graph/eval_metrics_graph.py", line 106, in __init__
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self._construct_graph()
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py", line 150, in _construct_graph
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self._session, [constants.EVAL_TAG], self._path)
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return func(*args, **kwargs)
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 269, in load
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return loader.load(sess, tags, import_scope, **saver_kwargs)
[2019-05-16 16:14:01,857] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 420, in load
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec **saver_kwargs)
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 350, in load_graph
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec meta_graph_def, import_scope=import_scope, **saver_kwargs)
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec **kwargs))
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return_elements=return_elements)
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec return func(*args, **kwargs)
[2019-05-16 16:14:01,858] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec _ProcessNewOps(graph)
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for c_op in c_api_util.new_tf_operations(self)
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec for c_op in c_api_util.new_tf_operations(self)
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec ret = Operation(c_op, self)
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
[2019-05-16 16:14:01,859] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec self._traceback = tf_stack.extract_stack()
[2019-05-16 16:14:01,860] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec
[2019-05-16 16:14:01,860] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 2 values, but the requested shape has 1
[2019-05-16 16:14:01,860] {base_task_runner.py:101} INFO - Job 1910: Subtask darkrai.ModelValidator.exec [[node Reshape (defined at /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow_model_analysis-0.13.2-py3.6.egg/tensorflow_model_analysis/eval_saved_model/load.py:150) ]] [while running 'EvalCurrentModel/ExtractAndEvaluate/Predict/Predict']
[2019-05-16 16:14:02,707] {logging_mixin.py:95} INFO - [2019-05-16 16:14:02,705] {jobs.py:2630} WARNING - State of this instance has been externally set to failed. Taking the poison pill.
[2019-05-16 16:14:02,723] {helpers.py:281} INFO - Sending Signals.SIGTERM to GPID 23997
[2019-05-16 16:14:02,937] {helpers.py:263} INFO - Process psutil.Process(pid=24027, status='terminated') (24027) terminated with exit code None
[2019-05-16 16:14:03,110] {helpers.py:263} INFO - Process psutil.Process(pid=23997, status='terminated') (23997) terminated with exit code -15
[2019-05-16 16:14:03,112] {logging_mixin.py:95} INFO - [2019-05-16 16:14:03,110] {jobs.py:2562} INFO - Task exited with return code 0
Issue Analytics
- State:
- Created 4 years ago
- Comments:8 (8 by maintainers)
Top Results From Across the Web
Combining ModelValidator with Evaluator - GitHub
With a given evaluation run, we can now import two models: one candidate model and one baseline model. The evaluation will not only ......
Read more >The ModelValidator TFX Pipeline Component (Deprecated)
We still think that validation is useful, but since the model Evaluator has already computed all the metrics you want to validate against...
Read more >Using the `evaluator` - Hugging Face
The text classification evaluator can be used to evaluate text models on classification datasets such as IMDb. Beside the model, data, and metric...
Read more >Quick start — PyTorch-Ignite v0.4.10 Documentation
Objects trainer and evaluator are instances of Engine - main component of Ignite. Engine is an abstraction over your training/validation loop.
Read more >PyTorch-Ignite: training and evaluating neural networks ...
Similarly, model evaluation can be done with an engine that runs a single time over the validation dataset and computes metrics.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@benjamintanweihao There are two options you can do that:
After many days of fighting with TFX, success!
The main thing was to use
tf.parse_exampleinstead oftf.parse_single_example.