Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

KubePodOperator sometimes fails with `Unable to parse string [failed]`

See original GitHub issue

Apache Airflow version: 2.0

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.15

Environment: Kubernetes, using the KubernetesExecutor as well

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
  • Kernel (e.g. uname -a): Linux airflow-7f86fbfdc-6s4l6 3.10.0-957.10.1.el7.x86_64
  • Others: I am using the official Airflow Docker Image

What happened:

I was trying to execute a DAG but for some reason it works at certain times but there are times it throws an error. Below I have attached two execution logs (I am executing the DAG twice, one after the other), the first execution throws a pendulum.parsing.exceptions.ParserError: Unable to parse string [failed] error.

Below is the log for the first execution

airflow@airflow-7f86fbfdc-6s4l6:/opt/airflow$ airflow dags test AtinXSxy 2020-10-10
[2020-12-01 01:30:57,766] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
[2020-12-01 01:30:58,308] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.warc-to-json 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:03,202] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=warc-to-json
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:03,443] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Pending
[2020-12-01 01:31:03,444] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-4a01397600d247a89026e9dca8b920f6
[2020-12-01 01:31:04,475] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Pending
[2020-12-01 01:31:04,475] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-4a01397600d247a89026e9dca8b920f6
[2020-12-01 01:31:05,533] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Pending
[2020-12-01 01:31:05,534] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-4a01397600d247a89026e9dca8b920f6
[2020-12-01 01:31:06,587] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Running
[2020-12-01 01:31:07,622] {pod_launcher.py:136} INFO - warc_to_json.py starting...
[2020-12-01 01:31:07,622] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/4/8/base.json`
[2020-12-01 01:31:09,628] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Succeeded
[2020-12-01 01:31:09,628] {pod_launcher.py:289} INFO - Event with job id warc-to-json-4a01397600d247a89026e9dca8b920f6 Succeeded
[2020-12-01 01:31:09,680] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Succeeded
[2020-12-01 01:31:09,681] {pod_launcher.py:289} INFO - Event with job id warc-to-json-4a01397600d247a89026e9dca8b920f6 Succeeded
[2020-12-01 01:31:09,818] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=warc-to-json, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013109
[2020-12-01 01:31:09,892] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:09,956] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 5 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2020-12-01 01:31:10,013] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-id 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:10,463] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-id
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:10,743] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Pending
[2020-12-01 01:31:10,744] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-34424163821f4c1595936541737eaf1b
[2020-12-01 01:31:11,781] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Pending
[2020-12-01 01:31:11,781] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-34424163821f4c1595936541737eaf1b
[2020-12-01 01:31:12,813] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Pending
[2020-12-01 01:31:12,813] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-34424163821f4c1595936541737eaf1b
[2020-12-01 01:31:13,846] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Running
[2020-12-01 01:31:17,186] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/5/9/id.json`
[2020-12-01 01:31:17,186] {pod_launcher.py:136} INFO - extract_id.py starting...
[2020-12-01 01:31:19,092] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Succeeded
[2020-12-01 01:31:19,093] {pod_launcher.py:289} INFO - Event with job id add-id-34424163821f4c1595936541737eaf1b Succeeded
[2020-12-01 01:31:19,123] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Succeeded
[2020-12-01 01:31:19,124] {pod_launcher.py:289} INFO - Event with job id add-id-34424163821f4c1595936541737eaf1b Succeeded
[2020-12-01 01:31:19,226] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-id, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013119
[2020-12-01 01:31:19,277] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:19,341] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 4 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 4
[2020-12-01 01:31:19,429] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-username-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:19,801] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-username-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:20,160] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:20,161] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:21,190] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:21,191] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:22,440] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:22,440] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:23,467] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:23,467] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:24,494] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:24,494] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:25,521] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Running
[2020-12-01 01:31:27,674] {pod_launcher.py:136} INFO - extract_username.py starting...
[2020-12-01 01:31:27,674] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/6/10/username.json`
[2020-12-01 01:31:29,565] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Succeeded
[2020-12-01 01:31:29,565] {pod_launcher.py:289} INFO - Event with job id add-username-field-77d091d0586e4ddda4ac67f1e4064612 Succeeded
[2020-12-01 01:31:29,594] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Succeeded
[2020-12-01 01:31:29,595] {pod_launcher.py:289} INFO - Event with job id add-username-field-77d091d0586e4ddda4ac67f1e4064612 Succeeded
[2020-12-01 01:31:29,683] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-username-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013129
[2020-12-01 01:31:29,719] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:29,756] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 3 | succeeded: 3 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 3
[2020-12-01 01:31:29,802] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-timestamp-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:29,958] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-timestamp-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:30,223] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:30,224] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:31,269] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:31,269] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:32,297] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:32,298] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:33,352] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:33,352] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:34,454] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Running
[2020-12-01 01:31:35,987] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/7/11/timestamp.json`
[2020-12-01 01:31:35,988] {pod_launcher.py:136} INFO - extract_timestamp.py starting...
[2020-12-01 01:31:37,865] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Succeeded
[2020-12-01 01:31:37,866] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 Succeeded
[2020-12-01 01:31:37,895] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Succeeded
[2020-12-01 01:31:37,895] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 Succeeded
[2020-12-01 01:31:37,994] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-timestamp-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013137
[2020-12-01 01:31:38,060] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:38,116] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 2 | succeeded: 4 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
[2020-12-01 01:31:38,165] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-hashtags-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:38,363] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-hashtags-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:38,562] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:38,563] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:39,591] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:39,591] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:40,622] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:40,622] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:41,651] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:41,652] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:42,686] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:42,686] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:43,725] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Running
[2020-12-01 01:31:46,063] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/8/12/hashtags.json`
[2020-12-01 01:31:46,064] {pod_launcher.py:136} INFO - extract_hashtags.py starting...
[2020-12-01 01:31:47,974] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Succeeded
[2020-12-01 01:31:47,974] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 Succeeded
[2020-12-01 01:31:48,008] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Succeeded
[2020-12-01 01:31:48,008] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 Succeeded
[2020-12-01 01:31:48,131] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-hashtags-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013148
[2020-12-01 01:31:48,187] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:48,250] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 1 | succeeded: 5 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 1
[2020-12-01 01:31:48,321] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-mentions-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:48,474] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-mentions-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:48,696] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Pending
[2020-12-01 01:31:48,696] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-faac94be589a41408bc2d3245c182866
[2020-12-01 01:31:49,804] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Pending
[2020-12-01 01:31:49,804] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-faac94be589a41408bc2d3245c182866
[2020-12-01 01:31:50,836] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Pending
[2020-12-01 01:31:50,836] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-faac94be589a41408bc2d3245c182866
[2020-12-01 01:31:51,901] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Running
[2020-12-01 01:31:52,003] {taskinstance.py:1402} ERROR - Unable to parse string [failed]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1087, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1306, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 340, in execute
    final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 486, in create_new_pod_for_operator
    final_state, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py", line 135, in monitor_pod
    last_log_time = pendulum.parse(timestamp)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 29, in parse
    return _parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 45, in _parse
    parsed = base_parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 74, in parse
    return _normalize(_parse(text, **_options), **_options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 128, in _parse
    raise ParserError("Unable to parse string [{}]".format(text))
pendulum.parsing.exceptions.ParserError: Unable to parse string [failed]
[2020-12-01 01:31:52,009] {taskinstance.py:1446} INFO - Marking task as FAILED. dag_id=AtinXSxy, task_id=add-mentions-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013152
[2020-12-01 01:31:52,037] {debug_executor.py:83} ERROR - Failed to execute task: Unable to parse string [failed].
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/debug_executor.py", line 78, in _run_task
    ti._run_raw_task(job_id=ti.job_id, **params)  # pylint: disable=protected-access
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 63, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1087, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1306, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 340, in execute
    final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 486, in create_new_pod_for_operator
    final_state, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py", line 135, in monitor_pod
    last_log_time = pendulum.parse(timestamp)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 29, in parse
    return _parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 45, in _parse
    parsed = base_parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 74, in parse
    return _normalize(_parse(text, **_options), **_options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 128, in _parse
    raise ParserError("Unable to parse string [{}]".format(text))
pendulum.parsing.exceptions.ParserError: Unable to parse string [failed]
[2020-12-01 01:31:52,064] {backfill_job.py:219} ERROR - Task instance <TaskInstance: AtinXSxy.add-mentions-field 2020-10-10 00:00:00+00:00 [failed]> failed
[2020-12-01 01:31:52,084] {dagrun.py:432} ERROR - Marking run <DagRun AtinXSxy @ 2020-10-10 00:00:00+00:00: backfill__2020-10-10T00:00:00+00:00, externally triggered: False> failed
[2020-12-01 01:31:52,088] {dagrun.py:598} WARNING - Failed to record first_task_scheduling_delay metric:
unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
[2020-12-01 01:31:52,093] {backfill_job.py:388} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 5 | running: 0 | failed: 1 | skipped: 0 | deadlocked: 0 | not ready: 0
Some task instances failed:
DAG ID    Task ID             Execution date               Try number
--------  ------------------  -------------------------  ------------
AtinXSxy  add-mentions-field  2020-10-10 00:00:00+00:00             1

Below is the log for the second execution

airflow@airflow-7f86fbfdc-6s4l6:/opt/airflow$ airflow dags test AtinXSxy 2020-10-10
[2020-12-01 01:41:56,401] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
[2020-12-01 01:41:56,917] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.warc-to-json 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:01,773] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=warc-to-json
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:02,051] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:02,052] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:03,090] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:03,091] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:04,191] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:04,191] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:05,234] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:05,234] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:06,314] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Running
[2020-12-01 01:42:06,848] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/4/8/base.json`
[2020-12-01 01:42:06,849] {pod_launcher.py:136} INFO - warc_to_json.py starting...
[2020-12-01 01:42:08,736] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Succeeded
[2020-12-01 01:42:08,736] {pod_launcher.py:289} INFO - Event with job id warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 Succeeded
[2020-12-01 01:42:08,766] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Succeeded
[2020-12-01 01:42:08,766] {pod_launcher.py:289} INFO - Event with job id warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 Succeeded
[2020-12-01 01:42:08,894] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=warc-to-json, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014208
[2020-12-01 01:42:08,959] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:09,039] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 5 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2020-12-01 01:42:09,124] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-id 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:09,396] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-id
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:09,687] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Pending
[2020-12-01 01:42:09,688] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-f429c835d2f340dd8566b1789bd538ab
[2020-12-01 01:42:10,864] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Pending
[2020-12-01 01:42:10,864] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-f429c835d2f340dd8566b1789bd538ab
[2020-12-01 01:42:11,895] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Pending
[2020-12-01 01:42:11,895] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-f429c835d2f340dd8566b1789bd538ab
[2020-12-01 01:42:12,925] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Running
[2020-12-01 01:42:16,396] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/5/9/id.json`
[2020-12-01 01:42:16,397] {pod_launcher.py:136} INFO - extract_id.py starting...
[2020-12-01 01:42:18,277] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Succeeded
[2020-12-01 01:42:18,278] {pod_launcher.py:289} INFO - Event with job id add-id-f429c835d2f340dd8566b1789bd538ab Succeeded
[2020-12-01 01:42:18,330] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Succeeded
[2020-12-01 01:42:18,330] {pod_launcher.py:289} INFO - Event with job id add-id-f429c835d2f340dd8566b1789bd538ab Succeeded
[2020-12-01 01:42:18,420] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-id, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014218
[2020-12-01 01:42:18,486] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:18,552] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 4 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 4
[2020-12-01 01:42:18,634] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-username-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:18,948] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-username-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:19,174] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Pending
[2020-12-01 01:42:19,175] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-27348aafaaba44b68474700497b94be2
[2020-12-01 01:42:20,205] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Pending
[2020-12-01 01:42:20,206] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-27348aafaaba44b68474700497b94be2
[2020-12-01 01:42:21,254] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Pending
[2020-12-01 01:42:21,254] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-27348aafaaba44b68474700497b94be2
[2020-12-01 01:42:22,285] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Running
[2020-12-01 01:42:24,051] {pod_launcher.py:136} INFO - extract_username.py starting...
[2020-12-01 01:42:24,051] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/6/10/username.json`
[2020-12-01 01:42:25,987] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Succeeded
[2020-12-01 01:42:25,987] {pod_launcher.py:289} INFO - Event with job id add-username-field-27348aafaaba44b68474700497b94be2 Succeeded
[2020-12-01 01:42:26,018] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Succeeded
[2020-12-01 01:42:26,019] {pod_launcher.py:289} INFO - Event with job id add-username-field-27348aafaaba44b68474700497b94be2 Succeeded
[2020-12-01 01:42:26,162] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-username-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014226
[2020-12-01 01:42:26,202] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:26,250] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 3 | succeeded: 3 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 3
[2020-12-01 01:42:26,306] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-timestamp-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:26,492] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-timestamp-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:26,776] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:26,777] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:27,811] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:27,817] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:28,854] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:28,855] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:29,892] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:29,893] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:30,931] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:30,935] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:32,061] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Running
[2020-12-01 01:42:33,223] {pod_launcher.py:136} INFO - extract_timestamp.py starting...
[2020-12-01 01:42:33,223] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/7/11/timestamp.json`
[2020-12-01 01:42:35,133] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Succeeded
[2020-12-01 01:42:35,134] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 Succeeded
[2020-12-01 01:42:35,179] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Succeeded
[2020-12-01 01:42:35,181] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 Succeeded
[2020-12-01 01:42:35,304] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-timestamp-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014235
[2020-12-01 01:42:35,355] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:35,388] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 2 | succeeded: 4 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
[2020-12-01 01:42:35,433] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-hashtags-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:35,558] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-hashtags-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:35,891] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:35,892] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:36,988] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:36,989] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:38,024] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:38,024] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:39,098] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:39,099] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:40,131] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Running
[2020-12-01 01:42:42,831] {pod_launcher.py:136} INFO - extract_hashtags.py starting...
[2020-12-01 01:42:42,832] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/8/12/hashtags.json`
[2020-12-01 01:42:44,725] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Succeeded
[2020-12-01 01:42:44,725] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-d47b7007849246eeb851cb55030c9a18 Succeeded
[2020-12-01 01:42:44,771] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Succeeded
[2020-12-01 01:42:44,771] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-d47b7007849246eeb851cb55030c9a18 Succeeded
[2020-12-01 01:42:44,849] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-hashtags-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014244
[2020-12-01 01:42:44,903] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:44,968] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 1 | succeeded: 5 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 1
[2020-12-01 01:42:45,030] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-mentions-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:45,134] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-mentions-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:45,340] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Pending
[2020-12-01 01:42:45,340] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b
[2020-12-01 01:42:46,373] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Pending
[2020-12-01 01:42:46,373] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b
[2020-12-01 01:42:47,405] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Pending
[2020-12-01 01:42:47,406] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b
[2020-12-01 01:42:48,438] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Running
[2020-12-01 01:42:51,813] {pod_launcher.py:136} INFO - extract_mentions.py starting...
[2020-12-01 01:42:51,814] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/9/13/mentions.json`
[2020-12-01 01:42:53,699] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Succeeded
[2020-12-01 01:42:53,699] {pod_launcher.py:289} INFO - Event with job id add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b Succeeded
[2020-12-01 01:42:53,732] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Succeeded
[2020-12-01 01:42:53,732] {pod_launcher.py:289} INFO - Event with job id add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b Succeeded
[2020-12-01 01:42:53,852] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-mentions-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014253
[2020-12-01 01:42:53,889] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:53,930] {dagrun.py:447} INFO - Marking run <DagRun AtinXSxy @ 2020-10-10 00:00:00+00:00: backfill__2020-10-10T00:00:00+00:00, externally triggered: False> successful
[2020-12-01 01:42:53,931] {dagrun.py:598} WARNING - Failed to record first_task_scheduling_delay metric:
unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
[2020-12-01 01:42:53,937] {backfill_job.py:388} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 6 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
[2020-12-01 01:42:53,962] {backfill_job.py:830} INFO - Backfill done. Exiting.

What you expected to happen: I expect the DAG to run always just like in the case of the second execution

Anything else we need to know: The error occurs quite randomly, in my testing I found out that the error comes once in every 4-5 DAG runs

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:20 (12 by maintainers)

github_iconTop GitHub Comments

3reactions
potiukcommented, Jun 14, 2021

Today!

1reaction
ashbcommented, May 3, 2021

@bmfisher thanks for those logs - unfortunately I need the pod logs to look at what the problematic line is, the airflow task logs just tell us where in the Airflow code the error is occurring, but I need to see the input line that is causing it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

[GitHub] [airflow] bmfisher commented on issue #12728
[GitHub] [airflow] bmfisher commented on issue #12728: KubePodOperator sometimes fails with `Unable to parse string [failed]`.
Read more >
Error: Unable to parse string "*" at position 6116 - Convert ...
After getting suggestions from #DYZ and #MaxU, it was an error due to the special character * in a row in in my...
Read more >
Release Notes — Airflow Documentation
py file in the DAGs folder is a zip compressed file, parsing it will fail with an exception. auth_backends includes session (#21640)¶. To...
Read more >
Changelog - Apache Airflow Documentation
Fixes failing formatting of DAG file containing {} in docstring (#9779) ... [AIRFLOW-7002] Get rid of yaml “parser” in bash (#7646).
Read more >
apache-airflow Changelog - pyup.io
Rewrite recursion when parsing DAG into iteration (25898) - Find cross-group tasks in ``iter_mapped_dependants`` (25793) - Fail task if mapping upstream ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

KubernetesPodOperator can't mount sercret as volume

Next page

No Support for special Characters in Passwords for get_uri() method in DbApiHook