Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

v2.8.0 kubernetes.io~csi/alluxio-airflow-pv/mount file exists

See original GitHub issue

Alluxio Version: v2.8.0 (v2.7.1 same error)

Describe the bug Intermittent error when creating pods

[2022-05-25 21:31:13,119] {kubernetes_pod.py:366} INFO - creating pod with labels {'dag_id': 'etl_processing_dag', 'task_id': 'etl_gift_stats_also_buy_product', 'execution_date': '2022-05-24T2100000000-020f21c22', 'try_number': '6'} and launcher <airflow.providers.cncf.kubernetes.utils.pod_launcher.PodLauncher object at 0x7f31157e9cf8>
[2022-05-25 21:31:13,157] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Pending
[2022-05-25 21:31:13,157] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347
[2022-05-25 21:31:14,166] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Pending
[2022-05-25 21:31:14,166] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347
[2022-05-25 21:31:15,174] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Pending
[2022-05-25 21:31:15,174] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347
[2022-05-25 21:31:16,183] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Pending
[2022-05-25 21:31:16,183] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347
[2022-05-25 21:31:17,194] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Pending
[2022-05-25 21:31:17,194] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347
[2022-05-25 21:31:18,202] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Failed
[2022-05-25 21:31:18,202] {pod_launcher.py:308} ERROR - Event with job id etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 Failed
[2022-05-25 21:31:18,223] {pod_launcher.py:193} ERROR - Error parsing timestamp. Will continue execution but won't update timestamp
[2022-05-25 21:31:18,224] {pod_launcher.py:149} INFO - failed to try resolving symlinks in path "/var/log/pods/airflow_etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347_cf0d1c24-a325-433b-b371-464e63181b95/base/0.log": lstat /var/log/pods/airflow_etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347_cf0d1c24-a325-433b-b371-464e63181b95/base/0.log: no such file or directory
[2022-05-25 21:31:19,240] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Failed
[2022-05-25 21:31:19,240] {pod_launcher.py:308} ERROR - Event with job id etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 Failed
[2022-05-25 21:31:19,247] {pod_launcher.py:198} INFO - Event: etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 had an event of type Failed
[2022-05-25 21:31:19,247] {pod_launcher.py:308} ERROR - Event with job id etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347 Failed

'state': {'running': None,
                                              'terminated': {'container_id': 'docker://671240747c7f34a0718dc7b097dafa4f58fdfae6de7af1306626cb1b01e4fc43',
                                                             'exit_code': 128,
                                                             'finished_at': datetime.datetime(2022, 5, 25, 21, 31, 16, tzinfo=tzlocal()),
                                                             'message': 'error '
                                                                        'while '
                                                                        'creating '
                                                                        'mount '
                                                                        'source '
                                                                        'path '
                                                                        "'/var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': "
                                                                        'mkdir '
                                                                        '/var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: '
                                                                        'file '
                                                                        'exists',
                                                             'reason': 'ContainerCannotRun',
                                                             'signal': None,
                                                             'started_at': datetime.datetime(2022, 5, 25, 21, 31, 16, tzinfo=tzlocal())},
                                              'waiting': None}}],

To Reproduce see above

Expected behavior mount success

Urgency Describe the impact and urgency of the bug.

Are you planning to fix it No.

Additional context …

Issue Analytics

State:
Created a year ago
Comments:18 (8 by maintainers)

Top GitHub Comments

2reactions

ssz1997commented, Aug 8, 2022

@tak Glad to hear it’s resolved. Thanks for reporting!

1reaction

ssz1997commented, Jul 13, 2022

@takersk Seems like this is not a deterministic failure right? This pr https://github.com/Alluxio/alluxio/pull/15788 may fix it. You can try it out.

Top Results From Across the Web

Deployments - Kubernetes

A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the ......

file exists error for /etc/kubernetes when running w/ local ...

Running the "Local Docker-based (recommended starting point)" method from the upstream project, using this Vagrantfile: $ vagrant up ...

Unable to mount Azure file share - Microsoft Learn

Symptoms; Before troubleshooting; Mount error(2): No such file or ... When you run the kubectl describe pods command, you may see one of...

Kubernetes V2 Provider - Spinnaker

The kubeconfig file allows Spinnaker to authenticate against your cluster and to have read/write access to any resources you expect it to manage ......

Configuring an IP masquerade agent in Standard clusters

If the ip-masq-agent DaemonSet exists, then the output is similar to the following: ... image: gke.gcr.io/ip-masq-agent:v2.8.0