Git Sync for DAGs fails

Apache Airflow version: 2.1.2

Kubernetes version (if you are using kubernetes) (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“19”, GitVersion:“v1.19.3”, GitCommit:“1e11e4a2108024935ecfcb2912226cedeafd99df”, GitTreeState:“clean”, BuildDate:“2020-10-14T18:49:28Z”, GoVersion:“go1.15.2”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“20+”, GitVersion:“v1.20.8-gke.700”, GitCommit:“8924a6876b2ff4905020e178132365eca3164869”, GitTreeState:“clean”, BuildDate:“2021-06-28T09:20:44Z”, GoVersion:“go1.15.13b5”, Compiler:“gc”, Platform:“linux/amd64”}

Environment:

• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a): GCP kubernetes cluster
• Install tools: helm chart
• https://github.com/apache/airflow/tree/main/chart
• Others: We are trying to deploy apache airflow on GCP GKE cluster using this helm chart.

What happened: trying to sync DAGs files from our internal git repository and it fails with the following error:

(base) i039273@C02X92M3JGH6 apache-airflow % kubectl get pods NAME READY STATUS RESTARTS AGE airflow-scheduler-6b475bbc8b-6vhlt 2/3 Error 1 17s airflow-statsd-6bbb58d7dd-77tnq 1/1 Running 0 6m22s airflow-webserver-5c578bcf5d-4fss6 1/1 Running 0 4m59s

(base) i039273@C02X92M3JGH6 apache-airflow % kubectl logs airflow-scheduler-6b475bbc8b-6vhlt -c git-sync INFO: detected pid 1, running init handler I0721 23:01:17.731667 12 main.go:495] “level”=0 “msg”=“starting up” “args”=[“/git-sync”] “pid”=12 I0721 23:01:17.731729 12 main.go:784] “level”=0 “msg”=“cloning repo” “origin”=“https://github.tools.sap/I039273/airflow-dags.git” “path”=“/git” E0721 23:01:18.153810 12 main.go:168] “msg”=“too many failures, aborting” “error”=“Run(git clone --no-checkout -b master --depth 1 https://github.tools.sap/I039273/airflow-dags.git /git): exit status 128: { stdout: "", stderr: "Cloning into ‘/git’…\nfatal: could not read Username for ‘https://github.tools.sap’: No such device or address\n" }” “failCount”=0 (

values.yaml gitSync: enabled: true repo: https://github.tools.sap/I039273/airflow-dags.git branch: master
rev: HEAD
root: “/git”
dest: “repo”
depth: 1
subPath: “dags” credentialsSecret: git-credentials

git-user.yaml apiVersion: v1 kind: Secret metadata: name: git-credentials data: #username: my_username GIT_SYNC_USERNAME: 64bit encryted value of my_username #password: my_password GIT_SYNC_PASSWORD: 64bit encrypted value of my_password

What you expected to happen:

exit status 128: { stdout: "", stderr: "Cloning into ‘/git’…\nfatal: could not read Username for ‘https://github.tools.sap’: No such device or address\n" }"

When we use exactly the same DAG git repository with the community helm chart, it works fine.

How to reproduce it:

Well, git repository requires username and password, using HTTP git URL, try to set up git Sync.

Anything else we need to know:

How often does this problem occur? Every time Any relevant logs to include? Put them here in side a detail tag: It looks like an error from helm chart on how to handle git username / password because exactly the same git link works on the community airflow chart. We are trying to switch it to the official apache airflow and encountered this error.

[jedcunningham]

Secrets created with data should be base64 encoded, and stringData should be “plain”. @potiuk is probably right, your base64 encoded value is probably wrong (a common one is getting an extra newline with echo "foo" | base64 instead of echo -n "foo" | base64).

[kaxil]

https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-a-private-github-repo-using-git-sync-sidecar explains it nicely (although with SSH key)