Git LFS sometimes fails on PR jobs only

Your checklist for this issue

• Jenkins version

• Plugin version

• Bitbucket cloud

• Bitbucket server and version

Description

• Jenkins version 2.190.1
• Plugin version 2.5.0
• Bitbucket server version 6.1.2
• Git LFS version git-lfs/2.8.0 (GitHub; linux amd64; go 1.12.6)
• Git version 2.11.0

Sometimes when an LFS file change is merged to master, all PR CI jobs will start failing on the merge step with:

Started by user Jupiter CI
Restarted from build #11, stage Build Terminal
Obtained Jenkinsfile from 5033c6390fdbbfa53f57975da66b0d55282353a3+bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] milestone
Trying to pass milestone 11
[Pipeline] milestone
Trying to pass milestone 12
[Pipeline] stage
[Pipeline] { (Build Terminal)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/jupiter_PR-99
[Pipeline] {
[Pipeline] checkout
using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
using credential e9b7d03e-0956-4e3c-b405-7c95be8c34de
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from 2 remote Git repositories
 > git config remote.origin.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Fetching without tags
Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/pull-requests/99/from:refs/remotes/origin/PR-99
 > git config remote.upstream.url http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Fetching without tags
Fetching upstream changes from http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress -- http://gtswtbitsyslnxvm1:7990/scm/JUP/jupiter.git +refs/heads/master:refs/remotes/upstream/master
Merging remotes/upstream/master commit bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 into PR head commit 5033c6390fdbbfa53f57975da66b0d55282353a3
Enabling Git LFS pull
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials 
 > git lfs pull origin
 > git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667 # timeout=10
ERROR: Timeout after 10 minutes
Enabling Git LFS pull
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5033c6390fdbbfa53f57975da66b0d55282353a3
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials 
 > git lfs pull origin
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
[Bitbucket] Notifying pull request build result
[Bitbucket] Build result notified
hudson.plugins.git.GitException: Command "git merge bd68a4fb34dc8b2daf36f07c0d88bd8a740bc667" returned status code 128:
stdout: 
stderr: Downloading apps/dashboard.tar.gz (3.8 MB)
Terminated
error: external filter 'git-lfs filter-process' failed
fatal: apps/dashboard.tar.gz: smudge filter lfs failed

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2140)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2136)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1741)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$3.execute(CliGitAPIImpl.java:841)
	at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:122)
	at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1100)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1193)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

This only seems to happen on PR jobs where it tries to merge master first before building. Normal branch jobs don’t encounter this problem.

The problem is, I am having a hard time reproducing this behavior outside of Jenkins. It seems like Jenkins is lying about which git commands it is actually running, because if I attach to the docker container running Jenkins, cd to the PR workspace folder, and then run the git commands one by one, they all run without issue until I get to the git merge command at which point it complains about user and email not being configured. If I configure a user/email, then it complains about there not being an editor. But this works most of the time, so I’m wondering if it is secretly running git merge --no-edit or something. Oh, and if I do git merge --no-edit (by the way) it runs fine without error.

I’m not even sure whose fault this issue is and why it only seems to happen in Jenkins. Could be a Git LFS bug, Jenkins, I’m not sure. I’m coming here first because it seems like the plugin is responsible for issuing all these git commands.

[RPGillespie6]

In case anyone else is running into this issue, it’s due to a bug in git-client:

https://issues.jenkins-ci.org/browse/JENKINS-45228

Basically, default bitbucket-branch-source-plugin PR jobs will encounter this issue anytime target branch has new LFS objects. As of Janurary 2020, the fix has not been released yet.

[RPGillespie6]

My mistake, I will close this issue.

And yes, I am using Bitbucket’s LFS. 10 minutes seems like a huge timeout though…