Monorepo race condition running for multiple components

[NOTE: Originaled posted here semantic-release/issues/1628]

We are using semantic-release with the plugin semantic-release-monorepo. Our configuration for each component is fairly simple (see below):

The problem comes when running this on CI pipelines for multiple components of the monorepo. If you run this in parallel for component A, B and C, when component A push a tags (and notes) to git, component B and C fails to push due to being outdated.

I’m calling it a “race condition” because it only happens when some commits trigger more than one component at the same time. On these cases, the firsts components (maybe one or two, depending how long each one takes) get published correctly, and all others after that fails. But it fails on pushing git notes (not release notes) and not on pushing tags.

Is there anything that can be done at the plugin level to fix this race condition?

Config:

"dependencies": {  
  "semantic-release": "17.1.1",  
  "semantic-release-monorepo": "7.0.2"  
},  
"release": {  
  "extends": "semantic-release-monorepo",  
  "plugins": [  
    "@semantic-release/commit-analyzer",  
    "@semantic-release/release-notes-generator"  
  ]  
}

Logs:

[semantic-release] An error occurred while running semantic-release: { Error: Command failed with exit code 1: git push https://[secure]@github.com/MY_USER/MY_REPO.git refs/notes/semantic-release
To https://github.com/MY_USER/MY_REPO.git
 ! [rejected]          refs/notes/semantic-release -> refs/notes/semantic-release (fetch first)
error: failed to push some refs to 'https://[secure]@github.com/MY_USER/MY_REPO.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
    at makeError (/builds/__MY_REPO__/node_modules/execa/lib/error.js:59:11)
    at handlePromise (/builds/__MY_REPO__/node_modules/execa/index.js:114:26)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  shortMessage:
   'Command failed with exit code 1: git push https://[secure]@github.com/MY_USER/MY_REPO.git refs/notes/semantic-release',
  command:
   'git push https://[secure]@github.com/MY_USER/MY_REPO.git refs/notes/semantic-release',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: '',
  stderr:
   'To https://github.com/MY_USER/MY_REPO.git\n ! [rejected]          refs/notes/semantic-release -> refs/notes/semantic-release (fetch first)\nerror: failed to push some refs to \'https://[secure]@github.com/MY_USER/MY_REPO.git\'\nhint: Updates were rejected because the remote contains work that you do\nhint: not have locally. This is usually caused by another repository pushing\nhint: to the same ref. You may want to first integrate the remote changes\nhint: (e.g., \'git pull ...\') before pushing again.\nhint: See the \'Note about fast-forwards\' in \'git push --help\' for details.',
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false }

[UPDATE: We are not using lerna and it doesn’t apply to our case because we are not publishing js modules]

[ChristianUlbrich]

@guumaster There are two problems currently here. One problem is, that semantic-release is not doing an atomic git operation. Each semantic-release boils down to:

• pushing a tag
• pushing git notes

This could be changed in semantic-release; they introduced using git notes not so long ago, before that they were using git tags solely. So while this could potentially be alleviated, the other problem is bigger:

Running semantic-release takes time, if you happen to have changes to the same component, it will create a race condition. This is a conceptual problem and you will stumble upon this problem pretty fast. If semantic-release were to use an atomic git operation, you could come up with a recover strategy (pull, merge, run semantic-release again).

However, after having spent quite some days the last weeks with implementing semantic-release for a rather small monorepo, it is not worth the effort. If you are optimizing you pipeline you are better off with (as @pmowrer suggested) a serialized approach; each pipeline job, that does the actual release must run serialized. In typical CI systems you can optimize this pretty easily by using pipeline artifacts and some locking feature. If you happen to use Azure DevOps, you can trivially come up with a serialized pipeline concept via either exclusive locks on environments or dedicated batched pipelines.

[ChristianUlbrich]

FWIW the actual problem lies within semantic-release. Using it in a monorepo simply makes the race condition more likely to happen.