CI to push spec openapi document into OL website branch

Problem

The current release.sh script defined in https://github.com/OpenLineage/OpenLineage/blob/main/spec/release.sh does the following:

# check if there are any changes in spec in the latest commit
if git diff --name-only --exit-code $PREV_SPEC_COMMIT HEAD 'spec/*.json' >> /dev/null; then
  echo "no changes in spec detected, skipping publishing spec"
  exit 0
fi

echo "Copying spec files from commit $PREV_SPEC_COMMIT"

# Mark last commit on which we finished copying spec
echo "$CIRCLE_SHA1" > "$WEBSITE_COMMIT_FILE"

# Copy changed spec fils to target location
git diff --name-only HEAD~1 HEAD 'spec/*.json' | while read LINE; do
  # extract target file name from $id field in spec files
  URL=$(cat $LINE | jq -r '.["$id"]')

  # extract target location in website repo
  LOC="${WEBSITE_DIR}/${URL#*//*/}"
  LOC_DIR="${LOC%/*}"

  # create dir if necessary, and copy files
  mkdir -p $LOC_DIR
  cp $LINE $LOC
done

So, basically it only adds whatever is ‘changed’ in the spec file, and then puts it into the website repo (so eventually it is available via website). However, it does NOT do the following:

• Generate openapi.html (spec documentation)
• pushes the updated openapi.html into the appropriate location in website.

Solution

• Add the code to generate openapi.html (using redoc-cli)
• pushes the updated openapi.html to the appropriate website location
• Make sure all the files version are correct, as well as their json spec urls.

How to install and run redoc-cli

• https://redocly.com/docs/redoc/deployment/cli/ has comprehensive instructions on installing redoc-cli Running it is easy. For example, if you want to generate the full openapi.html doc, simply run the below command:

redoc-cli build --output ~/temp/openapi.html ./openlineage.yml

[rossturk]

The first half of this has now been done with #1295 #1296 #1298 #1299 and #1300.

The second half is being worked on in the website repo: https://github.com/OpenLineage/website/pull/70 - once that is merged successfully, we can close this issue.

[mobuchowski]

I think this should be done inside website CI, not in this repo CI. The parts that interact between different repos should be as small as possible, if we need to do a lot of cross-repo communication it might be better to just put the website into this monorepo.