make shell scripts more strict

shell scripts by default skip over errors and continue. it’s best practices to use set to harden this using the following flags:

• -e: Exit immediately if a command exits with a non-zero status.
• -u: Treat unset variables as an error when substituting.
• possibly also -o pipefail: If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully. By default, pipelines only return a failure if the last command errors. When used in combination with set -e, pipefail will make a script exit if any command in a pipeline errors.

so far i’ve used set -eu, but i’ve seen that OpenSearch has also used set -e -o pipefail (see example below).

i think somebody with expert knowledge in this area (i wouldn’t consider myself an expert here) should define the standard to be used in all scripts (or rules for when to use what if there are differentiating needs).

some scripts already have this set, e.g.: https://github.com/opensearch-project/OpenSearch/blob/c8ac0373895fe6b884792569029bdc9d465850b5/distribution/src/bin/opensearch-env#L3

others don’t, e.g. https://github.com/opensearch-project/OpenSearch/blob/main/distribution/packages/src/deb/init.d/opensearch

this is relevant not just for this repository but for all repositories (i’m not aware of a meta-repository where to open such tickets, feel free to move this ticket if such a place exists).

standardising and applying this globally will ensure that errors are not (silently) hidden.

[CEHENKLE]

Hey @rursprung ! I agree, standardizing this would keep us from falling into pits, which is always good.

We have a couple of global-ish repos like clients/plugins/meta. @dblock do you have a thought on where this should go?

At the very least, we could set up the defaults in core (opensearch-project/opensearch) while we figure out the right way to track it against all repos. @owaiskazi19 can you take a swing at that for core?

[dblock]

I would start doing this work here, then we can expand it into a campaign which likely belongs in project-meta.