EmrEtlRunner: switch AMI bootstrap scripts to HTTPS

Hello!

Since today (15-01-2020) we are not able to download the commons-codec from Maven, which means we cannot bootstrap our EMR.

When checking the error logs, both from our application and Amazon, we saw that the bootstrap is failing on downloading that said library because it tries to connect via HTTP:

--2020-01-15 18:12:38-- http://central.maven.org/maven2/commons-codec/commons-codec/1.5/commons-codec-1.5.jar Resolving central.maven.org (central.maven.org)... 151.101.4.209 Connecting to central.maven.org (central.maven.org)|151.101.4.209|:80... connected. HTTP request sent, awaiting response... 501 HTTPS Required 2020-01-15 18:12:38 ERROR 501: HTTPS Required.

With a quick research on the web, we found out that since today (15-01-2020) Maven no longer supports plain HTTP, so we tried to connect via HTTPS. However, that failed too with the message bellow:

--2020-01-15 19:18:23-- https://central.maven.org/maven2/commons-codec/commons-codec/1.5/commons-codec-1.5.jar Resolving central.maven.org (central.maven.org)... 151.101.204.209 Connecting to central.maven.org (central.maven.org)|151.101.204.209|:443... connected. ERROR: no certificate subject alternative name matches requested host name central.maven.org. To connect to central.maven.org insecurely, use –no-check-certificate’. `

Then, we created another script to add as a custom Bootstrap. We managed to add this script, but it failed again, this time pointing out that it cannot remove commons-codec-1.4.jar.

Right now, we are trying to pass 1.5 as a parameter and see if it will work. We will try to update it here as soon as possible - no matter if it fixes or not.

Also, I’d like to point out that we are on sa-east-1.

Project:
Stream Enrich

Version:

Expected behavior:

EMR Bootstraping builds the cluster.

Actual behavior: EMR fails at Bootstrap.

Steps to reproduce:

1. Start a new EMR using snowplow-ami4-bootstrap-0.2.0.sh as the script.

2. Change http:// to https:// on snowplow-ami4-bootstrap-0.2.0.sh. Try to run it as you would.

3. Change https://central to https://repo1 on snowplow-ami4-bootstrap-0.2.0.sh. Try to run it as you would normally.

4. Create a new self hosted bootstrap script and add it to your EMR’s Bootstrap. Make sure the new script has the same changes as those above.

Cheers!

[hizumisen]

@chuwy thanks, our process has finished correctly.

[chuwy]

Hi @rcapp. We hotfixed the AMI4 script (the AMI5 script which is used for releases since ~2018 was updated yesterday) - your pipeline should start properly now.