Travis failure in master: EC2 instance failed to create

Not clear if this was one-off or likely to re-occur.

https://travis-ci.com/pulumi/pulumi-cloud/builds/58711465

It looks like one of the two VMs that was created never responded that it was ready (presumably never fired the cfn-signal in it’s userdata script?).

The VM that never responded doesn’t show anything too unusual happening - it was running for the full 15 minute period:

https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Instances:sort=instanceState

Unfortunately we don’t have logging for the EC2 VM boot process so it’s hard to see what might have gone wrong on the VM.

[mmdriley]

Here’s one way we can get logs from these instances, if we catch this within an hour of the failure:

1. Go to the list of EC2 instances in the pulumi-testing account in us-east-2 (Ohio). Look for terminated instances. https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=us-east-2#Instances:instanceState=terminated;sort=instanceId

2. Right-click each terminated instance and select “Get System Log” under “Instance Settings”.

The CloudFormation event log will show us the instance that successfully came up, e.g.: so we can look at the other one.

If there are a lot of instances, we can filter them to the appropriate autoscaling group. Looking at the CloudFormation stack will give you the physical ID of the autoscaling group. From there you can filter the EC2 instances that have a matching aws:autoscaling:groupName tag.

[mmdriley]

I tore down the termination-watcher, but here’s the code for posterity:

import * as pulumi from "pulumi";
import * as aws from "@pulumi/aws";

import * as awssdk from "aws-sdk";

const terminatedInstanceEventFilter = {
    "source": [ "aws.ec2" ],
    "detail-type": [ "EC2 Instance State-change Notification" ],
    "detail": {
        "state": [ "terminated" ],
    },
};

function onTerminate(ev: any, ctx: aws.serverless.Context, cb: (error: any, result: any) => void) {
    let ec2 = new awssdk.EC2();
    let instanceId = ev["detail"]["instance-id"];

    ec2.getConsoleOutput({ InstanceId: instanceId }, (err, data) => {
        if (err) {
            console.error(err);
            cb(null, null);
            return;
        }

        let output = Buffer.from(data.Output!, 'base64');

        console.log(data.InstanceId);
        console.log(output.toString());
        cb(null, null);
    });
}

let handlerFunction = new aws.serverless.Function("handlerFunction", {
    policies: [
        aws.iam.AWSLambdaFullAccess,

        // EC2ReadOnlyAccess can't use ec2:GetConsoleOutput
        aws.iam.AmazonEC2FullAccess,
    ],
}, onTerminate);

let eventRule = new aws.cloudwatch.EventRule("eventRule", {
    eventPattern: JSON.stringify(terminatedInstanceEventFilter),
});

let eventTarget = new aws.cloudwatch.EventTarget("eventTarget", {
    rule: eventRule.name,
    arn: handlerFunction.lambda.arn,
});

let callPermission = new aws.lambda.Permission("callPermission", {
    action: "lambda:InvokeFunction",
    function: handlerFunction.lambda,
    principal: "events.amazonaws.com",
    sourceArn: eventRule.arn,
});