Python JSONDecodeError on CDK deploy

🐛 Bug Report

What is the problem?

When attempting to do a cdk deploy from a Stack created in python, i am getting a JSONDecodeError (shown a little further down). This is occurring while running on a self-hosted Azure DevOps build agent. The exception only occurs when running this build agent in an ECS EC2 hosted container on the same account where we are attempting to deploy, however if we run the build agent container on a local dev machine, it is able to run the cdk deploy without any difficulty. This leads us to believe that the deployment process is having some kind of access issue connecting to a specific AWS API, or that the returned JSON response from one of these APIs is missing or invalid. We have not determined which construct is causing the issue at this point, but were asked by AWS Support to go ahead and submit this as an issue on this github project.

The below exception, based on the verbose logging we are viewing, is occurring right before the deploy process is identifying the stack dependencies and starting deployment.

2019-09-13T17:08:11.6233423Z Traceback (most recent call last):
2019-09-13T17:08:11.6271925Z   File "portal_app.py", line 26, in <module>
2019-09-13T17:08:11.6272102Z     app.synth()
2019-09-13T17:08:11.6294644Z   File "/var/vsts/0e0e1478c340-agent/r1/a/_KES_Portal/DockerBuildArtifact/Infrastructure/kes-portal-cdk/.env/lib/python3.6/site-packages/aws_cdk/core/__init__.py", line 2840, in synth
2019-09-13T17:08:11.6294812Z     return jsii.invoke(self, "synth", [])
2019-09-13T17:08:11.6295230Z   File "/var/vsts/0e0e1478c340-agent/r1/a/_KES_Portal/DockerBuildArtifact/Infrastructure/kes-portal-cdk/.env/lib/python3.6/site-packages/jsii/_kernel/__init__.py", line 104, in wrapped
2019-09-13T17:08:11.6295373Z     return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
2019-09-13T17:08:11.6295769Z   File "/var/vsts/0e0e1478c340-agent/r1/a/_KES_Portal/DockerBuildArtifact/Infrastructure/kes-portal-cdk/.env/lib/python3.6/site-packages/jsii/_kernel/__init__.py", line 267, in invoke
2019-09-13T17:08:11.6295995Z     args=_make_reference_for_native(self, args),
2019-09-13T17:08:11.6296406Z   File "/var/vsts/0e0e1478c340-agent/r1/a/_KES_Portal/DockerBuildArtifact/Infrastructure/kes-portal-cdk/.env/lib/python3.6/site-packages/jsii/_kernel/providers/process.py", line 346, in invoke
2019-09-13T17:08:11.6296538Z     return self._process.send(request, InvokeResponse)
2019-09-13T17:08:11.6296931Z   File "/var/vsts/0e0e1478c340-agent/r1/a/_KES_Portal/DockerBuildArtifact/Infrastructure/kes-portal-cdk/.env/lib/python3.6/site-packages/jsii/_kernel/providers/process.py", line 308, in send
2019-09-13T17:08:11.6297059Z     self._next_message(), _ProcessResponse_R
2019-09-13T17:08:11.6297512Z   File "/var/vsts/0e0e1478c340-agent/r1/a/_KES_Portal/DockerBuildArtifact/Infrastructure/kes-portal-cdk/.env/lib/python3.6/site-packages/jsii/_kernel/providers/process.py", line 259, in _next_message
2019-09-13T17:08:11.6297952Z     return json.loads(self._process.stdout.readline(), object_hook=ohook)
2019-09-13T17:08:11.6298075Z   File "/usr/local/lib/python3.6/json/__init__.py", line 367, in loads
2019-09-13T17:08:11.6298158Z     return cls(**kw).decode(s)
2019-09-13T17:08:11.6298258Z   File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
2019-09-13T17:08:11.6298340Z     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-09-13T17:08:11.6298425Z   File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
2019-09-13T17:08:11.6298533Z     raise JSONDecodeError("Expecting value", s, err.value) from None
2019-09-13T17:08:11.6298620Z json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2019-09-13T17:08:11.6339409Z Subprocess exited with error 1
2019-09-13T17:08:11.7034597Z Error: Subprocess exited with error 1
2019-09-13T17:08:11.7036332Z     at ChildProcess.proc.on.code (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:110:23)
2019-09-13T17:08:11.7036808Z     at ChildProcess.emit (events.js:198:13)
2019-09-13T17:08:11.7037209Z     at ChildProcess.EventEmitter.emit (domain.js:448:20)
2019-09-13T17:08:11.7037597Z     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)

Reproduction Steps

Working on finding a more simple stack that can reproduce this same issue. Currently cannot share the full stack I am working on.

Verbose Log

Working on removing confidential information.

Environment

• CDK CLI Version: CDK toolkit version: 1.8.0 (build 5244f97)
• Module Version: 1.8.0
• OS: Ubuntu 16.04
• Language: Python

[mtngld]

Also seeing this issue on MacOS with node-13.x, when downgraded to node-10.x it indeed works fine.

[imonthercks]

@RomainMuller, I have seen the issue occur again, so I am going to try to get this pared down to a small reproducible stack. May not happen today, but I’ll get it to you as soon as I can.