The last few log lines are missing rarely in 3.1

Description

When running a lambda, rarely some ending log lines are missing from the CloudWatch output. I did not see this behavior in 2.1, and I am seeing this behavior in 3.1.

Reproduction Steps

Create a lambda with the attached code. Assign it a role that allows writing to CloudWatch and launching lambdas. Set up a CloudWatch event to invoke the lambda every 20 minutes with an empty input. Give it a 900 second timeout. Export the logs from CloudWatch and notice rarely the line ‘Finished ‘ReproMissingLogs’ with RequestId’ will be missing from the output.

To build the lambda you need to add Newtonsoft and AWSSDK.Lambda via nuget.

Function.zip

Logs

Here is an example of a failure case:

2020-10-21T21:57:08.463Z START RequestId: d9512f18-cdb5-479d-ab43-e8addfd2f272 Version: $LATEST

2020-10-21T21:57:08.468Z Starting 'ReproMissingLogs' with RequestId 'd9512f18-cdb5-479d-ab43-e8addfd2f272'. Input: {"BuildStep":1,"WorkIndex":2}.
2020-10-21T21:57:09.531Z Starting work 2...
2020-10-21T21:57:09.531Z Work will take 39699 milliseconds
2020-10-21T21:57:49.246Z END RequestId: d9512f18-cdb5-479d-ab43-e8addfd2f272

2020-10-21T21:57:49.246Z REPORT RequestId: d9512f18-cdb5-479d-ab43-e8addfd2f272	Duration: 40786.58 ms	Billed Duration: 40800 ms	Memory Size: 256 MB	Max Memory Used: 70 MB	Init Duration: 261.47 ms	

Here is an example of a success case:

2020-10-21T21:37:09.886Z START RequestId: 64963e8f-1cc0-4458-aaa9-36569dfb5031 Version: $LATEST

2020-10-21T21:37:09.894Z Starting 'ReproMissingLogs' with RequestId '64963e8f-1cc0-4458-aaa9-36569dfb5031'. Input: {"BuildStep":1,"WorkIndex":16}.
2020-10-21T21:37:10.919Z Starting work 16...
2020-10-21T21:37:10.919Z Work will take 8827 milliseconds
2020-10-21T21:37:19.748Z Finished work
2020-10-21T21:37:19.750Z Finished 'ReproMissingLogs' with RequestId '64963e8f-1cc0-4458-aaa9-36569dfb5031' with '890.136' seconds left.
2020-10-21T21:37:19.750Z END RequestId: 64963e8f-1cc0-4458-aaa9-36569dfb5031

2020-10-21T21:37:19.750Z REPORT RequestId: 64963e8f-1cc0-4458-aaa9-36569dfb5031	Duration: 9864.03 ms	Billed Duration: 9900 ms	Memory Size: 256 MB	Max Memory Used: 69 MB	Init Duration: 161.26 ms

Environment

• Build Version: Amazon.Lambda.Core 1.1.0, Amazon.Lambda.Serialization.SystemTextJson 2.0.0, AWSSDK.Lambda 3.3.110.18
• OS Info: Built and Deployed on Windows 10
• Build Environment: Visual Studio
• Targeted .NET Platform: .NET Core 3.1

Resolution

• 👋 I can/would-like-to implement a fix for this problem myself

Find and fix what is causing these missing logs.

I reported this as issue 717 but haven’t had a chance to write a reproduction until now.

I did notice I’m a bit behind on Nuget packages. I will update and see if it is still reproducing, but unless something has been fixed I suspect this is still a problem.

---

This is a 🐛 bug-report

[hunanniu]

We believe the issue is on Lambda service side rather than in SDK. So we will close this one. If Lambda thinks otherwise after they investigate, you can reopen.

[dan-glenn-ivendi]

@hunanniu Could you link to the new issue (Lambda) please?