Integration testing for the sentry-native new unwinder

We’ve updated sentry-native with a new stack unwinder. Initial testing was successful but we need more real device testing.

This is a good candidate for a proper e2e test on a device farm.

We need to test the latest version (with the new stackwalker): https://github.com/getsentry/sentry-java/releases/tag/4.4.0-alpha.1

vs and older version: https://github.com/getsentry/sentry-java/releases/tag/4.3.0

The idea is to load the sample project and hit the native crash button. Then copy the generated envelope and send it somewhere for analysis.

This somewhere could be a Sentry event. The issue could be the Android SDK version and the device name/version could be a tag. The envelope can be an attachments. This way we can compare the attachments of both versions. Or perhaps more easily is to push this to a GCP bucket so we can script out a side-by-side comparisson between the two version.

Make sure to keep around the debug symbols in case we need to ingest these events (or if we can restart the app the event will actually be sent already and properly symbolicated). This way best to use a new sentry project in sentry-sdks so we can look at the data there.

This code doesn’t need to be merged here so hack away.

@Swatinem do we need anything specific in terms of stacktrace? Do we need some number of frames like 10 or whatever we have on the sample app today is fine?

The current sample native crash actually has a LOT of frames: https://sentry.io/organizations/sentry-sdks/issues/2260073983/?project=5428559&query=is%3Aunresolved

[marandaneto]

I think we can have a 2nd Android example that mimics the behavior of this test and we compile/run manually on AppCenter when necessary, it’d be the least flaky I guess.

[denrase]

Short update:

• I have created two UI tests. The first one inits the sentry sdk and triggers the native crash. The second one checks for the file in cache/sentry/outbox.

• The tests are executed with espresso.

• The problem here, is that the first test takes the whole test process down.

• Setup Microsoft AppCenter and uploaded signed APKs, same result there, obviously.

• A promising solution was to use Android Test Orchestrator, so individual tests are run in isolation and crashes don’t take down the whole process.

• This works, the first test always fails as expected and the second one passes when the file is there and we are in FlightMode.

• The problem is, I only managed to get this working locally, the version with Orchestrator does not work on AppCenter. Neither release or debug builds of the APKs.

• Next step is to try another testing framework (Appium) and see if we can get around the process crashing limitation there.