Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

spark: fix flaky tests

See original GitHub issue

Here fails io.openlineage.spark.agent.SparkContainerIntegrationTest [10] spark_v2_drop.py, pysparkV2DropTableStartEvent.json, pysparkV2DropTableCompleteEvent.json, true

https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/4630/workflows/8fe28cb2-7737-4924-bda5-91d7c7471a47/jobs/56363

and here testFilteringDeltaEvents https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/4634/workflows/3acb2e0a-ee15-45b2-9d5e-b38c00228d6c/jobs/56458

Issue Analytics

State:
Created a year ago
Comments:5 (4 by maintainers)

Top GitHub Comments

1reaction

mobuchowskicommented, Oct 12, 2022

It still is a very bad experience. We should strive to remove this problem, maybe by rewriting tests or making them work with larger datasets if that’s what will allow us to emit events in time.

https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/4816/workflows/962d0d0f-e8ce-4aba-947e-e46dc9db59e3/jobs/60092

0reactions

pawel-big-lebowskicommented, Oct 27, 2022

@mobuchowski indeed Maciej, they didn’t merge the fix for Spark 3.3.1.

To rephrase the problem: Before we are able to process SparkListenerSQLExecutionStart, the SparkListenerSQLExecutionEnd is published and query execution is removed from executionIdToQueryExecution which we do require for processing start event.

Are somehow able to check within processing start event if there is already an end event and get queryExecution from it?

Top Results From Across the Web

[#SPARK-28358] Fix Flaky Tests - ASF JIRA

I took a look related issue to this. Seems like it's very likely the issue from the old PyPy. It will be fixed...

[SPARK-40096][CORE][TESTS][FOLLOW-UP] Fix flaky test ...

This PR is a followup of #37533 that fix the flaky test case. Why are the changes needed? The test case is flaky,...

How to Deal with Flaky Tests - The New Stack

1. Visualizing Test Runs · 2. Quarantining Flaky Tests · 3. Cleaning up State · 4. Looking for Timeouts · 5. Using Test...

[jira] [Updated] (SPARK-33273) Fix Flaky Test ... - The Mail Archive

[jira] [Updated] (SPARK-33273) Fix Flaky Test: ThriftServerQueryTestSuite. subquery_scalar_subquery_scalar_subquery_select_sql · Dongjoon Hyun (Jira) Thu, ...

Spark 2.0.1-1703 Release Notes

In addition, Spark 2.0.1-1703 includes backports of all the fixes contained in ... be401c8, 2017/02/06, [SPARK-17624][SQL][STREAMING][TEST] Fixed flaky ...