Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

IdempotencyTests is flaky on Windows

See original GitHub issue

Starting ~~with the merge of https://github.com/lampepfl/dotty/pull/11210~~ (EDIT: actually long before that, see comments below), it looks like test_windows_full has been failing IdempotencyTests from time to time, from https://github.com/lampepfl/dotty/runs/2178741539:

Test 'CheckPosIdempotency from idempotency/check' failed with output:           
Idempotency test failed between out\idempotency\posIdempotency1\pos\i6507b\Test.tasty and out\idempotency\posIdempotency2\pos\i6507b\Test.tasty
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at dotty.tools.vulpix.ChildJVMMain.runMain(ChildJVMMain.java:40)
	at dotty.tools.vulpix.ChildJVMMain.main(ChildJVMMain.java:47)
Caused by: java.lang.AssertionError: assertion failed: Failed 1 idempotency checks (out of 5116)
	at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
	at IdempotencyCheck$.checkIdempotency(IdempotencyCheck.scala:70)
	at Test$.main(CheckPosIdempotency.scala:4)
	at Test.main(CheckPosIdempotency.scala)
	... 6 more

@bishabosha could you investigate this (I’ve forwarded you the access information for the Windows machine if you want to investigate there)? /cc @liufengyun

Issue Analytics

State:
Created 2 years ago
Comments:13 (5 by maintainers)

Top GitHub Comments

2reactions

griggtcommented, Mar 25, 2021

The discrepancy seems to be in the positions section:

--- out/idempotency/posIdempotency1/debug-idemp/i6507b/tasty.txt	2021-03-25 12:16:34.354135344 -0700
+++ out/idempotency/posIdempotency2/debug-idemp/i6507b/tasty.txt	2021-03-25 12:16:53.166135344 -0700
@@ -264,7 +264,7 @@
    436:         STRINGconst 56 [tests/debug-idemp/i6507b.scala]
    438:
 
- 291 position bytes:
+ 288 position bytes:
    lines: 11
    line sizes: 26, 0, 13, 79, 36, 56, 3, 0, 28, 1, 0
    positions:
@@ -337,7 +337,7 @@
        339: 197 .. 215
        341: 193 .. 193
        347: 183 .. 196
-       350: 200 .. 215
+       350: 183 .. 183
        353: 183 .. 196
        357: 200 .. 215
        364: 197 .. 215

   284:       INLINED(130)
   287:         BLOCK(124)
   289:           TYPED(112)
   291:             APPLY(107)
   293:               TYPEAPPLY(102)
   295:                 SELECTin(95) 41 [*:[Signed Signature(List(2, java.lang.Object),scala.Product) @*:]]
   298:                   INLINED(89)
   300:                     BLOCK(83)
   302:                       TYPED(71)
   304:                         APPLY(66)
   306:                           TYPEAPPLY(61)
   308:                             SELECTin(54) 41 [*:[Signed Signature(List(2, java.lang.Object),scala.Product) @*:]]
   311:                               INLINED(48)
   313:                                 BLOCK(42)
   315:                                   TYPED(30)
   317:                                     APPLY(25)
   319:                                       TYPEAPPLY(20)
   321:                                         SELECTin(13) 41 [*:[Signed Signature(List(2, java.lang.Object),scala.Product) @*:]]
   324:                                           INLINED(7)
   326:                                             SHAREDterm 151
   329:                                             IDENTtpt 6 [Test[ModuleClass]]
   331:                                               SHAREDtype 18
   333:                                           SHAREDtype 241
   336:                                         INTconst 98
   339:                                         SHAREDtype 112
   341:                                       TERMREFdirect 347
   344:                                     SHAREDterm 253
   347:                                   VALDEF(8) 44 [x[Unique $ 1]]
   350:                                     SHAREDtype 336
   353:                                     INTconst 98
   356:                                     SYNTHETIC

The output files from this test run: i11885-output.zip

1reaction

liufengyuncommented, Apr 16, 2021

This flaky test just failed again. I thought it was disabled?

https://github.com/lampepfl/dotty/runs/2360396782?check_suite_focus=true#step:4:594

It looks like the test filter does not work well due to the difference in path separators on Windows (#11950). I’ll have a look.

Top Results From Across the Web

Preempting Flaky Tests via Non-Idempotent-Outcome Tests

To proactively detect, i.e., preempt, flaky tests, we propose to detect non-idempotent-outcome (NIO) tests, a novel category related to flaky tests. In ...

How to Fix Flaky Tests - Semaphore CI

Randomly failing tests are the hardest to debug. Here's a framework you can use to fix them and keep your test suite healthy....

Flaky tests - GitLab Docs

It's a test that sometimes fails, but if you retry it enough times, it passes, eventually. What are the potential cause for a...

A Survey of Flaky Tests - ACM Digital Library

Tests that fail inconsistently, without changes to the code under test, are described as flaky. Flaky tests do not give a clear indication ......

Manage flaky tests - Azure Pipelines - Microsoft Learn

A flaky test is a test that provides different outcomes, such as pass or fail, even when there are no changes in the...