Platform dependent failure of test_compare_to_ELKI

For instance on https://travis-ci.com/github/scikit-learn/scikit-learn/jobs/468229055:

_____________________________ test_compare_to_ELKI _____________________________
[gw2] linux -- Python 3.9.1 /tmp/tmp.I1bENuyzQA/venv/bin/python

    def test_compare_to_ELKI():
        # Expected values, computed with (future) ELKI 0.7.5 using:
        # java -jar elki.jar cli -dbc.in csv -dbc.filter FixedDBIDsFilter
        #   -algorithm clustering.optics.OPTICSHeap -optics.minpts 5
        # where the FixedDBIDsFilter gives 0-indexed ids.
        r1 = [np.inf, 1.0574896366427478, 0.7587934993548423, 0.7290174038973836,
              0.7290174038973836, 0.7290174038973836, 0.6861627576116127,
              0.7587934993548423, 0.9280118450166668, 1.1748022534146194,
              3.3355455741292257, 0.49618389254482587, 0.2552805046961355,
              0.2552805046961355, 0.24944622248445714, 0.24944622248445714,
              0.24944622248445714, 0.2552805046961355, 0.2552805046961355,
              0.3086779122185853, 4.163024452756142, 1.623152630340929,
              0.45315840475822655, 0.25468325192031926, 0.2254004358159971,
              0.18765711877083036, 0.1821471333893275, 0.1821471333893275,
              0.18765711877083036, 0.18765711877083036, 0.2240202988740153,
              1.154337614548715, 1.342604473837069, 1.323308536402633,
              0.8607514948648837, 0.27219111215810565, 0.13260875220533205,
              0.13260875220533205, 0.09890587675958984, 0.09890587675958984,
              0.13548790801634494, 0.1575483940837384, 0.17515137170530226,
              0.17575920159442388, 0.27219111215810565, 0.6101447895405373,
              1.3189208094864302, 1.323308536402633, 2.2509184159764577,
              2.4517810628594527, 3.675977064404973, 3.8264795626020365,
              2.9130735341510614, 2.9130735341510614, 2.9130735341510614,
              2.9130735341510614, 2.8459300127258036, 2.8459300127258036,
              2.8459300127258036, 3.0321982337972537]
        o1 = [0, 3, 6, 4, 7, 8, 2, 9, 5, 1, 31, 30, 32, 34, 33, 38, 39, 35, 37, 36,
              44, 21, 23, 24, 22, 25, 27, 29, 26, 28, 20, 40, 45, 46, 10, 15, 11,
              13, 17, 19, 18, 12, 16, 14, 47, 49, 43, 48, 42, 41, 53, 57, 51, 52,
              56, 59, 54, 55, 58, 50]
        p1 = [-1, 0, 3, 6, 6, 6, 8, 3, 7, 5, 1, 31, 30, 30, 34, 34, 34, 32, 32, 37,
              36, 44, 21, 23, 24, 22, 25, 25, 22, 22, 22, 21, 40, 45, 46, 10, 15,
              15, 13, 13, 15, 11, 19, 15, 10, 47, 12, 45, 14, 43, 42, 53, 57, 57,
              57, 57, 59, 59, 59, 58]
    
        # Tests against known extraction array
        # Does NOT work with metric='euclidean', because sklearn euclidean has
        # worse numeric precision. 'minkowski' is slower but more accurate.
        clust1 = OPTICS(min_samples=5).fit(X)
    
>       assert_array_equal(clust1.ordering_, np.array(o1))
E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatched elements: 2 / 60 (3.33%)
E       Max absolute difference: 2
E       Max relative difference: 0.13333333
E        x: array([ 0,  3,  6,  4,  7,  8,  2,  9,  5,  1, 31, 30, 32, 34, 33, 38, 39,
E              35, 37, 36, 44, 21, 23, 24, 22, 25, 27, 29, 26, 28, 20, 40, 45, 46,
E              10, 17, 11, 13, 15, 19, 18, 12, 16, 14, 47, 49, 43, 48, 42, 41, 53,
E              57, 51, 52, 56, 59, 54, 55, 58, 50])
E        y: array([ 0,  3,  6,  4,  7,  8,  2,  9,  5,  1, 31, 30, 32, 34, 33, 38, 39,
E              35, 37, 36, 44, 21, 23, 24, 22, 25, 27, 29, 26, 28, 20, 40, 45, 46,
E              10, 15, 11, 13, 17, 19, 18, 12, 16, 14, 47, 49, 43, 48, 42, 41, 53,
E              57, 51, 52, 56, 59, 54, 55, 58, 50])

This test used to pass, so maybe this is caused by a change of version in a dependency (e.g. numpy or scipy and their embedded BLAS).

Note that I cannot reproduce on macos/arm64 with the latest numpy 1.19.4 and scipy 1.6.0 versions (from conda-forge).

[drew-parsons]

This test is passing for arm64 in Debian builds, see https://buildd.debian.org/status/fetch.php?pkg=scikit-learn&arch=arm64&ver=0.23.2-4&stamp=1610905951&raw=0

But the test failure is found on i386, see https://buildd.debian.org/status/fetch.php?pkg=scikit-learn&arch=i386&ver=0.23.2-4&stamp=1611004993&raw=0

[alfaro96]

My investigation was pretty straightforward. For Python 3.6, scipy==1.5.4 version is used and the test is passing. For Python 3.7 and newer, scipy==1.6.0 is used and the test is failing.

Still further investigation required, but it seems that it is a numerical rounding error. The difference is a swap in the ordering (15 and 17).