question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Discrepancy in counterfactual indexing for CLUE generator

See original GitHub issue

Hello!

When generating counterfactuals using CLUE counterfactual generator, the resulting counterfactuals in the dataframe are ordered using a RangeIndex instead of the original indexing found in the factuals. This is a problem when the factuals’ indices are not ordered as a range index, e.x. by using .sample(n). This can be seen by running this code:

data_name = "compas"
model = MLModelCatalog(dataset, "ann", backend="pytorch")
model.train(...)

hyperparams = {...}
cl = Clue(dataset, model, hyperparams)

wa = Wachter(model, {...})

factuals = predict_negative_instances(model, dataset._df).sample(10)

cl_counterfactuals = cl.get_counterfactuals(factuals)
wa_counterfactuals = wa.get_counterfactuals(factuals)

display(factuals.index)
display(cl_counterfactuals.index)
display(wa_counterfactuals.index)

This yields:

Int64Index([4886, 4389, 2317, 797, 3154, 4685, 956, 3014, 99, 510], dtype='int64')
RangeIndex(start=0, stop=10, step=1)
Int64Index([4886, 4389, 2317, 797, 3154, 4685, 956, 3014, 99, 510], dtype='int64')

Now, if we try to benchmark the CLUE counterfactual generator:

benchmark = Benchmark(model, cl, factuals)
benchmark.run_benchmark()

we get:

ValueError: Can only compare identically-labeled DataFrame objects

Stemming from the constraint_violation check in carla\evaluation\violations.py

The counterfactuals get reordered in counterfactuals.check_counterfactuals for CLUE, FACE, GrowingSpheres and REVISE since they all pass in a list of counterfactuals as opposed to a pandas dataframe, so at least those methods can have problems with that.

One way of fixing this issue would be to pass an index attribute to check_counterfactuals and add a index=indices on line 32 to set the proper indices. Other would be to simply do cfs_df.index = factuals.index after the function is called to reorder the indices.

Since the counterfactuals now get new indices, and both dataframes are of the same size, restoring the ones from factuals shouldn’t cause any further problems I can spot and will only improve the consistency across the recourse methods.

If any of the proposed fixes are acceptable, or there are other ways to fix the issue, I would be happy to perform them and be assigned to this issue. 😃

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:12 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
drobiucommented, May 24, 2022

Since my commit doesn’t change these files, I skipped the pre-commit install step. The error will still appear if one tries to push with pre-commit. It also happened before I made changes so I guess these files have to be fixed.

1reaction
JohanvandenHeuvelcommented, May 11, 2022

That would be great! I tried replicating the problem, but for me it works. So it’s a bit difficult for me to test if something fixes it. I’m mainly wondering why it doesn’t work for you but does for me.

Read more comments on GitHub >

github_iconTop Results From Across the Web

arXiv:2104.00929v1 [cs.CL] 2 Apr 2021 - GitHub Pages
In the customize stage, the premise, counterfactual condition and the skeleton are provided as contexts to a generator to produce proper words to...
Read more >
CUUS406_ch11.qxd
counterfactual subscripts, or do(*) can safely be proclaimed as inadequately ... idea that confounding stands for “bias,” namely, a “difference between two.
Read more >
Anterior cingulate cortex causally supports flexible learning ...
Motivational demand was indexed by varying gains and losses during learning, while cognitive demand was varied by increasing the uncertainty ...
Read more >
Conviction by Contradiction
A contradiction in the defense's alibi doesn't necessarily prove them guilty if the prosecution otherwise failed to make its case; but for the...
Read more >
Counterfactual Mugging tag - LessWrong 2.0 viewer
Counterfactual mugging is a thought experiment for testing and differentiating decision theories, stated as follows: Omega, a perfect predictor, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found