pdbbind scaffold test set appears to be truncated

deepchem/contrib/atomicconv/acnn/refined/get_acnn_refined.sh

     test = dc.data.DiskDataset("datasets/scaffold_test")                                                            
     print len(test.ids)

This test set only has 708 entries (others have ~740), and they stop exactly at 8pK (there are no high affinity compounds).

It seems unlikely that high affinity compounds (at precisely the 8.00 cutoff) are missing because of scaffold clustering (as opposed to the list getting truncated somewhere). It’s pretty suboptimal for more than a third of the affinity range to be missing from the test set.

[dkoes]

What split makes the most sense depends on what sort of generalization error you are trying to measure. If you are interested in how well you will generalize to new targets, you should split by targets (which is what we do, with a significant difference in sequence identity). If you are interested in how well you generalize to new chemotypes, a scaffold split makes sense (although this is a bit tricky; e.g. compounds with different scaffolds may still have the same “warheads”).

I don’t find the time split or the non-core/core split particularly attractive, but to each their own. They also aren’t particularly amenable to cross-validation or bootstrapping.

[rbharath]

Closing this old discussion. Feel free to re-open if there are new points to consider.