CSVDataSource ignoring first column

The first column in my CSV file is always ignored i.e. doesn’t get loaded into the feature map.

Example: if Role is the first column in the file, it doesn’t appear when the feature map is logged. If I insert a “dummy” column, then role appears.

        Map<String, FieldProcessor> fieldProcessorMap = new HashMap<>();

        fieldProcessorMap.put("YearsExperience", new DoubleFieldProcessor("YearsExperience") );
        fieldProcessorMap.put("Location", new IdentityProcessor("Location") );
        fieldProcessorMap.put("Role", new IdentityProcessor("Role"));

        var responseProcessor = new FieldResponseProcessor<>("CurrentSalary", "0", new LabelFactory());

        var id = new IntExtractor("ID");

        var rowProcessor = new RowProcessor<>(Collections.singletonList(id), null, responseProcessor, fieldProcessorMap, Collections.emptySet());

        var csvDataSource = new CSVDataSource<>(Paths.get("data/dummy/sample1.csv"), rowProcessor, true);

        var dataSplitter = new TrainTestSplitter<>(csvDataSource, 0.7, 1L);

        var trainingDataset = new MutableDataset<>(dataSplitter .getTrain());
        var testingDataset = new MutableDataset<>(dataSplitter .getTest());

        for (var i : trainingDataset.getFeatureMap())
            log.info(i);

[leccelecce]

Thanks Craig for a very quick diagnosis! Excel 365 on Windows also has this behaviour it seems. I had opened the file in Notepad++ to check formatting and it looked fine but hadn’t checked the encoding.

I’ve experimented and unhelpfully the “default” (i.e. top of the file type list) CSV UTF-8 option in Excel adds a BOM (even though in theory UTF-8 shouldn’t have one…thanks Microsoft?). 10 items lower down the list is CSV (Comma delimited) which doesn’t add the BOM and so works fine with Tribuo, so I’ll use that going forward.

I’m surprised I’ve never encountered this issue before as I’ve been using CSV files with Java for a while. Interestingly, looking at Jackson, they gave in and implemented auto-detection of a BOM:

https://github.com/FasterXML/jackson-dataformat-csv/blob/4e2b89f8c904b1dcb2df3e33bc5b3990314ab095/src/main/java/com/fasterxml/jackson/dataformat/csv/impl/CsvParserBootstrapper.java#L294

Apache Commons CSV doesn’t seem to auto-detect it for you, but they do offer BOMInputStream that you can use to transparently wrap an InputStream and discard the BOM if found.

It seems OpenCSV are slightly less generous in solutions to the problem.

[Craigacp]

This will come in the 4.0.2 bug fix release which will happen sometime in the next week or so. I’d hoped for it to happen this week, but I’d like to get the external models tutorial into this release too, and we’re also waiting for the OLCUT 5.1.5 dependency to arrive in Maven Central as that has a necessary bug fix for the provenance system.

In general the frequency of bug fix releases will be controlled by the severity of the bugs. In 4.0.1 there are some serious bugs in JsonDataSource and the RegressionInfo object and we’d like to get those fixed very soon. We’ll pull in the docs updates branch that we’re working on and the csv BOM fix among other smaller things. Feature releases are likely to happen on a longer cadence than monthly as we’re aiming to have feature based releases rather than time based ones.