Column types are wrong with larger scan depth and empty values, how to handle empty values

Hello I am using choETL version 1.0.9.9. I like your lib. a lot 🥇 since it offers Data type discovery, some of this does not work with new releases so I opted to stay at that version.

I need help in configuring this correctly to handle Type discovery even if there are empty values.

The Cost1 Cost2,..3. columns data type, shows up wrongly as String, when it should be double/number. using WithMaxScanRows(6), but if I change it to WithMaxScanRows(3) it works fine.

using (var choReader = new ChoCSVReader(someCsvFile).WithFirstLineHeader().WithMaxScanRows(3))
         {   
             var   dataTable = choReader.AsDataTable();

             foreach (DataColumn column in table.Columns)              
              var checkType =  SQLGetType(column) ; // if scan rows is 3, it works fine, but if its 6 i get string, 
              // how can I ensure it gives me the proper dataype even if there are empty data cells.
             
            ...
         }

But when I set WithMaxScanRows(3) the Cost1..2..3. show up correctly as double.

---

CSV file

Date,Project,Login,Area,Cost1,Cost2,Cost3,Cost4,Cost5 2018-03-19 23:00:00,jasonLogin,sr1,taskscompleted,21.0,0.0,0.0,0.0,0.0 2018-06-27 23:00:00,jasonLogin,bblackmon,aircraftdelivery,480.0,0.0,0.0,0.0,0.0 2017-12-19 23:00:00,jasonLogin,sfinishing1,hoursearned,94.73,0.23333333333333334,0.0,0.0,0.0 2018-03-27 23:00:00,jasonLogin,bblackmon,hoursearned,46.5,0.0,0.0,0.0,0.0 2017-12-05 23:00:00,jasonLogin,tmodrick,toolissues,0.0 2018-03-11 23:00:00,jasonLogin,jclark,hoursearned,72.39,0.0,0.0,0.0,0.0 2018-02-28 23:00:00,jasonLogin,jmartinez,housekeeping1,30.0,0.0,0.0,0.0,0.0 2018-02-17 23:00:00,jasonLogin,sfinishing1,budgetrevenue,2018.94,0.0,0.0,0.0,0.0 2018-05-30 23:00:00,jasonLogin,khall,hoursworked,45.7,0.0,0.0,0.0,0.0 2018-03-28 23:00:00,jasonLogin,finish1,budgetrevenue,4834.77,0.0,0.0,0.0,0.0 2017-12-15 23:00:00,jasonLogin,btwilley,aircraftmove1,0.0

[Cinchoo]

On the top ‘Include Prerelease’ checkbox, check that box.

[Cinchoo]

well, RecordFieldTypeAssessment is called for each record, to take control of assessing the field type from its contents.

On the other hand, there is another event MembersDiscovered, this get called once where you can assign field types without considering the payload.

p.MembersDiscovered += (o, e) =>
{
    var ft = e.Value; 

    ft["Id.x"] = typeof(short);
    ft["Name"] = typeof(string);
    ft["City"] = typeof(string);
};

PS. found a bug that this method get called twice in the life of CSV load. Put a fix in 1.1.0.5-beta6 package.