Huge memory consumption when processing file

Hello!

When I’m reading file manually with code like this: ` int rowCntr = 0;

            List<List<object>> rows = new List<List<object>>();

            List<object> row;

            using (var reader = ExcelReaderFactory.CreateReader(stream))
            {
                try
                {
                    do
                    {
                        rowCntr++;
                        while (reader.Read())
                        {
                            int columnsCount = reader.FieldCount;
                            row = new List<object>();
                            for (int i = 0; i < columnsCount; i++)
                            {
                                row.Add(reader.GetValue(i));
                            }
                            rows.Add(row);
                        }
                    } while (reader.NextResult());
                }
                catch (Exception)
                {
                    MessageBox.Show(rowCntr.ToString(), "");
                }
            }

` my application grows to approx 400-500 mbytes of operative memory.

When I’m trying to read xls file like this: `

            using (var reader = ExcelReaderFactory.CreateReader(stream))

            {
                try
                {
                    result = reader.AsDataSet();
                }
                catch (Exception)
                {
                    MessageBox.Show(", "");
                }
            }

` my application grows up to approx 800-900 mbytes of operative memory. It looks like a problem. When I debugged first approach, I found that rows collection contains… 1.048.567 elements! My file contains just 17 (!!!) rows. It looks like Excel Reader doesn’t see the end of non-empty part of the file.

[appel1]

For AsDataSet I think it makes more sense to include empty lines except for trailing empty lines by default to better match what you see in Excel which I think most expect. The DataSet can’t afaik include metadata so trailing lines with just metadata should count as empty solving this issue. Doesn’t hurt with a row filter callback also for more advanced scenarios.

[andersnm]

I think we can resolve this issue (and #160) by implementing a new row filtering callback option for AsDataSet(). By default this option can exclude ALL empty rows, and the caller can override it if needed.

(it won’t allow for skipping only trailing empty rows, but isn’t a requirement)