Treatment of missing values during CEMS transformation

The EPA cems transformation process currently replaces missing values for gross_load_mw and heat_content_mmbtu with the following code: raw_df.fillna({ “gross_load_mw”: 0.0, “heat_content_mmbtu”: 0.0 })

I was wondering if there is a strong reason why these missing values are being filled?

As I’ve been exploring this data further, I have been finding that a null value and a zero value are not interchangable. For example, during startup, heat input and operating time can both be nonzero, but gross load can be zero.

I’d like to be able to identify some methods to fill missing values in CEMS using estimated values, but the current datapackage makes it difficult to identify which values are missing and which are actually reported as zero, since all the missing values have been filled.

I would propose removing the above fillna code in the pudl/src/pudl/transform/epacems.py file.

I don’t think that keeping these values as NA should break any other code further down the line, as the co2_mass_tons column also contains some missing values that were left alone.

[grgmiller]

Just to add some additional support for doing this… according to the EPA’s Power Sector Emissions Data Guide:

A blank in the data (emissions, heat input, load) is not the same thing as a zero. A blank likely indicates that the EGU is not required to report a particular parameter based on the program(s) it is affected by. Checking the program code(s) for the EGU may help explain why certain parameters appear or do not appear in the data. Refer to Table 1 under “What data are collected?” for more information on program reporting requirements.

[aesharpe]

Update: removed the fillna(0) for both gross load and heat content.