datapkg_to_sqlite fails to load all of EPA CEMS

After doing a full ETL of all years and states in CEMS, the datapkg_to_sqlite script doesn’t seem to load all of that data into the SQLite database. Rather, it only loads the last year of data into the database. However, the process terminates quickly, so it’s probably not even attempting to load all the data. Suspect it’s an issue with the iteration and/or partitioning…

[cmgosnell]

Hey @roll! this was totally just our mistake, so don’t worry about it. We are making a bunch of data source specific data packages and then squishing them together into one package, I set up a process for determining how to generate a new data package without duplicating elements of the metadata… but I messed up and it was only grabbing one of the CEMS resources. It was a very simple fix once we figured out what was happening.

[zaneselvans]

No, still getting the error with the most recent versions of the datapackage libraries. A very simple version with a couple of resources in a group seems to work as expected, but the simplest PUDL output that tests the behavior doesn’t work. I’m trying to simplify that resource group output one step at a time until I get to a minimal example to share with you.