feature: Offer a method to clean up old states in meltano.db

Feature scope

API

Description

Our Meltano project has a relatively large state (like 20MB), so the size of meltano.db grows quickly. To my best understanding, currently the StateService offers no way to remove old run/job records and I haven’t found a clue in the config for any housekeeping, so we probably have to work around by SQLiting into the DB and delete them by ourselves. It would be nice if we can do this in an official way.

Can we possibly:

1. Have a CLI command to clean up old states? and/or
2. Config meltano for housekeeping, cleaning up old states at some point?

IMO, the feature makes sense even for normal-sized states, for we eventually need to deal with evergrowing runs table.

[simonpai]

I made a PR #6823 to resolve this issue. I’m sharing more context here:

Our use case and the motivation behind this feature request

We are maintainers of target-miso. Many of users’ projects involve handling updates and hard deletes from sources that don’t support log-based replication method, and we solve the issue by a full table method combined with record hash comparison stored in the state. This is really powerful pattern with the only caveat of growing size of meltano.db. This feature helps us keep the size (rather) bounded.

How we work around at the moment

A cronjob which deletes old runs using SQLite, keeping a fixed number of latest runs per state.

[tayloramurphy]

@simonpai thanks for the issue!

There’s an argument for not offering a method to clean up the state DB because it’s potentially useful data for understanding past runs. But I understand the problem here and would be supportive of adding a meltano state remove/delete command. By default it could delete everything but the past X runs but I could see that being configurable as well.

We won’t be prioritizing this ourselves in the near term but would happily take a PR!