Bounty: Public database syncing system (150$)

There should be a system that allows anyone to be able to have a near-realtime (5 mins optimal, 20 mins at least) mirror at least the sponsorTimes table.

This was discussed in Discord(message) / Matrix(message).

Possible options:

• Use a system similar to MusicBrainz where a Postgres replication feed gets mirrored in a read-only way for other users to download. There then should be a way for the server to automatically read from this replication feed and import it into their local database. This would have to be done in a secure way that only replicated the public tables.

• Reuse the existing Webhook implementation and create a recieving end for this that can be configured to save this info to the database.

• Host the CSVs via IPFS and have a wrapper to automatically retrieve latest versions of this using IPFS. This should work as IPFS stores files as chunks, and will know which chunks and new and which are old. This also makes it so when someone is first downloading the entire db, everyone’s mirror will act as a seeder, even if they are on a slightly outdated version of the file (since they have access to the specific chunks).

---

The solution should be able to be run easily when starting up the SponsorBlock server. Docker would probably be needed to ease deployment if it requires external dependencies outside of just node.

---

Bounty has been solved. Bounty of $150 USD via PayPal to whoever solves this issue. To avoid duplicate work, make sure to comment on this issue with what you are doing.

[ajayyy]

This is now complete! Check out @mchangrh’s project to see how to host a mirror yourself: https://github.com/mchangrh/sb-mirror

[mchangrh]

Not really, by that logic crypto currencies could never exist.

Yes but sponsorblock isn’t a cryptocurrency, it doesn’t need a single source of truth, the source of truth is done by verifying with other peers. Quite a while ago, when the BTC market was booming, there were concerns since there was a single pool that mined 3 consequitive blocks. If a malicious actor wanted, they could spin up a majority of gun peers and override most of the database. It’s cool tech but doesn’t fit in with master/slave and single source of truth that sponsorblock requires.

This is required by sponsorblock since the vipUsers table should only be accessible to the admin, a bad actor could change the vipUsers table and remove, add locked segments or ban users.

It’s very cool but even if we only use gun for only sponsorTimes there would still be the issues of distributing the other tables.