Excessive data download

The problem

When a wallet is opened in Decrediton, proposal voting data is immediately downloaded from Politeia. This includes all vote data for all proposals, which currently represents a >40MB download every single time Decrediton opens a mainnet wallet.

This is very wasteful, and seems to be getting worse every time voting is opened on a new proposal.

I can see that gzip compression is already being applied to the data, so I don’t think we have any easy savings there. Server response time is not great, so there may be savings possible there, but for this issue lets focus on the sheer amount of data downloaded.

Analysis of the data

/api/v1/proposals/{proposal_id}/votes is called once per proposal (example), downloading ~3MB each time. Each of the responses comprises:

• Tickets IDs eligible for voting (80% of total data)
• Votes cast
  • Signatures (6.5%)
  • Ticket IDs (3.5%)
  • Tokens (3.5%)

Suggestions

• The vast majority of the data, the eligible tickets, is static and known from the moment voting begins. This could be downloaded once and cached by decrediton indefinitely. Downloading it multiple times is wasteful.
• For votes which are known to be concluded, data is static and can also be cached. Only votes which are currently in progress need to be polled.
• Does decrediton even need all of this data? The vote signatures, tokens and ticket IDs for all votes?
• When a wallet is closed and re-opened (without closing decrediton) the data already in memory should be persisted and not re-downloaded.
• Perhaps we could defer downloading the data for historical votes which have already concluded? Does historical data really need to be downloaded immediately upon opening the wallet?

[lukebp]

@vctt94 more like this:

• Each time a proposal starts a vote, calls VoteResults once, if not called.
• Cache the eligible tickets snapshot in some plain text file
• When need to vote use the cached eligible tickest snapshot to determine which of the user’s tickets can vote.
• Once the user votes, mark the tickets as voted and save this information locally. This is how Decredition will know whether the user’s tickets have voted. Previously, it would fetch the VoteResults.CastVotes to determine this, but this call is too expensive and should be avoided.
• After proposal finishes vote, delete cached information.

[lukebp]

Here’s the problem with a batch-votes endpoint. If you’re fetching all of the cast vote data at once, it’s going to be a lot of data. Far more data than just the eligible ticket snapshot. So if you’re fetching this data for multiple proposals on the “Finished Voting” tab, the performance is going to be unacceptably slow.

If you’re proposing an endpoint that allows you to specify specific ticket hashes that you want cast vote data for, that’s a privacy issue. We don’t want an endpoint that encourages people to tell the server what tickets they control.

What I’m proposing is to only display the cast vote data on the “Finished Voting” tab for proposals where the user voted using decrediton. So when a user votes using decrediton, you save that data locally somewhere and use it to populate the “Finished Voting” tab. If a user voted using some other method or if they restore an existing wallet then decrediton will have no knowledge of prior cast vote data and it will not be displayed on the “Finished Voting” tab.

Another option would be to not display the cast vote data on the “Finished Voting” tab at all. If a user wanted to look back at what votes they cast then there would be some button that they would have to click and it would fetch all of the cast vote data for that proposal, at which point decrediton can filter out the users votes. This would prevent having to fetch the cast vote data for multiple proposals at once.