Using text parameters for replication

Hi everybody! I’m playing with the logical decoding feature, but I fail to see how to actually use the data I receive.

I explain a little further.

Messages of type DeleteMessage, FullDeleteMessage, FullUpdateMessage, IndexUpdateMessage, InsertMessage, KeyDeleteMessage, UpdateMessage contain at least one list (actually Memory<>) of TupleData. TupleData contains either:

• NULL wihth two meaning

  • null when Kind == TupleDataKind.Null
  • unchanged when Kind == TupleDataKind.UnchangedToastedValue

• the stringified value (when Kind == TupleDataKind.TextValue).

This follows what pgoutput exports. The stringified version is not the .ToString of C#, but the result of the text output function for the type of the related column (as in CREATE TYPE).

Now if I wanted to replay the transaction to a secondary server I would need to:

• open a connection to the secondary database;
• generate a proper query;
• add the parameters with the values I received;
• execute the command.

This would possibly happen in a transaction. However, I only have the string representation of the parameters, so I can’t find a way to pass the non-string parameters as strings.

If I was using libpq, that would be very trivial as I would just call PQexecParams or similar and specifying the OIDS of my types (taken from the RelationMessage), my string values, and 0-initialized array for the paramFormats parameter. This is also guaranteed to work for user-defined types as textual input-output functions are required.

On the other side, I can’t find anything in the documentation that allows me to do such a thing with npgsql, except encoding the text directly in the query which implies escaping (with all the problems it brings).

As a last note, to my surprise, NpgsqlDbType enum does not use the well-known type OIDs under the hood, so AFAIK there is no way with Npgsql to map a type OID returned by its own API (in the RelationMessage::Column.DataTypeId field) to a usable NpgsqlDbType. While this is (of course) a minor problem it still makes it difficult to actually use the replication feature.

[marco6]

Hi @roji ! I read all of your comments but I didn’t have time to answer yet. I hope to have a couple of hours during this weekend 😃 Thank you for your support and thoughts so far!

[roji]

@marco6 most of the discussion above seems to be covered by the other linked issues, so I’m going to go ahead and close this one. But we can continue having general conversation here for sure.