PIP-210: Admin API should be able to return messages in human-readable formats.

Motivation

Currently, pulsar-admin topics peek-messages and the REST equivalent return the message payload in their original binary form. In the case that the topic has a complex schema, for example, AVRO, protobuf-encoded binary is returned. This results in two issues.

1. The output can be unreadable with hidden charractors.
2. The response from the REST API doesn’t conform to HTTP standard. Binary data are not properly encoded or have the right headers set.

Goal

Ability to ask Pulsar broker’s Admin API to return properly encoded string payload, instead of binary payload.

API Changes

1. API /admin/v2/:schema/:tenant/:namespace/:topic/subscription/:subName/position/:messagePosition will have an optional query parameter asString. Valid values are true or false. When set to be true, the returned payload will be encoded according to the topic’s schema.
2. pulsar-admin topics peek-messages will add an optional flag --string. If set to true, output would have the payload encoded as string per the topic’s schema.

Implementation

When handling peek request, broker admin currently first retrieves the entry from managed ledger. Then read the binary form from the entry before packaging it as part of REST response.

Before the binary is returned, the broker would make a request to schemaRegistryService asking for the schema from the schema storage. Once found, the broker would read the schema and encode the payload accordingly.

Alternatives

No response

Anything else?

No response

[teabot]

When we speak about “string” format, we are being too general. You can have a many different string formats after all: Base64, Yaml, JSON, etc. So I think it is a mistake to refer to this as string. Instead I’d recommend explicitly naming the format, and also including an extensible option mechanism like:

--format=binary     # default for backwards compatibility

--format=json

[codelipenghui]

And in this case is the user constrained to use the java client? How about the use case where the caller is a UI browser?

I think https://www.npmjs.com/package/avro-js and https://github.com/protobufjs/protobuf.js/ can help with the avro and protobuf format.