Filter mechanism for server transport creation and deletion

To support a per-connection throttling mechanism used inside google, gRPC needs to allow user to add filters to the server that:

1. Gets called when a server transport is created and handshake-completed, and has access to transport-specific attributes such as client address and identity.
2. Adds per-transport attributes to ServerCall attributes.
3. Gets called when a server transport is terminated.

I discussed a few options with @ejona86:

ServerInterceptor

The filter returns a ServerInterceptor when transport is created. The interceptor is transport-scoped, unlike ordinary interceptors that are service-scoped.

interface ServerTransportFilter {
  ServerInterceptor transportCreated(SomePublicServerTransportType transport);
  void transportTerminated(SomePublicServerTransportType transport);
}

Pros:

1. Doesn’t touch transport API (ServerTransport, ServerStream) which are internal.
2. Re-uses ServerInterceptor for purposes that ServerInterceptor is designed for (modifying ServerCall attributes etc).

Cons:

1. Changes the scope of ServerInterceptor. @ejona86 thinks bonding ServerInterceptor to a transport is seemingly generic but actually an ad-hoc solution for a particular problem. I don’t necessarily agree though.

Decorating transport

The filter returns a decorated ServerListener at registration. Through a few more layers of wrapping, the filter would be able to decorate ServerListener.transportCreated(),ServerTransportListener.transportTerminated() and ServerStream.attributes().

Pros:

1. Minimal API change

Cons:

1. Heavily coupled with transport API which is internal.
2. Duplicates the existing interceptor mechanism, but on ServerStream instead of ServerCall. It would be ideal to just decorate ServerCall, which is what ServerInterceptor does. ServerInterceptor currently is per-service, but our use case need it to be per-connection.
3. Even more layers of decoration than the interceptors. The filter can be annoying to write.

Ad-hoc

Just for our original requirements, the filter would return per-transport attributes that are to be merged into ServerCall attributes.

interface ServerTransportFilter {
  Attributes transportCreated(SomePublicServerTransportType transport);
  void transportTerminated(SomePublicServerTransportType transport);
}

It probably won’t work in this form, because the attributes such as peer security identity may not be available immediately after the transport is created, if the security handshake has not completed yet. On the other hand, the first two options can work around this situation by reading the attributes only when the first stream is created.

[zhangkun83]

@ejona86 @louiscryan and I had a further discussion about this issue. We feel that the Context-based interface earlier has two issues:

1. Obscure interface: Using Context cancellation to signal transport termination is implicit and somehow obscure.
2. Context abuse: Context is for passing data that is either consumed by application or propagated downstream. Data generated by the filter is not necessarily consumed by application or propagated downstream. It seems like abusing the Context.

Our consensus is that we keep ServerCall.attributes(), use it to pass data from the filter to interceptors, and call out transport termination explicitly. We have agreed on the following interface:

abstract class ServerTransportFilter {
  // Called after all handshake (e.g., SSL) is done.
  // transportAttrs: contains original transport information, such as
  // socket address and SSL context.
  // return: will be merged into ServerCall.attrs
  Attributes transportReady(Attributes transportAttrs) {
    return transportAttrs;
  }

  // attrs: contains all the data returned by transportReady of the same filter
  void transportTerminated(Attributes attrs);
}

@lukaszx0 WRT Attributes vs. Context, we have drawn a clear line between them. I hope this addresses your confusion.

• Context: for passing data that is either consumed by application or propagated downstream. E.g., deadline, tracing context, end-user credentials.
• Attributes: for passing data from transport to ServerInterceptor, but not necessarily meant to be consumed by application or propagated downstream. Typically per-connection information such as client address, SSL context etc.

ServerInterceptor is the one that consumes Attributes and produces Context.

[lukaszx0]

@zhangkun83 sorry for missing original mention.

I like the idea of using context for everything and removing ServerCall attributes. Even now, when I have fairly good knowledge of internals in this particular area I am sometimes confused if stuff is either in context or call attributes. Consolidating them into one place - context - would make things simpler.

Looping in @jhump who might have opinions on the main issue discussed here.