Expose fields and objects from `Transaction.Context` through the public agent API

Why?

The current public agent API only exposes a small part of the model that the APM Server expects. These are mainly things directly on Transaction and Span. But Transaction.Context and Span.Context are not exposed. On those contexts we can set things like HTTP request related fields (HTTP method, URL, etc.) or Database related fields.

Since there are lots of libraries that we currently don’t support with auto instrumentation we should expose these fields and let users set those when they rely on the public agent API.

This PR already contains a use case: https://github.com/elastic/apm-agent-dotnet/pull/118

Problem description

To avoid a too high-level discussion, I’ll focus specifically on Transaction.Context.Request, but we should came up with a solution that works with other things on Transaction.Context and Span.Context.

Request has 2 required fields:

• Method
• Url

This means that both of these fields must be set, otherwise the the server would reject the Transaction.

Requirements

(not set it stone, feel free to disagree)

• The main challenge is that we have to make sure that required fields are filled, otherwise the APM server would reject the request. This means we should not have partially initialized objects. E.g.: when the user does this: Agent.Tracer.CurrentTransaction.Request.Method = "GET"; We could lazily create the Request object, but the problem is that the Url property is still null, so if the user does not set that property later then this Request object is invalid.

• Avoid throwing exceptions. For example with the previous example (where the server’d reject the transaction, since not all required fields are filled on Transaction.Context.Request) we could build some verification before we send the data to the apm server (or at another point) and notify the user by throwing an exception and forcing them to avoid partly initialized data. As a monitoring tool one basic principle is to be as non-intrusive as possible, so I think throwing exceptions is not acceptable.

• Avoid skipping data. Another option would be to not send that data and print a warning (or any other log message). I think this’d cause confusion.

• Required fields should be forced on the API level. We should prefer solutions where it’s not even possible to have partly initilized objects. This’d mean the user either sets all the required fields at once, or none of those.

• There should be no NullReferenceExceptions in case something is not initialized. For example if we have let’s say an API where the Request is a property, then something like this should not throw an exception (even if the user did not initialize the Request property before): Agent.Tracer.CurrentTransaction.Request.Method = "GET"; Similarly this should also never throw an exception: var requestMethod = Agent.Tracer.CurrentTransaction.Request.Method

Potential solutions:

1. Lazy initialization (original attempt)

public interface ITransaction
{
   IRequest Request { get; }
}

internal class Transaction : ITransaction
{
  public IRequest Request => _context.Value?.Request;
}

internal class Context
{
  private readonly Lazy<Request> _request = new Lazy<Request>();
  public Response Response => _response.Value;
}

Advantage:

• No NullReferenceException, if the user writes this: Agent.Tracer.CurrentTransaction.Request.Method we immediately initalize a Request

Disadvantage:

• This can cause partially initialized data. (the line above already does that).

2. Exposing 2 methods on ITransaction:

• Setter: Required parameters for required fields and has multiple optional parameters for optional fields
• Getter: Uses ValueTuple and nullchecks if the user haven’t initialized the Request it returns default values if it’s non-initialized, otherwise it returns the values.

public interface ITransaction
{
   void SetRequest(string method, string urlProtocol, string urlFull, string urlHostName, 
      string urlRaw = null, bool socketEncrypted = false, string socketRemoteAddress = null,
      string httpVersion = null, object body = null);

   (string method, string urlProtocol, string urlFull, string urlHostName, string urlRaw, 
      bool socketEncrypted, string httpVersion, object body) GetRequest();
}

internal class Request
{
 // with a ctor we force to pass required fields. 
  public Request(Url url, string method)
  {
    Url = url;
    Method = method;
  }	
  public string HttpVersion {//get&set}
  public object Body {//get&set}
  public string Method {//get&set}
  public Socket Socket { get; set; }
  public Url Url { get; set; }
}


internal class Transaction : ITransaction
{
  public void SetRequest(string method, string urlProtocol, string urlFull, string urlHostName, 
      string urlRaw = null, bool socketEncrypted = false, string socketRemoteAddress = null, 
      string httpVersion = null, object body = null)
     => _context.Value.Request = new Request(new Url() { Full =  urlFull, Protocol =  urlProtocol, 
         Raw =  urlRaw, HostName =  urlHostName}, method){Body =  body, Socket
          =  new Socket() { Encrypted =   socketEncrypted, RemoteAddress =  socketRemoteAddress},
         HttpVersion =  httpVersion};
   
}

public (string method, string urlProtocol, string urlFull, string urlHostName, string urlRaw, bool socketEncrypted,
  string httpVersion, object body)
    GetRequest() => _context.Value.Request == null ?
      (string.Empty, string.Empty, string.Empty, string.Empty, string.Empty, false, string.Empty, null) :
        (_context.Value.Request.Method, _context.Value.Request.Url.Protocol, _context.Value.Request.Url.Full,
          _context.Value.Request.Url.HostName, _context.Value.Request.Url.Raw, _context.Value.Request.Socket.Encrypted,
          _context.Value.Request.HttpVersion, _context.Value.Request.Body);

Here is how the user code’d look like:

transaction.SetRequest(method: "GET", urlProtocol: "HTTP", urlFull:"https://myUrl.com", urlHostName: "MyUrl", httpVersion: "2.0");
				
 //later in the code:
var request = transaction.GetRequest();
Console.WriteLine(request.method);
Console.WriteLine(request.httpVersion); //etc...

Advantages:

• The API forces the user to only create fully initialized objects
• There are no NullReferenceExceptions, if someone writes this var request = transaction.GetRequest().method we just return string.empty. In case of object we return null, but these are always fields that are at the end of the chain…
• We don’t need interfaces… there is no IRequest or IUrl, since the Request and Url objects are completely hidden.

Disadvantage:

• Since there is 1 setter and 1 getter users have to set everything in a single step. If e.g. the body is available only at a later point then the whole call must be moved to a later point in the code.
• We can’t extend the getter… there is nothing like overloads for ValueTuples as return types… the number of parameters for GetRequest() is fixed.

One modification of this approach would be to use specific types as parameters.

3. Introducing intermediate types that work as public API

Implemented in elastic/apm-agent-dotnet#130

4. exposing the intake API as it is.

Implemented in https://github.com/elastic/apm-agent-dotnet/pull/134

@elastic/dotnet: maybe someone has an idea, opinion or just a bright comment.

[gregkalapos]

Discussed this with @SergeyKleyman and @Mpdreamz.

Decision: We expose the model that we have to the APM server with the Intake API as it is and we won’t introduce any intermediate layer on the API. (proposed solution 4.)

Mainly because adding an intermediate layer (like in #130) would mean that we have to maintain a mapping, which is not trivial.

Regarding breaking changes: we assume that on the intake API we will only have breaking changes on major versions and in those cases we will also have breaking changes in the C# API which is acceptable.

The other advantage is that everything which is in the server docs and in the intake API doc automatically applies to the .NET API.

cc: @elastic/apm-agent-devs @roncohen We decided to expose most part of the intake API as it is in the .NET API (e.g.: transaction.context.request can be directly set in C#). If someone has a good argument to not to do this, this would be a good time to let us know.

[gregkalapos]

Done and merged.

We went with “4. exposing the intake API as it is.”. Both Transaction.Context and Span.Context are exposed to users through the Public Agent API.