Modify EventCounter usage to support the new metric APIs as well

Tasks:

• HostingEventSource
• ~ConcurrencyLimiterEventSource~ (obsolete in .NET 8)
• KestrelEventSource
• HttpConnectionsEventSource
• Rate limiting - https://github.com/dotnet/aspnetcore/pull/47758
• Move to Microsoft.Extensions.Metrics - https://github.com/dotnet/aspnetcore/issues/47618
• Fix counter names - https://github.com/dotnet/aspnetcore/issues/48309
• Review metrics tag types - https://github.com/dotnet/aspnetcore/issues/48530
• Docs about how to export metrics to Prometheus and Grafana
• Docs about how to unit test metrics - https://github.com/dotnet/AspNetCore.Docs/pull/30068
• Add protocols to Kestrel connection metric - https://github.com/dotnet/aspnetcore/issues/47831
• ~Add authn/authz counters~ Moved a new issue: https://github.com/dotnet/aspnetcore/issues/47603

---

I’ve been experimenting with how a library could optionally support the new Meter APIs while also being back-compatible with pre-existing EventCounters. It would be nice to test this on a more real-world use case in ASP.Net to see if we like how it plays out. If it doesn’t look good of course we could always change the pattern.

The rough pattern I had was to:

1. Add the new Meter API usage SxS with the EventCounters.
2. Switch code that was directly invoking into EventCounter.WriteMetric + IncrementingEventCounter.WriteMetric so that it instead calls Counter.Add() or Histogram.Record().
3. Add a forwarder so that those Counter.Add() + Histogram.Record() calls with also chain to invoking the pre-existing WriteMetric().
4. For PollingCounter + IncrementingPollingCounter, pull the data from the same API that ObservableCounter/ObservableGauge pull it from.

In total it looked like this… New Meter code + app logic:

using System;
using System.Collections.Generic;
using System.Diagnostics.Metrics;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApp
{
    class Program
    {
        static Meter s_meter = new Meter("ContusoHatStore");
        internal static Counter<int> s_hatsSold = 
            s_meter.CreateCounter<int>("Hats-Sold", "Hats", "Number of Hats Sold");
        internal static ObservableCounter<long> s_transactionsProcessed = 
            s_meter.CreateObservableCounter<long>("TransactionsProcessed", GetTransactions, "Transactions", "Number of Transactions queued");
        internal static ObservableGauge<long> s_heapSize = 
            s_meter.CreateObservableGauge<long>("Heap-Size", GetHeapSize,
            "B", "Size of GC heap");
        internal static Histogram<double> s_histogram = 
            s_meter.CreateHistogram<double>("Request-Latency", "ms", "Request Latencies");

        static long s_transactions;
        internal static long GetHeapSize() => GC.GetTotalMemory(false);
        internal static long GetTransactions() => s_transactions;

        static async Task Main(string[] args)
        {
            ContusoHatStoreEventSource contusoHatStoreEventSource = new ContusoHatStoreEventSource();

            CancellationTokenSource cts = new CancellationTokenSource();
            Task t = ProcessPretendTransactions(cts.Token);
            Console.WriteLine("Transactions running, hit enter to stop");
            Console.ReadLine();
            cts.Cancel();
            await t;
        }

        static async Task ProcessPretendTransactions(CancellationToken token)
        {
            while(!token.IsCancellationRequested)
            {
                PretendTransaction();
                await Task.Delay(10);
            }
        }

        static void PretendTransaction()
        {
            Random r = new Random();
            s_hatsSold.Add(r.Next(1, 5), KeyValuePair.Create<string,object>("Size", r.Next(3,10)), KeyValuePair.Create<string,object>("Color", r.Next(1000) > 500 ? "Blue" : "Red"));
            s_transactions++;
            s_histogram.Record(r.Next(100, 500));
        }
    }
}

Pre-existing EventCounter code + forwarder:

using System;
using System.Collections.Generic;
using System.Diagnostics.Metrics;
using System.Diagnostics.Tracing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp
{
    [EventSource(Name ="ContusoHatStore")]
    class ContusoHatStoreEventSource : EventSource
    {
        IncrementingEventCounter m_hatsSold;
        IncrementingPollingCounter m_transactionsProcessed;
        PollingCounter m_heapSize;
        EventCounter m_requestLatencies;
        InstrumentForwarder _forwarder = new InstrumentForwarder();

        public ContusoHatStoreEventSource() : base("ContusoHatStore")
        {
            m_hatsSold = new IncrementingEventCounter("Hats-Sold", this)
            {
                DisplayName = "Hats Sold",
                DisplayUnits = "Hats"
            };
            m_transactionsProcessed = new IncrementingPollingCounter("Transactions", this, () => Program.GetTransactions())
            {
                DisplayName = "Transactions",
                DisplayUnits = "Transactions"
            };
            m_heapSize = new PollingCounter("Heap-Size", this, () => Program.GetHeapSize())
            {
                DisplayName = "Heap Size",
                DisplayUnits = "B"
            };
            m_requestLatencies = new EventCounter("Request-Latency", this)
            {
                DisplayName = "Request Latency",
                DisplayUnits = "ms"
            };

            _forwarder.Forward(Program.s_hatsSold, value => m_hatsSold.Increment(value));
            _forwarder.Forward(Program.s_histogram, value => m_requestLatencies.WriteMetric(value));
        }
    }

    class InstrumentForwarder
    {
        MeterListener _listener = new MeterListener();
        public void Forward<T>(Counter<T> counter, Action<T> action) where T : struct
        {
            _listener.SetMeasurementEventCallback<T>((instrument, value, tags, state) => { ((Action<T>)state)(value); });
            _listener.EnableMeasurementEvents(counter, action);
        }
        public void Forward<T>(Histogram<T> histogram, Action<T> action) where T : struct
        {
            _listener.SetMeasurementEventCallback<T>((instrument, value, tags, state) => { ((Action<T>)state)(value); });
            _listener.EnableMeasurementEvents(histogram, action);
        }
    }
}

@shirhatti

[noahfalk]

I forgot we had this issue. The current path I want to explore is if we can have the runtime automatically create a Meter with the same name for any EventSource that creates counters. If that path was successful ASP.Net wouldn’t need to do any specific work until you wanted new features not available with EventCounters (such as histograms for latency percentiles).

[JamesNK]

Added a couple of work items to do in .NET 8.

@JamesNK Is this done now?

It will never be done 😈