gRPC health checks in kubernetes scenario - unable to execute on demand only subset of healthchecks

What version of gRPC and what language are you using?

2.53.0

What operating system (Linux, Windows,…) and version?

macos

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

6.0.16

What did you do?

Trying to configure kubernetes grpc healthchecks for my service. I want to have different healthchecks executed on demand for startup and liveness probe.

I’ve tried to use the following configuration:

        services
            .AddGrpcHealthChecks(o =>
            {
                // startup probe checks dependencies
                o.Services.MapService("startup", r => r.Tags.Contains("startup"));
                // startup probe checks dependencies
                o.Services.MapService( "liveness", r => r.Tags.Contains("liveness"));
                o.Services.MapService("", r => r.Tags.Contains("liveness");
                o.UseHealthChecksCache = false;
            })
            ... //definitions here

           //auto update just liveness ones
           services.Configure<HealthCheckPublisherOptions>(options =>
           {
             options.Predicate = (r) => r.Tags.Contains("liveness");
           });

What did you expect to see?

grpc_cli call IP:PORT grpc.health.v1.Health/Check "service: 'startup'" will run only checks tagged with startup grpc_cli call IP:PORT grpc.health.v1.Health/Check "service: 'liveness'" will run only checks tagged with liveness

What did you see instead?

grpc_cli call IP:PORT grpc.health.v1.Health/Check "service: 'startup'" doesnt not run startup tagged checks at all!

Code hint

In the newest code I’ve found that the CheckHealthAsync is executed with different predicate than GetStatus. Is it by design?

        HealthCheckResponse.Types.ServingStatus status;
        if (_grpcHealthCheckOptions.Services.TryGetServiceMapping(service, out var serviceMapping))
        {
            var result = await _healthCheckService.CheckHealthAsync(_healthCheckOptions.Predicate, cancellationToken);
            status = HealthChecksStatusHelpers.GetStatus(result, serviceMapping.Predicate);
        }

[JamesNK]

Ok, I understand.

Rather than have two predicates, what I’d like to do is:

• Obsolete the current predicate on ServiceMapping that takes a HealthResult.
• Add a new predicate that takes the health registration name, tags, and maybe description. Those values are common between HealthCheckRegistration and HealthResult.
• The new predicate can be used in both places:
  • The health checks publisher to filter health results
  • CheckHealthAsync to determine what health checks should run for a service name.

[banatm-abb]

Potential solution could be to:

• add new predicate in ServiceMapping Func<HealthCheckRegistration, bool> ExecutionPredicate { get; } with default value to return always true
• use bothservice.ExecutionPredicate and _healthCheckOptions.Predicate in GetHealthCheckResponseAsync

        HealthCheckResponse.Types.ServingStatus status;
        if (_grpcHealthCheckOptions.Services.TryGetServiceMapping(service, out var serviceMapping))
        {
            var result = await _healthCheckService.CheckHealthAsync(
                (r) =>
                    (_healthCheckOptions.Predicate == null || _healthCheckOptions.Predicate(r))
                    && serviceMapping.ExecutionPredicate(r),
                cancellationToken
            );
            status = HealthChecksStatusHelpers.GetStatus(result, serviceMapping.Predicate);
        }