Grpc Error in `dotnet-isolated` in net7 and docker
See original GitHub issueI have two functions, one is a CronTrigger, which appears to be working fine the other is an OrchestrationTrigger function. This is running in dotnet-isolated
, net7, in docker.
I can see in my logs that the cron is launching, it also successfully schedules the other function, i can see the instanceId in the logs.
[2022-12-17T07:00:04.364Z] I "Start: 3fdccdf3-be99-40e3-95c8-53cf6c590910 Microsoft.Azure.Functions.Worker.TimerInfo" {"category":"Recovery.Functions.HelloProviders"}
[2022-12-17T07:00:04.383Z] Start: 3fdccdf3-be99-40e3-95c8-53cf6c590910 Microsoft.Azure.Functions.Worker.TimerInfo
[2022-12-17T07:00:04.616Z] I "Started: 5c832cad72f1403ead887f06e6456f8a..." {"category":"Recovery.Functions.HelloProviders"}
[2022-12-17T07:00:04.617Z] Started: 5c832cad72f1403ead887f06e6456f8a...
[2022-12-17T07:00:04.728Z] Executed 'Functions.CronTest' (Failed, Id=2e972014-43a1-479e-8b10-a412039749a2, Duration=6640ms)
[2022-12-17T07:00:04.728Z] System.Private.CoreLib: Exception while executing function: Functions.CronTest. System.Private.CoreLib: Result: Failure
...
The line where I call WaitForInstanceCompletionAsync(instanceId, ...)
is where it throws an exception. Here are the details:
{
"created": "@1671260404.629909577",
"description": "Error received from peer ipv4:127.0.0.1:4001",
"file": "/var/local/git/grpc/src/core/lib/surface/call.cc",
"file_line": 1070,
"grpc_message": "Exception was thrown by handler.",
"grpc_status": 2
}
I’m not sure how this can happen. This is running in a pod, which has both processes…
Example code
using System.Threading;
using Microsoft.Azure.Functions.Worker;
using Microsoft.DurableTask;
namespace Recovery.Functions;
public class HelloExample
{
private readonly ILogger logger;
public HelloExample(ILogger<HelloExample> logger) => this.logger = logger;
[Function(nameof(HelloExample))]
public async Task<string> Hello([OrchestrationTrigger] TaskOrchestrationContext context, string name)
{
var instanceId = context.InstanceId;
this.logger.LogInformation("Hello Workflow Running: {name} ({instanceId})", name, instanceId);
return await Task.FromResult($"Hello {name}!");
}
[Function(nameof(CronTest))]
public async Task CronTest(
[TimerTrigger("* * * * * *")] TimerInfo timerInfo,
[DurableClient] DurableClientContext durableContext,
FunctionContext context
)
{
this.logger.Info($"Start: {context.FunctionId} {timerInfo}");
var instanceId = await durableContext.Client.ScheduleNewOrchestrationInstanceAsync(nameof(HelloExample), "Justin");
this.logger.LogInformation("Started: {instanceId}...", instanceId);
var result = await durableContext.Client.WaitForInstanceCompletionAsync(instanceId, CancellationToken.None, true);
this.logger.LogInformation("Result: {SerializedOutput}", result.SerializedOutput);
await Task.CompletedTask;
}
}
Full error and stack
System.Private.CoreLib: Exception while executing function: Functions.CronTest. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Unknown", Detail="Exception was thrown by handler.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1671260404.629909577","description":"Error received from peer ipv4:127.0.0.1:4001","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Exception was thrown by handler.","grpc_status":2}"))
---> Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1671260404.629909577","description":"Error received from peer ipv4:127.0.0.1:4001","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Exception was thrown by handler.","grpc_status":2}")
at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.WaitForInstanceCompletionAsync(String instanceId, CancellationToken cancellationToken, Boolean getInputsAndOutputs)
at Recovery.Functions.HelloExamples.CronTest(TimerInfo timerInfo, DurableClientContext durableContext, FunctionContext context) in /var/recovery/apps/recovery-provider-api/src/recovery-provider-api/Functions/HelloExamples.cs:line 33
at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 18
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 44
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.DurableTaskFunctionsMiddleware.Invoke(FunctionContext functionContext, FunctionExecutionDelegate next) in /_/src/Worker.Extensions.DurableTask/DurableTaskFunctionsMiddleware.cs:line 18
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 82.
Issue Analytics
- State:
- Created 9 months ago
- Comments:9
Top Results From Across the Web
Unable to connect to net core gRPC service hosted in Docker
After running docker-compose file, the console app is just not able to connect to gRPC. Using http, I get this error: Grpc.Core.RpcException ......
Read more >Guide for running C# Azure Functions in an isolated worker ...
Learn how to use a .NET isolated worker process to run your C# functions in Azure, which supports non-LTS versions of .NET and...
Read more >Troubleshoot gRPC on .NET
The following error message is returned when calling a gRPC service without a trusted certificate: Unhandled exception.
Read more >Azure Functions running in Azure Container Apps with Dapr
Troubleshooting. Possible errors you might face when running the docker-compose: Bind for 0.0.0.0:6379 failed: port is already allocated.
Read more >simple-monitor having fatal gRPC error when starting up
I'm trying to run the simple-monitor 2.4.1 via intelliJ locally, and following the instruction from the GitHub page, having a zeebe broker ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
i have resolved the issue need register on program.cs
I have same issue , how to ?