[BUG] Publisher Error 413 RequestedSizeExceeded

Release version

v4.0.2

Describe the bug

Currently we’re getting an Exception using the Publisher Tool when we try to publish an API that use an OpenAPI JSON file bigger than 7 MB. I’ve already tried to switch to the latest version (4.6.0), but this exception persist. To understand this problem I’ve tried to debug this tool and seems that this is a system limitation of the Azure API REST, I’ve also tried to change it manually to a newer version (from 2021-12-01-preview to 2022-08-01 or 2023-03-01-preview) but this limitation persist. The funny thing, is that if we try to manual publish the same OpenAPI from the Azure Portal works! So this is a limitation of the Azure API REST and not of the APIM resource.

Here the Output error that we get from the Debugger.

crit: Publisher[0]
      System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/apis/***?api-version=2021-12-01-preview failed with status code 413. Content is '{"error":{"code":"RequestedSizeExceeded","message":"The size of requested exceeded the system limitation. Diagnostic information: timestamp '20230725T124805Z', subscription id 'f360b703-2e57-417e-9af1-a05e70bc12f5', tracking id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99', request correlation id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99'."}}'.
         at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Http.cs:line 87
         at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Http.cs:line 72
         at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext() in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Program.cs:line 168
      --- End of stack trace from previous location ---
         at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 273
         at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 371
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Functional.cs:line 45
         at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 365
         at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Service.cs:line 53
         at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 84
         at publisher.Publisher.Run(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 69
         at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 47
Exception thrown: 'System.InvalidOperationException' in publisher.dll
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
Microsoft.Hosting.Lifetime: Information: Application is shutting down...
fail: Microsoft.Extensions.Hosting.Internal.Host[9]
      BackgroundService failed
      System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/apis/***?api-version=2021-12-01-preview failed with status code 413. Content is '{"error":{"code":"RequestedSizeExceeded","message":"The size of requested exceeded the system limitation. Diagnostic information: timestamp '20230725T124805Z', subscription id 'f360b703-2e57-417e-9af1-a05e70bc12f5', tracking id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99', request correlation id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99'."}}'.
         at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Http.cs:line 87
         at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Http.cs:line 72
         at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext() in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Program.cs:line 168
      --- End of stack trace from previous location ---
         at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 273
         at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 371
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()

         at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 365
         at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Service.cs:line 53
         at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 84
         at publisher.Publisher.Run(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 69
         at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 47
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
Microsoft.Extensions.Hosting.Internal.Host: Error: BackgroundService failed

System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/apis/***?api-version=2021-12-01-preview failed with status code 413. Content is '{"error":{"code":"RequestedSizeExceeded","message":"The size of requested exceeded the system limitation. Diagnostic information: timestamp '20230725T124805Z', subscription id 'f360b703-2e57-417e-9af1-a05e70bc12f5', tracking id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99', request correlation id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99'."}}'.
   at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Http.cs:line 87
   at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Http.cs:line 72
   at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext() in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Program.cs:line 168
--- End of stack trace from previous location ---
   at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 273
   at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 371
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
--- End of stack trace from previous location ---
   at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\common\Functional.cs:line 45
   at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Api.cs:line 365
   at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Service.cs:line 53
   at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 84
   at publisher.Publisher.Run(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 69
   at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in C:\Users\mattia.contessa\Dev\apiops\tools\code\publisher\Publisher.cs:line 47
   at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

Expected behavior

Be able to publish API with an OpenAPI file bigger that 7 MB.

Actual behavior

The tool obtain an 413 error from the Azure REST API with the error message

System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.ApiManagement/service/***/apis/***?api-version=2021-12-01-preview failed with status code 413. Content is '{"error":{"code":"RequestedSizeExceeded","message":"The size of requested exceeded the system limitation. Diagnostic information: timestamp '20230725T124805Z', subscription id 'f360b703-2e57-417e-9af1-a05e70bc12f5', tracking id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99', request correlation id '8c70fd0e-1720-4fd8-a9e6-1149d4833d99'."}}'.

Reproduction Steps

Run the Publisher tool to publish at least one API with the specification.json file bigger than 7 MB

[number-phi]

No. As long as the apim instance can access the file then you should be fine. Exposing it publicly would pose a security risk. Please keep us posted so we can update this thread and close it.

Absolutely, as I said, today I’m not able to test it, tomorrow I will create a private storage (reachable by the APIM) and try this solution. I will update you @waelkdouh with a new comment asap. Thanks!!

[waelkdouh]

We are looking into it from the apiops side.