Why would a ObjectDisposedException occur when writing to a stream?
See original GitHub issueIn an Asp.Net core 3.1 app on linux using the gRPC framework, a generic Throttle class (AmzThrottle
) calls Amazon to get product information. When the API calls to Amazon hit a threshold to begin throttling (wait), a Timer
is set to fire and resume the requests at the appropriate time and then the thread exits. Once the Timer
elapses and the thread begins doing Amazon work again, the priceApi.OnCompletion
callback (WritePriceToResponseStream
) is fired which in turn calls priceRespStream.WriteAsync
. At this point an exception hits because IFeatureCollection has been disposed.
The exception occurs prior to the GetPrices
method exiting. In the AmzThrottle
class for the Timer
callback, the first thing that is completed is to turn the Timer
off. There is just one thread invoking the callback to write to the gRPC stream at a time.
When inspecting the priceRespStream
variable in the WritePriceToResponseStream
callback in the debugger, the _writeTask
member reads: _writeTask = Id = 7, Status = RanToCompletion, Method = "{null}", Result = "System.Threading.Tasks.VoidTaskResult"
. The crash happens prior to the priceTcs
TaskCompletionSource being triggered with SetResult(true)
in the GetPrices
method.
Internet searches show that miss-use of HttpContext can cause this, but I’m not using that class directly at all.
What am I doing wrong to cause this exception?
Log information:
2020-11-24 13:14:00.578 -08:00 [ERR] Failed to write price response gRPC message to stream.
System.ObjectDisposedException: IFeatureCollection has been disposed.
Object name: 'Collection'.
at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
at Microsoft.AspNetCore.Http.DefaultHttpContext.get_RequestAborted()
at Grpc.AspNetCore.Server.Internal.HttpContextStreamWriter`1.WriteAsync(TResponse message)
at Amazon.Services.AmzProductsSvc.WritePriceToResponseStream(IServerStreamWriter`1 priceRespStream, EventArgs e) in d:\<snip>
Server:
public class AmzProductsSvc : AmazonProductsApi.AmazonProductsApiBase {
private readonly AmzThrottle<GetLowestPricedOffers> priceApi;
private int priceCount;
private int expectedPriceCount;
private TaskCompletionSource<bool> priceTcs;
public AmzProductsSvc(AmzThrottle<GetLowestPricedOffers> priceApi) {
this.priceApi = priceApi;
}
public override async Task GetPrices(PriceRequests request, IServerStreamWriter<PriceResponse> priceStream, ServerCallContext context) {
try {
var hiPri = request.Request.Where(r => Priority.High == r.Queue)?.Select(a => a.Asin)?.ToList();
var nPri = request.Request.Where(r => Priority.Normal == r.Queue)?.Select(a => a.Asin)?.ToList();
expectedPriceCount = hiPri.Count + nPri.Count;
priceTcs = new TaskCompletionSource<bool>();
priceApi.OnCompletion += async(sender, args) => await WritePriceToResponseStream(priceStream, args);
if(null != hiPri && 0 < hiPri.Count) {
var hiPriParams = new List<AmzApiParams>();
foreach(var hp in hiPri)
hiPriParams.Add(new AmzApiParams { Input = hp } );
priceApi.AddRangeToQueue(hiPriParams, Priority.High);
}
if(null != nPri && 0 < nPri.Count) {
var normPriParams = new List<AmzApiParams>();
foreach(var np in nPri)
normPriParams.Add(new AmzApiParams{ Input = np } );
priceApi.AddRangeToQueue(normPriParams, Priority.Normal);
}
await priceTcs.Task;
}
catch (Exception ex) {
Log.Error(ex, string.Empty);
}
}
private async Task WritePriceToResponseStream(IServerStreamWriter<PriceResponse> priceRespStream, EventArgs e) {
try {
var args = (GetLowestPricedOffersEventArgs)e;
await priceRespStream.WriteAsync(new PriceResponse {
Asin = args.ASIN,
Price = (double)args.Price,
Notfound = args.NotFound
}
);
}
catch (Exception ex) {
Log.Error(ex, "Failed to write price response gRPC message to stream.");
}
finally {
priceCount++;
if(expectedPriceCount == priceCount)
priceTcs.SetResult(true);
}
}
}
Client:
private async Task SetProductPricesFromAmazon(List<Product> products, string schedName) {
try {
using(var channel = GetGrpcChannel()) {
var client = new AmazonProductsApi.AmazonProductsApiClient(channel);
var req = new PriceRequests();
foreach(var prod in products) {
if(null == req.Request.FirstOrDefault(r => r.Asin.Equals(prod.ASIN, StringComparison.InvariantCultureIgnoreCase)))
req.Request.Add(new PriceRequest{ Asin = prod.ASIN, Queue = Priority.High });
}
var stream = client.GetPrices(req);
var tokenSource = new CancellationToken();
var numProcessed = 0;
await foreach(var price in stream.ResponseStream.ReadAllAsync(tokenSource)) {
numProcessed++;
var isFound = string.IsNullOrEmpty(price.Notfound) && string.IsNullOrEmpty(price.Error);
if(isFound) {
// We sometimes have multiple products with the same ASIN match
var prodList = products.Where(p => p.ASIN.Equals(price.Asin));
foreach(var prod in prodList) {
prod.AmznPrice = Convert.ToDecimal(price.Price);
prod.IsUpdated = true;
}
}
if(req.Request.Count == numProcessed) {
await Product.SaveAsync(products, schedName);
return;
}
}
}
}
catch (Exception ex) {
Log.Error(ex, string.Empty);
}
}
Asp.Net core startup:
public class Startup {
public void ConfigureServices(IServiceCollection services) {
services.AddGrpc();
services.AddSingleton<AmzThrottle<GetLowestPricedOffers>>();
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env) {
if (env.IsDevelopment()) {
app.UseDeveloperExceptionPage();
}
app.UseRouting();
app.UseEndpoints(endpoints => {
endpoints.MapGrpcService<AmzProductsSvc>();
endpoints.MapGet("/", async context => {
await context.Response.WriteAsync("Communication with gRPC endpoints must be made through a gRPC client.");
});
});
}
}
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:8 (4 by maintainers)
Top GitHub Comments
FYI https://www.nuget.org/packages/Grpc.AspNetCore with version 2.34.0-pre1 or later now gives a better error message. It’s something like:
Can't write message to completed gRPC call.
I’ve gathered the debug spew and edited it to remove excess noise from an overnight run. The code in the orginal post only includes the pricing API for the Amazon service since I was trying to minimize the post. There is another API call (/KeepaApi/GetProductInfo) that is in the spew below. I didn’t want to edit the gRPC spew that I wasn’t sure of so I left the keepa gRPC spew in there. The difference with the Keepa gRPC API is how I wait for the request to finish. In the
GetPrices
, the TaskCompletionSource is used. In the Keepa call,AwaitCancellation
is used. If you would look through the spew to see what is going wrong I’d appreciate it.