BigQuery: System.OutOfMemoryException thrown when deleting data from large tables

When executing “Delete” queries on large BigQuery tables (100 million+ rows), a System.OutOfMemoryException is being thrown from the BigQuery .NET driver. Occasionally, this will cause the process to terminate unexpectedly instead of throwing an exception.

An example table with this issue has 1.2 billion rows, 26 columns, and 292 GB of data

Environment details

• OS: Ubuntu 18.04
• .NET version: .NET Core 2.1.401
• Package name and version: Google.Cloud.BigQuery.V2 1.3.0-beta05

Steps to reproduce

1. Create a table with at least 100 million rows of data in BigQuery
2. Execute the following code:

var credential = GetGoogleCredential();
using (var client = await BigQueryClient.CreateAsync("projectId", credential)) {
   client.ExecuteQuery("DELETE FROM `projectId.dataset.table` WHERE TRUE");
}

Stack Trace

System.AggregateException: One or more errors occurred. —> System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown. at System.String.CreateStringFromEncoding(Byte* bytes, Int32 byteLength, Encoding encoding) at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count) at System.Net.Http.HttpContent.ReadBufferAsString(ArraySegment1 buffer, HttpContentHeaders headers) at System.Net.Http.HttpContent.ReadBufferedContentAsString() at System.Net.Http.HttpContent.<>c.<ReadAsStringAsync>b__36_0(HttpContent s) at System.Net.Http.HttpContent.<WaitAndReturnAsync>d__632.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Google.Apis.Services.BaseClientService.<DeserializeResponse>d__331.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Google.Apis.Requests.ClientServiceRequest1.<ParseResponse>d__34.MoveNext()

[jskeet]

I’ll look into exactly what’s going on here. We may need extra overloads for DML scenarios where we only want a row count in the result.

Thanks for the clear report.

[jskeet]

Great, thanks. I’ll add a test to verify that it’s the case, and add documentation to that effect - as well as raising it with other languages.