Running BulkCopy without parameters

Why BulkCopy run without parameters?

Simple insert generates SQL with parameters:

db.Insert(new SubmissionFile() { Status = “Uploaded2”});

INSERT INTO submission_files ( status, filename, created, created_by ) VALUES ( :Status, :Filename, :Created, :CreatedBy )

But BulkCopy generates SQL without parameters:

db.BulkCopy<SubmissionFile>(new List<SubmissionFile>() { new SubmissionFile() { Status = “Uploaded” }, new SubmissionFile() { Status = “Uploaded2”} });

INSERT INTO submission_files ( status, filename, created, created_by ) VALUES ('Uploaded','\\',NULL,'0001-01-01'::date,NULL,'0001-01-01'::date), ('Uploaded2','\\',NULL,'0001-01-01'::date,NULL,'0001-01-01'::date)

So, does this implementation has a relation to productivity or is it some else? Could I use bulk operation with parameters?

[AndrePScope]

Hi @sdanyliv ! I do not how, but I missed your comment from Mar 26. I use PostgreSQL. I tried to use BulkCopyType.MultipleRows. The result is the same: SQL generates without parameters, the values included in SQL directly.

In fact, the case when necessary to insert some limited (not thousands) number of rows is typical. So the possibility to insert data via BulkCopy with parameters will be really nice.

[to11mtm]

I have some notes on this while I started looking into #2960

I think this is not difficult, but we should do some refactoring to make it right. We can probably handle this in multiple phases:

Phase 1

• Pass in ‘correct’ maxParameters and maxSqlLength into MultipleRowsCopyHelper(Async). We may need to refactor methods like protected BulkCopyRowsCopied MultipleRowsCopy1(MultipleRowsHelper helper, IEnumerable source) so that they can have the parameter length passed in, since they pass private funcs into MultipleRowsCopyHelper(Async) for begin/add/finish.

  • SQL Server: 2099 (technically 2100, but I believe the way ADO Provider works, query text is a parameter)
  • Oracle: 32767
  • Postgres: 32767
  • SQLite: 999 (I think newer versions may take more, but this is pretty safe)

• Provide a Switch to prefer parameters on BulkCopyOptions (since it is a behavior change, should we default to OFF?)

• MultipleRowsHelper right now only checks whether we have ‘overflowed’ on string or parameter length -after- we have added the row. We should consider something like below, where we track the parameter/length -before- we add the row, and if it overflows, back-off to complete the statement.

//in BasicBulkCopy.cs
protected static async Task<BulkCopyRowsCopied> MultipleRowsCopyHelperAsync<T>
//....
await foreach (var item in source.ConfigureAwait(Common.Configuration.ContinueOnCapturedContext).WithCancellation(cancellationToken))
			{
				helper.LastRowStringStartIndex    = helper.StringBuilder.Length;
				helper.LastRowParameterStartIndex = helper.ParameterIndex;
				addFunction(helper, item!, from);

				if (helper.CurrentCount >= helper.BatchSize || helper.Parameters.Count > maxParameters || helper.StringBuilder.Length > maxSqlLength)
				{
					helper.Parameters.RemoveRange(helper.LastRowParameterStartIndex, helper.LastRowParameterStartIndex-helper.Parameters.Count);
					helper.StringBuilder.Length =
						helper.LastRowStringStartIndex;
					finishFunction(helper);
					if (!await helper.ExecuteAsync(cancellationToken).ConfigureAwait(Common.Configuration.ContinueOnCapturedContext))
						return helper.RowsCopied;
				}
			}

Phase 2

Phase 1 provides us parameterized inserts, but is a naive implementation; every time, we will still be re-building the parameters in the INSERT statement, and creating new parameters.

• In an Ideal state, I think we would be better to re-use parameters as long as data providers play nice with that.
• I’m imagining something where after the first pass, we are able to track the indexes to see whether we are in an ‘after first insert’ scenario in whatever addFunction is passed in above snippet, and decide whether we need to append to the string or can re-use existing string and/or params.