Troubleshooting Poor Performance in SaveChangesAsync

Ask a question

How do you troubleshoot the reason why SaveChangesAsync is running slow?

FYI:

• I just upgraded to EF7.0 but I’m not sure that is related.
• The app and SQL server are on the same machine (my PC)
• My PC is an i9-9900k w/ 64gb of ram and blazing fast SSDs.

My app spins up multiple tasks whose job it is to read records from a database, send the record to a web service, and then update the associated record with information returned from the service. At a high level, my code look like this:

        protected async Task<TJson?> InjectAndTrackAsync(TImportJob Input, CancellationToken Token) {
            var ret = await ProcessItemAsync(Input, InjectItemAsync, Token)
                .DefaultAwait()
                ;
            
            if (ShouldSaveChanges(Input)) {

                using var NewContext = this.ContextFactory();
                Attach(NewContext, Input);

                await NewContext.SaveChangesAsync(Token)
                    .DefaultAwait()
                    ;

            }

            return ret;
        }

        protected void Attach<T>(MigrationDbContext Context, T Entity) where T : class {
            Context.ChangeTracker.TrackGraph(Entity, x => {
                x.Entry.State = Microsoft.EntityFrameworkCore.EntityState.Modified;
            });
        }

Each item gets its own DB context that is used to update the single row in the database.

Sometimes SaveChanges runs quickly but other times, I see terrible performance like this:

SaveChangesAsync took 00:00:05.5997678
SaveChangesAsync took 00:00:03.2859983
SaveChangesAsync took 00:00:01.8060537
SaveChangesAsync took 00:00:03.8482817
SaveChangesAsync took 00:00:02.1440612
SaveChangesAsync took 00:00:03.1174782
SaveChangesAsync took 00:00:03.1120000

Seriously - 3.5 seconds to update a single row in my database.

SQL Profiler shows queries like this:

exec sp_executesql N'SET IMPLICIT_TRANSACTIONS OFF;
SET NOCOUNT ON;
UPDATE [__M_Communications_Participating_Contacts] SET [Comment] = @p0, [Id] = @p1, [Batch_Number] = @p2, [Final_Created_At] = @p3, [Final_Created_By] = @p4, [Final_Created_OnBehalfOf] = @p5, [Final_DisplayOrder] = @p6, [Final_Extra_Content] = @p7, [Final_Raw_Content] = @p8, [Final_Role] = @p9, [Final_Updated_At] = @p10, [Final_Updated_By] = @p11, [Final_Communication_Id] = @p12, [Final_Contact_Id] = @p13, [Original_Created_At] = @p14, [Original_Created_By] = @p15, [Original_Created_OnBehalfOf] = @p16, [Original_DisplayOrder] = @p17, [Original_Extra_Content] = @p18, [Original_Raw_Content] = @p19, [Original_Role] = @p20, [Original_Updated_At] = @p21, [Original_Updated_By] = @p22, [Original_Communication_Id] = @p23, [Original_Contact_Id] = @p24, [Result_DisplayNumber] = @p25, [Result_Id] = @p26, [Status_Date] = @p27, [Status_Description] = @p28, [Status_DisplayNumber] = @p29
OUTPUT 1
WHERE [Row] = @p30;
',N'@p30 bigint,@p0 nvarchar(4000),@p1 nvarchar(450),@p2 bigint,@p3 datetimeoffset(7),@p4 nvarchar(4000),@p5 nvarchar(4000),@p6 bigint,@p7 nvarchar(4000),@p8 nvarchar(4000),@p9 nvarchar(4000),@p10 datetimeoffset(7),@p11 nvarchar(4000),@p12 nvarchar(4000),@p13 nvarchar(4000),@p14 datetimeoffset(7),@p15 nvarchar(4000),@p16 nvarchar(4000),@p17 bigint,@p18 nvarchar(4000),@p19 nvarchar(4000),@p20 nvarchar(4000),@p21 datetimeoffset(7),@p22 nvarchar(4000),@p23 nvarchar(4000),@p24 nvarchar(4000),@p25 nvarchar(4000),@p26 nvarchar(4000),@p27 datetimeoffset(7),@p28 nvarchar(4000),@p29 nvarchar(4000)',@p30=36420,@p0=N'',@p1=N'494632459 --- To --- Person --- 1243401544 --- 0',@p2=10,@p3=NULL,@p4=N'',@p5=N'',@p6=0,@p7=N'',@p8=N'',@p9=N'To',@p10=NULL,@p11=N'',@p12=N'494632459',@p13=N'1243401544',@p14=NULL,@p15=N'',@p16=N'',@p17=0,@p18=N'',@p19=N'',@p20=N'To',@p21=NULL,@p22=N'',@p23=N'494632459',@p24=N'1243401544',@p25=N'2037980301',@p26=N'2037980301',@p27='2022-11-13 06:25:43.7570498 -06:00',@p28=N'',@p29=N''

Which looks fine to me (FYI ‘ROW’ is actually the primary key of the table, not Id).

If I pause my app and look at threads, I see this:

Which I don’t think is good but I’m not sure what to do about it.

When I look at Tasks in VS, I see this:

What would be my next step to determine why things are running slow and boost them?

Include provider and version information

EF Core version: 7.0 Database provider: Microsoft.EntityFrameworkCore.SqlServer Target framework: .NET 7.0 Operating system: Win11 IDE: Visual Studio 2022 17.4

[roji]

See my suggestion in https://github.com/dotnet/SqlClient/issues/1837#issuecomment-1312736476, first thing should probably be to isolate whether the long time occurs in SQL Server or not.

[TonyValenti]

@roji - Actually, I might not be totally out of the woods. After letting things run for a few, this is what I’m seeing:

SaveChanges took 00:00:06.6747568
SaveChanges took 00:00:04.3304368
SaveChanges took 00:00:04.3286715
SaveChanges took 00:00:06.6666748
SaveChanges took 00:00:06.6736879
SaveChanges took 00:00:06.6659064
SaveChanges took 00:00:10.9945973
SaveChanges took 00:00:00.0046404  <--Fast
SaveChanges took 00:00:00.0031156  <--Fast
SaveChanges took 00:00:10.9937690

Any ideas on what I should look into next?