`pac package deploy` times out when upgrade detected

When an existing solution need to be upgraded pac package deploy is timing out, e.g.

pac package deploy -p .\PackageDeploy.dll --logFile installog2.txt --logConsole

Gives the following logs.

Are there any timeout or config settings that can be used to tune? This tip from @ScottDurow doesn’t seem to work.

Thanks!

PackageDeployVerb Information: 8 : Message: ************** Solution IMPORT *****************
PackageDeployVerb Information: 8 : Message: Starting Processing Of Solution: My Amazing Solution(MyAmazingSolution), Version: 1.0.0.1, PackageVersion: 9.2
Beginning load of solution My Amazing Solution, version: 1.0.0.1.
PackageDeployVerb Information: 8 : Message: MyAmazingSolution : User Code Execution : OP=OverrideSolutionImportDecision : Status=Execute
PackageDeployVerb Information: 8 : Message: MyAmazingSolution : User Code Execution : OP=OverrideSolutionImportDecision : Status=Complete : Duration=00:00:00.0002229
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Processing Solution - Inbound version: 1.0.0.1 - Deployed Version 1.0.0.0, System Determined Import Action: Import
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - ImportSolution - Overwrite solution version My Amazing Solution with version 1.0.0.1
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Updating solution Version: 1.0.0.0 to Version : 1.0.0.1 - Importing Holding Solution - USING 7.2+ API
PackageDeployVerb Information: 8 : Message: MyAmazingSolution : User Code Execution : OP=PreSolutionImport : Status=Execute
PackageDeployVerb Information: 8 : Message: MyAmazingSolution : User Code Execution : OP=PreSolutionImport : Status=Complete : Duration=00:00:00.0002173
PackageDeployVerb Information: 8 : Message: ImportSolution - Async Supported = True
PackageDeployVerb Information: 8 : Message: DeleteAndPromote - Async Supported = True
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Executing Solution Import - UploadHoldingSolutionV2
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Fetching Data
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Import Progress is : 100
PackageDeployVerb Information: 8 : Message: MyAmazingSolution Solution Import Result = success - Duration: 00:01:16.9528092
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Updated solution Version: 1.0.0.0 to Version : 1.0.0.1
PackageDeployVerb Information: 8 : Message: MyAmazingSolution : User Code Execution : OP=RunSolutionUpgradeMigrationStep : Status=Execute
PackageDeployVerb Information: 8 : Message: MyAmazingSolution : User Code Execution : OP=RunSolutionUpgradeMigrationStep : Status=Complete : Duration=00:00:00.0003294
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Updating solution Version: 1.0.0.0 to Version : 1.0.0.1 - Deleting Holding Solution
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Executing Solution Import - PromoteHoldingSolutionV2
Microsoft.Xrm.Tooling.Connector.CrmServiceClient Error: 2 : ************ TimeoutException - DeleteAndPromote : Delete and Promote Solution to New version(DeleteAndPromoteSolutionRequest) |=> The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
Source  : mscorlib
Method  : HandleReturnMessage
Date    : 17/04/2022
Time    : 19:19:09
Error   : The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
Stack Trace     : Server stack trace:
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.Xrm.Sdk.IOrganizationService.Execute(OrganizationRequest request)
   at Microsoft.Xrm.Sdk.WebServiceClient.WebProxyClient`1.ExecuteAction[TResult](Func`1 action)
   at Microsoft.Xrm.Tooling.Connector.CrmServiceClient.CrmCommand_Execute(OrganizationRequest req, String errorStringCheck)
======================================================================================================================
Inner Exception Level 1 :
Source  : System.ServiceModel
Method  : ProcessGetResponseWebException
Date    : 17/04/2022
Time    : 19:19:09
Error   : The HTTP request to 'https://ipc-qa.crm11.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.2.47.7698' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout.
Stack Trace     : at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
======================================================================================================================
Inner Exception Level 2 :
Source  : System
Method  : GetResponse
Date    : 17/04/2022
Time    : 19:19:09
Error   : The operation has timed out
Stack Trace     : at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
======================================================================================================================

[... snip ...]

PackageDeployVerb Error: 2 : Message: MyAmazingSolution - Updating solution Version: 1.0.0.0 to Version : 1.0.0.1 - Deleting Holding Solution FAILED :  : ************ TimeoutException - DeleteAndPromote : Delete and Promote Solution to New version(DeleteAndPromoteSolutionRequest) |=> The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. => The HTTP request to 'https://myenv.crm11.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.2.47.7698' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout. => The operation has timed out[TerminalFailure] Failed to Execute Command - DeleteAndPromote : RequestID=ba768f36-0371-4c6e-b27c-678c8b0b675e : Delete and Promote Solution to New version(DeleteAndPromoteSolutionRequest) duration=00:02:00.0918443 ExceptionMessage = The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. => The HTTP request to 'https://myenv.crm11.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.2.47.7698' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout. => The operation has timed out
Source  : mscorlib
Method  : HandleReturnMessage
Date    : 19:19:09
Time    : 17/04/2022
Error   : The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
Stack Trace     : Server stack trace:
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.Xrm.Sdk.IOrganizationService.Execute(OrganizationRequest request)
   at Microsoft.Xrm.Sdk.WebServiceClient.WebProxyClient`1.ExecuteAction[TResult](Func`1 action)
   at Microsoft.Xrm.Tooling.Connector.CrmServiceClient.CrmCommand_Execute(OrganizationRequest req, String errorStringCheck)
======================================================================================================================
Inner Exception Level 1 :
Source  : System.ServiceModel
Method  : ProcessGetResponseWebException
Date    : 19:19:09
Time    : 17/04/2022
Error   : The HTTP request to 'https://myenv.crm11.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.2.47.7698' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout.
Stack Trace     : at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
======================================================================================================================
Inner Exception Level 2 :
Source  : System
Method  : GetResponse
Date    : 19:19:09
Time    : 17/04/2022
Error   : The operation has timed out
Stack Trace     : at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
======================================================================================================================
PackageDeployVerb Information: 8 : Message: errorDetails - ErrorCode  - -2146233083
PackageDeployVerb Information: 8 : Message: errorDetails - ErrorName  - NotProvided
PackageDeployVerb Information: 8 : Message: errorDetails - Message  - The request channel timed out while waiting for a reply after 00:01:59.9819968. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
PackageDeployVerb Information: 8 : Message: errorDetails - Type  - Application
PackageDeployVerb Information: 8 : Message: errorDetails - StatusCode  - 500
PackageDeployVerb Information: 8 : Message: errorDetails - Source  - PackageDeployer
PackageDeployVerb Information: 8 : Message: errorDetails - TraceText  - ......Server stack trace: ......   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)......   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)......   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)......   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)......   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)............Exception rethrown at [0]: ......   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)......   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)......   at Microsoft.Xrm.Sdk.IOrganizationService.Execute(OrganizationRequest request)......   at Microsoft.Xrm.Sdk.WebServiceClient.WebProxyClient`1.ExecuteAction[TResult](Func`1 action)......   at Microsoft.Xrm.Tooling.Connector.CrmServiceClient.CrmCommand_Execute(OrganizationRequest req, String errorStringCheck)
PackageDeployVerb Information: 8 : Message: errorDetails - ErrorCode  - -2146233083
PackageDeployVerb Information: 8 : Message: errorDetails - ErrorName  - NotProvided
PackageDeployVerb Information: 8 : Message: errorDetails - Message  - The HTTP request to 'https://myenv.crm11.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.2.47.7698' has exceeded the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout.
PackageDeployVerb Information: 8 : Message: errorDetails - Type  - Application
PackageDeployVerb Information: 8 : Message: errorDetails - StatusCode  - 500
PackageDeployVerb Information: 8 : Message: errorDetails - Source  - PackageDeployer
PackageDeployVerb Information: 8 : Message: errorDetails - TraceText  -    at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)......   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)......   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
PackageDeployVerb Information: 8 : Message: errorDetails - ErrorCode  - -2146233079
PackageDeployVerb Information: 8 : Message: errorDetails - ErrorName  - NotProvided
PackageDeployVerb Information: 8 : Message: errorDetails - Message  - The operation has timed out
PackageDeployVerb Information: 8 : Message: errorDetails - Type  - Application
PackageDeployVerb Information: 8 : Message: errorDetails - StatusCode  - 500
PackageDeployVerb Information: 8 : Message: errorDetails - Source  - PackageDeployer
PackageDeployVerb Information: 8 : Message: errorDetails - TraceText  -    at System.Net.HttpWebRequest.GetResponse()......   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
PackageDeployVerb Information: 8 : Message: RaiseFailEvent - update progress with fail event
PackageDeployVerb Information: 8 : Message: ************** BEGIN Queuing UpdateRibbonClientMetadata Job *****************
Beginning queueing up UpdateRibbonClientMetadata job.
PackageDeployVerb Information: 8 : Message: Trying to queue up UpdateRibbonClientMetadata Job.
PackageDeployVerb Information: 8 : Message: Time taken for queuing the ribbon job: 00:00:00.3296308
PackageDeployVerb Information: 8 : Message: ************** UpdateRibbonClientMetadata Job QUEUED *****************
PackageDeployVerb Information: 8 : Message: ****** PACKAGE DEPLOYMENT PROCESS COMPLETED. Result:FAILED  Duration:00:03:19.9664819 ******
PackageDeployVerb Information: 8 : Message: MyAmazingSolution - Pausing for 5 seconds before resuming operation. Retry# 0
Microsoft PowerPlatform CLI
Version: 1.14.2+g7205df7

Error: Updating solution My Amazing Solution version 1.0.0.0 to version 1.0.0.1: Deleting Holding solution.

Usage: pac package deploy [--logFile] [--logConsole] --package

  --logFile                   Log file path (alias: -lf)
  --logConsole                Output log to console (alias: -c)
  --package                   path to a package dll or zip file with a package (alias: -p)

[davidjenni]

@filcole the August refresh for pac CLI will initialize the PackageDeployer API to use async operations for solution import and deleteAndPromote, if those are enabled on the target environment. ETA: mid October

[MattB-msft]

@filcole ,
as @davidjenni mentioned here:

Here’s a tentative workaround/mitigation: edit the pac.exe.config that is located next to where you launch the pac.exe from (if in doubt, call where pac.exe in cmd or gcm pac.exe in PS/PSCore)

Add these 2 settings:

  <appSettings>
    <add key="UseAsyncForSolutionDeleteAndPromote" value="True" />
    <add key="UseAsyncForSolutionImport" value="True" />
  </appSettings>

This will tell the PD system to use Async for SolutionImport and Deleteandpromote by default. You can also override it in the package definition itself as well.

to do that, in the Inputconfig.xml file in your package, you add requiredimportmode=“async” to your solution reference. That will ask PD to attempt to use Async if available on the target environment.

for example:

<solutions>
    <configsolutionfile solutionpackagefilename="foo_1_0_0_0_managed.zip" overwriteunmanagedcustomizations="false" publishworkflowsandactivateplugins="true" missingdependencybehavior="fault" requiredimportmode="async"  />
  </solutions>