Glacial performance of package generation in force:source:deploy

Summary

We are attempting to migrate our large project repository (3000 metadata files) to source format and use force:source:deploy to continue to deploy to our fleet of UAT, QA and Packager orgs (none of which are scratch orgs) after the conversion. Historically, we use the Ant migration tool and a deploy takes about 3 minutes. We’re now finding that force:source:deploy of the same project takes 15-17 minutes. 10-12 of those minutes are spent entirely client side building the Zip file that gets sent up to the MDAPI for deployment

I see my CPU pegged for the entire 10 minutes (a single CPU on a quad core machine) with no indication of progress

Note: This ticket is not about incremental deployments of only changed source files – this is about deploying the entire project as part of a CI build pipeline.

Steps To Reproduce:

• Convert a large MDAPI project with about 3000 metadata components (classes, VF pages, static resources, SObjects, etc) to source format using force:mdapi:convert
• Try to deploy the converted source to a DE or EE org using force:source:deploy
• Expected: the duration of the deployment should be identical to the duration of using the Force.com Migration Tool (ant) to do a deployment
• Actual: The deployment takes 5x as long. It takes 10-12 minutes to build the package for deployment to the MDAPI

27783668 bytes written to C:\Users\esposito\AppData\Local\Temp\sdx_sourceDeploy_1568206314980.zip using 3087.629ms
Deploying C:\Users\esposito\AppData\Local\Temp\sdx_sourceDeploy_1568206314980.zip...

I tried gathering additional debug information using --verbose but this is the only thing written to the console of note. Is there a way I can troubleshoot further what’s going on during this 10 minute black hole?

SFDX CLI Version(to find the version of the CLI engine run sfdx --version):

sfdx --version
sfdx-cli/7.20.1-d88ae4707c win32-x64 node-v10.15.3

SFDX plugin Version(to find the version of the CLI plugin run sfdx plugins --core)

sfdx plugins --core
@oclif/plugin-commands 1.2.2 (core)
@oclif/plugin-help 2.2.1 (core)
@oclif/plugin-not-found 1.2.2 (core)
@oclif/plugin-plugins 1.7.8 (core)
@oclif/plugin-update 1.3.9 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/sfdx-trust 3.0.5 (core)
analytics 1.2.1 (core)
etcopydata 0.4.4
generator 1.1.1 (core)
salesforcedx 46.10.2
├─ force-language-services 46.14.0
└─ salesforce-alm 46.13.0

sfdx-cli 7.20.1 (core)

OS and version: Windows 10 Pro

[daveespo]

Alright, I’ve done more debugging on this than I care to admit. I upgraded my CLI to the latest plugins so now I’m

@oclif/plugin-commands 1.2.3 (core)
@oclif/plugin-help 2.2.1 (core)
@oclif/plugin-not-found 1.2.3 (core)
@oclif/plugin-plugins 1.7.8 (core)
@oclif/plugin-update 1.3.9 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/sfdx-trust 3.0.5 (core)
analytics 1.2.1 (core)
etcopydata 0.4.4
generator 1.1.1 (core)
salesforcedx 46.13.1
├─ force-language-services 46.18.0
└─ salesforce-alm 46.17.0

sfdx-cli 7.23.1 (core)

I cracked open the source code and have tracked this down to the implementation of _doLocalDelete in SourceDeployApi (salesforce-alm/dist/lib/source/sourceDeployApi.js)

I doctored up the method and the major tax is the call to PathUtils.cleanEmptyDirs

    async _doLocalDelete(ases) {
    	l.error('Begin _doLocalDelete()');
    	const a = Date.now();
        this.tmpBackupDeletions = await SourceUtil.createOutputDir('sourceDelete');
        ases.forEach((ase) => {
        	l.error(ase.getKey());
            ase
                .getPendingDeletedWorkspaceElements()
                .forEach(we => fsExtra.copySync(we.getSourcePath(), path.join(this.tmpBackupDeletions, path.basename(we.getSourcePath()))));
            l.error('before commitDeletes()');
            ase.commitDeletes([]);
            l.error('after commitDeletes(); calling cleanEmptyDirs(' + path.dirname(ase.getMetadataFilePath()) + ')');
            PathUtils.cleanEmptyDirs(path.dirname(ase.getMetadataFilePath()));
            l.error('after cleanEmptyDirs');
        });
        l.error('_doLocalDelete total duration: ' + (Date.now() - a) + 'ms');
    }

cleanEmptyDirs is incredibly inefficient because it does a directory listing on the directory containing the metdata type (ex. classes, objects, staticresources, etc) to find empty subdirectories. For directories with a large number of metadata files (our staticresources directory has 1000 files in it), it lists the entire directory to find any subdirectories. As you can see by this debug output, it’s taking over a second per metadata component. This adds up to 677 seconds or about 11 minutes in our project.

_doLocalDelete total duration: 677802ms

ERROR 2019-09-11T21:14:39.461Z:  StaticResource__pasadenaLayout
ERROR 2019-09-11T21:14:39.462Z:  before commitDeletes()
ERROR 2019-09-11T21:14:39.462Z:  after commitDeletes(); calling cleanEmptyDirs(C:\Users\esposito\Documents\PatronManager\patronticket-sfdx-3\force-app\main\default\staticresources)
ERROR 2019-09-11T21:14:40.502Z:  after cleanEmptyDirs
ERROR 2019-09-11T21:14:40.504Z:  StaticResource__pasadenaLayoutStyles
ERROR 2019-09-11T21:14:40.505Z:  before commitDeletes()
ERROR 2019-09-11T21:14:40.505Z:  after commitDeletes(); calling cleanEmptyDirs(C:\Users\esposito\Documents\PatronManager\patronticket-sfdx-3\force-app\main\default\staticresources)
ERROR 2019-09-11T21:14:41.523Z:  after cleanEmptyDirs
ERROR 2019-09-11T21:14:41.525Z:  StaticResource__PasadenaTicketTheme
ERROR 2019-09-11T21:14:41.526Z:  before commitDeletes()
ERROR 2019-09-11T21:14:41.527Z:  after commitDeletes(); calling cleanEmptyDirs(C:\Users\esposito\Documents\PatronManager\patronticket-sfdx-3\force-app\main\default\staticresources)
ERROR 2019-09-11T21:14:42.600Z:  after cleanEmptyDirs
ERROR 2019-09-11T21:14:42.602Z:  StaticResource__pasadenaV2Gallery
ERROR 2019-09-11T21:14:42.602Z:  before commitDeletes()
ERROR 2019-09-11T21:14:42.603Z:  after commitDeletes(); calling cleanEmptyDirs(C:\Users\esposito\Documents\PatronManager\patronticket-sfdx-3\force-app\main\default\staticresources)
ERROR 2019-09-11T21:14:43.606Z:  after cleanEmptyDirs

If I comment out the cleanEmptyDirs call, it reduces the _doLocalDelete to TWO POINT FIVE seconds:

_doLocalDelete total duration: 2570ms

This 11 minute tax is basically a blocker for our conversion to source format. Can a pair of engineer eyes take a look a this and see if there’s a more efficient way to track the empty directory cleanup?

I experimented with moving the cleanEmptyDirs outside of the loop over the ASEs and that seems to be the magic trick

[daveespo]

Can I assume this was released to the production channel? I’m now seeing way better sfdx force:source:deploy times. The build of the zip file now takes about 48 seconds where it previously took 11 minutes!