[BUG] GitLab - Large PRs have incomplete created_files, modified_files and deleted_files

Describe the bug The files listed under danger.git.*_files are incomplete - at least in GitLab. It seems only a limited amount of files are listed and possibly only the same number of files that GitLab will show in the diff view of the merge request.

GitLab shows 898+ files, danger-js only gives me 898 files, while a git diff --name-status origin/master...origin/my_branch|wc -l shows 2549 files.

This leads to a crucial pre-merge test not to be executed for many files that are actually contained.

To Reproduce Steps to reproduce the behavior:

1. Have a large merge request with 900+ files in GitLab.
2. Check danger.git.created_files or danger.git.modified_files or danger.git.deleted_files to contain all the files.

Expected behavior All files in the merge request are contained in danger.git.

Screenshots

This is what GitLab shows:

These are the counts of the danger.git API that I’ve printed out (0 added, 2 modified, 896 deleted):

And this is the expected count of files included in the merge request (3 added, 10 modified, 2528 deleted):

Your Environment

software	version
danger.js	11.0.2 (noticed with 10.6.6 then upgraded)
node	14.17.5
yarn	1.22.5
GitLab Enterprise	14.6.2
Operating System	alpine3.14

Additional context It is curious that the “898+” GitLab is showing matches so exactly to the “898” files that danger is giving me. Is there some pagination in the GitLab API that you are using and not pulling additional pages?

[fbartho]

@uncaught the DangerJS GitLab Provider is basically a plugin abstracting the operations that GitLab supports.

The internals of a provider are “private” to that provider, so if the GitLab-supporting-devs decide to use different API calls, or use appropriate CLI commands instead of using API calls, that’s totally a fair choice to balance between maintainability, consistency, and performance.

The goals are “correct”, followed by “fast enough” balanced with “easy enough to maintain”.

Remember:

• performance is often a non-obvious battle, if you’re making a performance change, you need to measure & justify the speed + memory + code tradeoffs
• Shelling out to a CLI command incurs the cost of ensuring that the CLI command 1. exists 2. Is at the right version 3. Has the right config 4. Has the right access to the project history
• Shelling out to any CLI commands also runs a security risk — user untrusted paths, data, potentially hostile actors, OS differences, etc

[orta]

Makes sense, the current implementation uses:

https://github.com/danger/danger-js/blob/1822fe124fbbd1d78ac43f3ee875685ef2b04392/source/platforms/gitlab/GitLabAPI.ts#L116-L125

Which doesn’t look like it is feasible to paginate: https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr-changes

Perhaps Danger should provide a warning when the number or changes are bigger than files changed?