Documentation generation is flaky

Describe the bug When running dokka, some files are not generated. This can apparently happen to any file, though it seems orders of magnitude more common for index.html and other files near the root of the output hierarchy. I suspect that this is due to file lock contention by the output writer.

We first discovered this by adding code that enforces that all expected files in our dackka integration tests (“goldens”) were actually generated (previously we had only enforced that all generated files were expected).

This seems like it would be a huge problem for us and for you, because it could result in missing files in developer.android.com/kotlinlang.org (depending on how your update process works and how it removes files that are intended to cease being generated from one version to the next.)

From a test and a rough estimate, out of 130,000 files generated by running unmodified upstream dokka on androidx framework/support, 400 files were generated in the second run that were not generated in the first. Around a quarter of these (100 files) were index.html, representing a flake rate for index.html of around 20%, and for other files of 0.2%.

• Dokka version: 1.7.10

Replication process: -Copy frameworks/support to an upstream-dokka checkout. –Delete compose/ui/ui-test/src/androidAndroidTest because dokka fails if you don’t (throws in org.jetbrains.kotlin.types.expressions–looks like the compiler). -run ./gradlew shadowJar in the upstream checkout. -run java -jar ./runners/cli/build/libs/dokka-cli-1.7.10-SNAPSHOT.jar -pluginsClasspath "./integration-tests/cli/build/libs/fat-base-plugin-1.7.10-SNAPSHOT.jar" -sourceSet "-src ./frameworks/support/" in the upstream checkout. -add dokka/root to git or otherwise copy it for later comparison. -run java -jar ./runners/cli/build/libs/dokka-cli-1.7.10-SNAPSHOT.jar -pluginsClasspath "./integration-tests/cli/build/libs/fat-base-plugin-1.7.10-SNAPSHOT.jar" -sourceSet "-src ./frameworks/support/" again. -notice that there is a diff (e.g. git status) between the docs generated by each run. I found roughly 400 out of 130,000 .html files that were generated by the second run but not by the first.

[owengray-google]

I notice that you have no test for written files, only for files stored in the FileWriter and nodes in the model, so it does look like the issue is after the tree in the FileWriter is generated.

[owengray-google]

It looks like the pages.json and flaky html files here are two complete different issues.

The first one is due to us lumping all multiplatform files into a single sourceSet at the moment. The second one is due to dokka trying to produce type links for generic type params like T, and looking in very strange places all across the unimported parts of the classpath, which resolves flakily.