Conditionally symlink 'latest' directory

Description

Add a configuration setting for [scheduler] which defaults to true, which is used for conditionally calling FileProcessorHandler._symlink_latest_log_directory().

Use case / motivation

(I’m happy to submit PR for this if it’s not rejected!)

If base_log_folder resides on a file mount, backed by a storage service that doesn’t support symlinks (for us, Azure Blobs or Azure Files), the scheduler logs receive these lines every time the FileProcessorHandler class is instantiated: {file_processor_handler.py:126} WARNING - OSError while attempting to symlink the latest log directory.

Since this will never work for us, it’s a distraction for humans reading logs, and a tiny amount of futile extra work for the scheduler.

Unsure how many people this affects. I suppose an alternative for our specific case would be to create an Azure file handler, use local os disk for base_log_folder, and clear out local files older than n hours.

Related Issues

I could not find a related issue in Github.

[flolas]

For Azure Files as a workaround I solved adding mfsymlinks option on mount command. See: https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-problems

Edit: Only works in AKS using a storageclass

[chinwobble]

@chinwobble - what is the behaviour you observe? Can you paste some logs/errors you see?

We are using externally mounted azure file share to store our logs (which is expensive). We aren’t getting any errors. We use a datadog agent to scan the logs folder and ingest all our log files. The symlinking is causing the same files to be ingested twice.