Values assigned to `exclude` and/or `include` parameters not being consumed by `download`, `synccopy`, or `upload` commands when using a YAML configuration file
See original GitHub issueProblem Description
The exclude
and include
parameters for the download
, synccopy
, and upload
commands do not work when using a YAML configuration file.
Azure blobxfer parameters output
============================================
Azure blobxfer parameters
============================================
blobxfer version: 1.9.4
platform: Linux-4.19.128-microsoft-standard-x86_64-with
components: CPython=3.7.5-64bit azstor.blob=2.1.0 azstor.file=2.1.0 crypt=2.8 req=2.22.0
transfer direction: Azure -> Azure
workers: disk=0 xfer=24 md5=0 crypto=0
log file: None
dry run: False
resume file: None
timeout: connect=10 read=200 max_retries=1000
source mode: StorageModes.Auto
dest mode: StorageModes.Auto
server side copy: True
skip on: fs_match=False lmt_ge=False md5=True
delete: extraneous=True only=False
overwrite: True
recursive: True
rename single: False
strip components: 0
access tier: None
============================================
Steps to Reproduce
-
Create
$PWD/config/config.yaml
file -
Add the following configuration to
$PWD/config/config.yaml
:version: 1 options: log_file: blobxfer.log progress_bar: true verbose: true azure_storage: endpoint: core.windows.net accounts: devstoreaccount1: XXXXXXXXXXXXX synccopy: - source: - devstoreaccount1: src destination: - devstoreaccount1: dest include: - "*" exclude: - "*.csv" options: mode: auto dest_mode: auto access_tier: null delete_extraneous_destination: true delete_only: false overwrite: true recursive: true rename: false server_side_copy: true skip_on: filesize_match: false lmt_ge: false md5_match: true
-
Upload files matching both
include
andexclude
file matching patterns to the source storage account(s) defined in$PWD/config/config.yaml
config -
Run the following command:
docker run -v $PWD/config:/config mcr.microsoft.com/blobxfer synccopy --config /config/config.yaml
Expected Results
Only files that match the include
file matching pattern, and not those matching the exclude
file matching pattern, be copied from devstoreaccount1: src
container to the devstoreaccount1: dest
container.
Actual Results
All files from the devstoreaccount1: src
container, including those explicitly defined to be excluded via the exclude
parameter, are copied to the devstoreaccount1: dest
container
Additional Logs
INFO - blobxfer start time: 2021-02-04 00:28:08.993581+00:00
DEBUG - spawning 24 transfer threads
DEBUG - 5 remote files to sync, waiting for copy completion of approx. 0.0000 MiB
DEBUG - attempting to delete extraneous blobs/files from: devstoreaccount1;core.windows.net;publish
INFO - deleted 0 extraneous blobs/files
INFO - elapsed copy time and throughput of 0.0000 GiB: 0.862 sec, 0.0000 Mbps (0.000 MiB/sec)
INFO - blobxfer end time: 2021-02-04 00:28:10.566358+00:00 (elapsed: 1.573 sec)
Additional Comments
The issue appears to be a result of using type multiple
for the _exclude_option
and _include_option
objects on cli.py, in conjunction with the condition in the _merge_setting
function on settings.py. For reference, When passing a default
with multiple=True
, the default value must be a list or tuple, otherwise it will be interpreted as a list of single characters.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:4
- Comments:5 (1 by maintainers)
Top GitHub Comments
Thanks for submitting your PR and apologies for the delay. After doing more tests on your PR change, there is a regression on bool values. A fix will be made a top your PR to account for this.
Thank you @alfpark