question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Values assigned to `exclude` and/or `include` parameters not being consumed by `download`, `synccopy`, or `upload` commands when using a YAML configuration file

See original GitHub issue

Problem Description

The exclude and include parameters for the download, synccopy, and upload commands do not work when using a YAML configuration file.

Azure blobxfer parameters output

============================================
         Azure blobxfer parameters
============================================
         blobxfer version: 1.9.4
                 platform: Linux-4.19.128-microsoft-standard-x86_64-with
               components: CPython=3.7.5-64bit azstor.blob=2.1.0 azstor.file=2.1.0 crypt=2.8 req=2.22.0
       transfer direction: Azure -> Azure
                  workers: disk=0 xfer=24 md5=0 crypto=0
                 log file: None
                  dry run: False
              resume file: None
                  timeout: connect=10 read=200 max_retries=1000
              source mode: StorageModes.Auto
                dest mode: StorageModes.Auto
         server side copy: True
                  skip on: fs_match=False lmt_ge=False md5=True
                   delete: extraneous=True only=False
                overwrite: True
                recursive: True
            rename single: False
         strip components: 0
              access tier: None
============================================

Steps to Reproduce

  1. Create $PWD/config/config.yaml file

  2. Add the following configuration to $PWD/config/config.yaml:

    version: 1
    
    options:
      log_file: blobxfer.log
      progress_bar: true
      verbose: true
    
    azure_storage:
      endpoint: core.windows.net
      accounts:
        devstoreaccount1: XXXXXXXXXXXXX
    
    synccopy:
      - source:
          - devstoreaccount1: src
        destination:
          - devstoreaccount1: dest
        include:
          - "*"
        exclude:
          - "*.csv"
        options:
          mode: auto
          dest_mode: auto
          access_tier: null
          delete_extraneous_destination: true
          delete_only: false
          overwrite: true
          recursive: true
          rename: false
          server_side_copy: true
          skip_on:
            filesize_match: false
            lmt_ge: false
            md5_match: true
    
  3. Upload files matching both include and exclude file matching patterns to the source storage account(s) defined in $PWD/config/config.yaml config

  4. Run the following command:

    docker run -v $PWD/config:/config mcr.microsoft.com/blobxfer synccopy --config /config/config.yaml
    

Expected Results

Only files that match the include file matching pattern, and not those matching the exclude file matching pattern, be copied from devstoreaccount1: src container to the devstoreaccount1: dest container.

Actual Results

All files from the devstoreaccount1: src container, including those explicitly defined to be excluded via the exclude parameter, are copied to the devstoreaccount1: dest container

Additional Logs

INFO - blobxfer start time: 2021-02-04 00:28:08.993581+00:00
DEBUG - spawning 24 transfer threads
DEBUG - 5 remote files to sync, waiting for copy completion of approx. 0.0000 MiB
DEBUG - attempting to delete extraneous blobs/files from: devstoreaccount1;core.windows.net;publish
INFO - deleted 0 extraneous blobs/files
INFO - elapsed copy time and throughput of 0.0000 GiB: 0.862 sec, 0.0000 Mbps (0.000 MiB/sec)
INFO - blobxfer end time: 2021-02-04 00:28:10.566358+00:00 (elapsed: 1.573 sec)

Additional Comments

The issue appears to be a result of using type multiple for the _exclude_option and _include_option objects on cli.py, in conjunction with the condition in the _merge_setting function on settings.py. For reference, When passing a default with multiple=True, the default value must be a list or tuple, otherwise it will be interpreted as a list of single characters.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:4
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
alfparkcommented, Apr 16, 2021

Thanks for submitting your PR and apologies for the delay. After doing more tests on your PR change, there is a regression on bool values. A fix will be made a top your PR to account for this.

0reactions
SailingYYCcommented, Apr 20, 2021

Thank you @alfpark

Read more comments on GitHub >

github_iconTop Results From Across the Web

dvc.yaml Files | Data Version Control
Parameters. Parameters are simple key/value pairs consumed by the command code from a structured parameters file. They are defined per-stage in the params...
Read more >
Forget command-line arguments: use a config file | Medium
In this article, I'll show you ready-to-go examples of how moving your hyper-parameter selections, command-line arguments, and default options ...
Read more >
Configuring - Understanding values.yaml - IBM
Parameter Description Default Value license License agreement. Set true to accept the license. false env.timezone Timezone UTC arch Node Architecture amd64
Read more >
Using codemagic.yaml
yaml is a highly customizable configuration file for setting up your CI/CD pipeline with Codemagic. Configure all your workflows in a single ...
Read more >
2.1. YAML Guide - Common Workflow Language
YAML is a file format designed to be readable by both computers and humans. This guide introduces the features of YAML relevant when...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found