All package uploads are broken

Currently all package uploads are failing with the following traceback:

anaconda_upload is not set.  Not uploading wheels: []
+ upload_or_check_non_existence /recipe_root conda-forge --channel=main
Using Anaconda API: https://api.anaconda.org
Using Anaconda API: https://api.anaconda.org
[ERROR] 
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 102, in add_package
    aserver_api.package(username, package_name)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 288, in package
    self._check_response(res)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 216, in _check_response
    raise ErrCls(msg, res.status_code)
binstar_client.errors.NotFound: ('"whitenoise" could not be found', 404)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/anaconda", line 6, in <module>
    sys.exit(binstar_client.scripts.cli.main())
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/scripts/cli.py", line 88, in main
    description=__doc__, version=version)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/scripts/cli.py", line 66, in binstar_main
    return args.main(args)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 242, in main
    args=args)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 179, in upload_package
    add_package(aserver_api, args, username, package_name, package_attrs, package_type)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/commands/upload.py", line 126, in add_package
    license_family=package_attrs.get('license_family')
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 356, in add_package
    self._check_response(res)
  File "/opt/conda/lib/python3.6/site-packages/binstar_client/__init__.py", line 216, in _check_response
    raise ErrCls(msg, res.status_code)
binstar_client.errors.Unauthorized: ('Authorization token is no longer valid', 401)
Traceback (most recent call last):
  File "/opt/conda/bin/upload_or_check_non_existence", line 122, in <module>
    main()
  File "/opt/conda/bin/upload_or_check_non_existence", line 111, in main
    upload(cli, meta, owner, channel)
  File "/opt/conda/bin/upload_or_check_non_existence", line 62, in upload
    env=os.environ)
  File "/opt/conda/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['anaconda', '--quiet', '-t', 'binstar.token', 'upload', '/feedstock_root/build_artefacts/linux-64/whitenoise-3.3.0-py27_0.tar.bz2', '--user=conda-forge', '--channel=main']' returned non-zero exit status 1.
+ exit 1
Exited with code 1

I’m not sure if this is a problem with conda-forge or with the anaconda platform.

[jakirkham]

Thanks for opening this @ngoldbaum. Will try to keep the discussion about this problem in this issue to make it easy to find info and to avoid having the discussion get too dispersed. Though will cross link this issue from other discussions, gitter, the mailing list, etc. to get the info out there.

What happened: We discovered recently that the Anaconda token was exposed in some recent CI logs. As soon as this issue was discovered, fixed the code responsible for the vulnerability, attempted to pull any associated logs (but were unable to), and pulled the token. As this token is responsible for uploads from all feedstocks, this has caused any recent uploads from feedstocks to fail. Further any new feedstocks created by staged-recipes were DOA due to this issue.

Action Plan: At our dev meeting earlier today, we discussed this issue and outlined the following plan. First generate a new token. Second change out the token at staged-recipes so that any new feedstocks created will not suffer from this issue (otherwise they will pile up with the existing broken feedstocks). Third for any feedstocks in need of an immediate update, readd the recipe through staged-recipes. Fourth write a script to update the remaining feedstocks focusing on those that have been recently updated first.

Qualification for near term fix: To elaborate a bit more on the second point (as that seems relevant to you @ngoldbaum and likely others who will find this issue soon), this should only be done if you are trying to get a release out today or if you have tried and failed to make a release in the last 24hrs. As for recipes that have been recently converted by staged-recipes, we will try to handle these directly ourselves (core and staged-recipes teams) for a number of reasons including to make sure we have gotten them all, to avoid confusion for new contributors, and to avoid stressing the system. If you do not fall in this category, please do wait for the mid term fix to be applied to your feedstock. If you know a release will be coming out soon, please raise an issue on your feedstock with a link to this issue and ping @conda-forge/core and we will try to move your feedstock up in the midterm queue.

Near term: If you do fall into this category of having tried to make release from a feedstock, please simply copy the recipe from your feedstock’s master branch and place it in staged-recipes again under the recipes directory. Ensure that your recipe directory name uses your package name. Please make sure to include [ci skip] [skip ci] in the commit message to try and avoid having CIs run on these recipes. In your PR title, please write "Readd <package name>" where <package name> will be the package in question. Also in the PR description, please include that you are readding your recipe to get a new token in the feedstock and include a link to this issue. Doing all of this should help ensure that we are able to find your PR and merge it in a timely manner.

Mid term: For everything else that is broken, but is not in need of an immediate release, we will be writing a script to send PRs to affected feedstocks in small batches. These will update the token and re-render feedstocks. Priority will be given to feedstocks that have been changed recently.

Long term: We will begin discussions at dev meetings on how to better handle and manage uploads in the future to minimize the fallout from similar issues (e.g. expired tokens).

As a general reminder, please keep in mind that we were all affected negatively by this issue. So it will take some time to get everything back in working order for everyone. That said, we are eager to do exactly that. Thanks for your understanding and cooperation.

cc @conda-forge/core @conda-forge/staged-recipes

Edit: Added a link to staged-recipes.

[nicoddemus]

Thanks @jakirkham for the update! I’m sure all conda-forge users appreciate your dedication and efforts in keeping everything running smoothly! 👍