setting secrets can't contain unescaped '$' on cli

Hi I found that I set secrets like

digdag secrets --local --set password=i_want_$50000
# or 
digdag secrets --local --set password # enter
password: i_want_$50000

and use it in dig file then these error occurs

java.lang.IllegalArgumentException: Illegal group reference
	at java.util.regex.Matcher.appendReplacement(Matcher.java:857)
	at io.digdag.util.UserSecretTemplate.replaceAll(UserSecretTemplate.java:64)
	at io.digdag.util.UserSecretTemplate.format(UserSecretTemplate.java:55)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$buildAccessor$1(GrantedPrivilegedVariables.java:62)
	at io.digdag.core.agent.GrantedPrivilegedVariables.get(GrantedPrivilegedVariables.java:81)
	at io.digdag.standards.operator.ShOperatorFactory.collectEnvironmentVariables(ShOperatorFactory.java:156)
	at io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:111)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:312)
	at io.digdag.cli.Run$OperatorManagerWithSkip.callExecutor(Run.java:694)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:254)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:137)
	at io.digdag.core.agent.LocalWorkspaceManager.withExtractedArchive(LocalWorkspaceManager.java:25)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:135)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:119)
	at io.digdag.cli.Run$OperatorManagerWithSkip.run(Run.java:676)
	at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:127)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

It can be avoided by setting with escaped form like this

digdag secrets --local --set password="i_want_\$50000"

but this is very useless, especially using masked input

[hiroyuki-sato]

Oh, I see, It seems a bug.

You can’t use $ character, until fix this issue.

Digdag: 0.9.20

Reproduce Dig file.

timezone: UTC

+test:
  sh>: env
  _env:
    PASSWORD: ${secret:password}

OK case

The password doesn’t contain $ character.

secrets --local --set password='i_want_50000'
digdag run -a fuga
...
PASSWORD=i_want_50000

NG Case

The password contains $ character.

digdag secrets --local --set password='i_want_$50000'
digdag run -a fuga
java.lang.IndexOutOfBoundsException: No group 5
	at java.util.regex.Matcher.start(Matcher.java:375)
	at java.util.regex.Matcher.appendReplacement(Matcher.java:880)
	at io.digdag.util.UserSecretTemplate.replaceAll(UserSecretTemplate.java:64)
	at io.digdag.util.UserSecretTemplate.format(UserSecretTemplate.java:55)
	at io.digdag.core.agent.GrantedPrivilegedVariables.lambda$buildAccessor$1(GrantedPrivilegedVariables.java:62)
	at io.digdag.core.agent.GrantedPrivilegedVariables.get(GrantedPrivilegedVariables.java:81)
	at io.digdag.standards.operator.ShOperatorFactory.collectEnvironmentVariables(ShOperatorFactory.java:156)
	at io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:111)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:312)
	at io.digdag.cli.Run$OperatorManagerWithSkip.callExecutor(Run.java:694)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:254)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:137)
	at io.digdag.core.agent.LocalWorkspaceManager.withExtractedArchive(LocalWorkspaceManager.java:25)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:135)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:119)
	at io.digdag.cli.Run$OperatorManagerWithSkip.run(Run.java:676)
	at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:127)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

[cafeal]

sorry for my short explanation 🙄

my dig file, foo.dig, is like below

_export:
   db:
      password: ${secret:aws.db.password}
+bar:
   sh>: python script.py
   _env:
      DB_PASSWORD: ${db.password}

DB_PASSWORD is used in script.py

then, I set my secrets and run digdag like:

$ digdag secrets --local --set aws.db.password # enter
$ password: ・・・・・・・・　　#=> i_want_$50000
$ digdag run --rerun foo.dig +bar

I use secrets for defining the environment variables through _export but I think it doesn’t matter