Use ArgumentList when invoking native executables

Handling Parameter Binding in Native Executables

PowerShell generally provides a useful experience when working with native executables, but there are a number of issues:

• passing embedded quoted strings is very problematic
• passing an empty string is very problematic (mostly impossible)
• passing a string which looks like a ScriptBlock requires extra care

A command line such as: msiexec /i testdb.msi INSTALLLEVEL=3 /l* msi.log COMPANYNAME="Acme ""Widgets"" and ""Gizmos""" is received as: msiexec /i testdb.msi INSTALLLEVEL=3 /l* msi.log COMPANYNAME="Acme Widgets and Gizmos". A command line such as: msiexec /i testdb.msi INSTALLLEVEL=3 /l* msi.log COMPANYNAME="Acme ""Widgets"" and ""Gizmos.""" is received as: msiexec /i testdb.msi INSTALLLEVEL=3 /l* msi.log COMPANYNAME="Acme Widgets and Gizmos." which strips the embedded double quotes.

Current behavior does not allow empty strings to be passed, so the following is not possible:

• useradd -g 501 -u 1001 -p '' sam
• emacsclient -u ''

These behaviors should be supported as there are many scenarios where an empty string is required as a parameter value.

Handling Quotes

Windows and Non-Windows have divergent behavior with regard to quotes. While both Windows and Unix shells recognize "one two three" as a single string, Windows does not recognize single quotes ' as string designator. Unix shells (and PowerShell) has 2 types of strings; Expandable strings "$a" where $a is expanded with the value of the variable $a, and literal strings '$a’where the literal string$a` is passed.

In the case of Windows a string such as 'foo bar' is 2 tokens ('foo and bar') where Unix will see a single string foo bar.

Null vs Empty Strings

Most shells don’t have the same definition of null that PowerShell does. In bash, for example, commands are invoked via strings, so naturally, an empty string can be easily notated with '' or "". Similarly, in CMD.EXE an empty string may be passed with "".

However, the following should not result in empty strings being added:

• $null
• Reference to a variable which is unassigned
• Reference to a variable which is assigned the value $null
• When a collection wherein an element of the collection has the value $null

Non-null elements of a collection will be bound:

• @($null, 1, $null) will bind a single value 1 as a parameter value
• @($null, 1, $null, 2, $null) will bind 2 values (1 and 2) as parameter values
• @($null, 1, $null, '', 2) will bind 3 values (1, '', 2) as parameter values

examples:

PS> useradd -g 501 -u 1001 -p $null sam # error - "sam" is used as password and no username is passed
PS> $a = $null
PS> useradd -g 501 -u 1001 -p $a sam # error - "sam" is used as password and no username is passed
PS> $a = @()
PS> useradd -g 501 -u 1001 -p $a sam # error - "sam" is used as password and no username is passed
PS> $a = @($null)
PS> useradd -g 501 -u 1001 -p $a sam # error - "sam" is used as password and no username is passed
PS> $a = ''
PS> useradd -g 501 -u 1001 -p $a sam # no error - empty string is passed as password
PS> $a = @('')
PS> useradd -g 501 -u 1001 -p $a sam # no error - empty string is passed as password
PS> $a = @('',"sam")
PS> useradd -g 501 -u 1001 -p $a # no error - empty string is passed as password and same is passed as username value
PS> $a = @("-g",501,"-u",1001,"-p",'',"sam")
PS> useradd $a # no error
PS> $a = @("-g",501,"-u",1001,$null,$null,$null,$null,"-p",'',"sam")
PS> useradd $a # no error - nulls are not passed

Additional Examples

S> msiexec /i A:\Example.msi PROPERTY="Embedded ""Quotes"" White Space" # no error - embedded quotes are passed to native executable
PS> msiexec /i A:\Example.msi PROPERTY="Embedded White Space" # no error - embedded spaces are passed to native executable
PS> msiexec /i A:\Example.msi PROPERTY="" # no error - empty string is passed to native executable

Globbing Considerations

Globbing will need to be done where appropriate. This behavior is platform dependent, so on Windows systems, globbing is not performed but is provided on Linux and Mac systems. This is because most utilities on Windows do their own globbing, but on Linux and Mac globbing is done by the shell. The current behavior for globbing does this and needs to remain unchanged to ensure that the scenarios continue to work on each platform. When a glob fails, the string provided shall be sent to the application without alteration.

The following example shows the difference based on platform:

PS> # on Windows
PS> .\echoit.exe rm c:\tmp\dd\f*
Argument 1 <rm>
Argument 2 <c:\tmp\dd\f*>

# on Mac/Linux
PS> trace-command -pshost -name parameterbinding { /bin/ls /tmp/dd/f* }
DEBUG: 2021-02-08 17:12:03.3886 ParameterBinding Information: 0 : BIND NAMED native application line args [/bin/ls]
DEBUG: 2021-02-08 17:12:03.3887 ParameterBinding Information: 0 :     BIND argument [/tmp/dd/f1 /tmp/dd/f2]
DEBUG: 2021-02-08 17:12:03.3956 ParameterBinding Information: 0 : CALLING BeginProcessing
/tmp/dd/f1
/tmp/dd/f2
PS> trace-command -pshost -name parameterbinding { /bin/ls /tmp/dd/fff* }
DEBUG: 2021-02-08 17:12:23.0600 ParameterBinding Information: 0 : BIND NAMED native application line args [/bin/ls]
DEBUG: 2021-02-08 17:12:23.0601 ParameterBinding Information: 0 :     BIND argument [/tmp/dd/fff*]
DEBUG: 2021-02-08 17:12:23.0670 ParameterBinding Information: 0 : CALLING BeginProcessing
ls: /tmp/dd/fff*: No such file or directory

Unsupported or Requiring Alteration

Some elements may not be used as they represent PowerShell tokens.

• An embedded semi-colon ; is not allowed. PowerShell will parse this as a statement separator.
• An open curly-brace will be interpreted as the beginning of a scriptblock

for example:

msiexec /p msipatch.msp;msipatch2.msp /n {00000001-0002-0000-0000-624474736554} /qb

• This is not allowed because of the embedded ; which PowerShell will turn into 2 commands (; is a statement separator) this string must be quoted.
• This is also not supported because of the use of ScriptBlock syntax. PowerShell can not determine if the ScriptBlock is a command (or in this case a guid). To execute this command, quote the problematic strings or use --% after the executable.

msiexec /p 'msipatch1.msp;msipatch2.msp' /n '{00000001-0002-0000-0000-624474736554}' /qb

or

msiexec --% /p msipatch1.msp;msipatch2.msp /n {00000001-0002-0000-0000-624474736554} /qb

NB: The behavior for supporting ScriptBlocks as strings could be supported via an allow-list for known executables. This may be needed because the amount of GUIDs (with braces) is used in a number of both Windows and Non-Windows utilities. However, managing the list of utilities may be burdensome.

Improved Tracing

The parameter binding tracing code for native executables is not currently implemented which makes debugging issues when execution native applications very difficult. Tracing for current parameter binding and new behavior shall be provided. In the case of the old style, the path to the executable and the string which makes up the Arguments property of the StartInfo object shall be provided. For the new behavior, since the arguments are a list, each element of the list shall be presented. The following transcript shows how the tracing shall appear.

# new style - native arguments are bound to ArgumentList property
PS > trace-command -PSHOST -Name ParameterBinding { ~/echoit foo="bar ""blob"" bar" zap foo:bar:baz,bip,bar }
DEBUG: 2021-02-04 17:28:54.5674 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/echoit]
DEBUG: 2021-02-04 17:28:54.5674 ParameterBinding Information: 0 :     BIND cmd line arg [foo=bar "blob" bar] to position [0]
DEBUG: 2021-02-04 17:28:54.5675 ParameterBinding Information: 0 :     BIND cmd line arg [zap] to position [1]
DEBUG: 2021-02-04 17:28:54.5675 ParameterBinding Information: 0 :     BIND cmd line arg [foo:bar:baz,bip,bar] to position [2]
DEBUG: 2021-02-04 17:28:54.5728 ParameterBinding Information: 0 : CALLING BeginProcessing
Argument 1 <foo=bar "blob" bar>
Argument 2 <zap>
Argument 3 <foo:bar:baz,bip,bar>

# old style - native arguments are bound to Arguments property
PS > trace-command -PSHOST -Name ParameterBinding { ~/echoit foo="bar ""blob"" bar" zap foo:bar:baz,bip,bar }
DEBUG: 2021-02-04 17:29:01.9987 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/echoit]
DEBUG: 2021-02-04 17:29:01.9987 ParameterBinding Information: 0 :     BIND argument ["foo=bar "blob" bar" zap foo:bar:baz,bip,bar]
DEBUG: 2021-02-04 17:29:02.0058 ParameterBinding Information: 0 : CALLING BeginProcessing
Argument 1 <foo=bar blob bar>
Argument 2 <zap>
Argument 3 <foo:bar:baz,bip,bar>
PS > 

# view of tracing when --% is used
PS /Users/james> trace-command -PSHOST -name parameterbinding { ~/echoit --% 'foo bar' a,b,c "one two" "a\ b\ c"
>> }
DEBUG: 2021-02-08 17:07:10.0199 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/echoit]
DEBUG: 2021-02-08 17:07:10.0199 ParameterBinding Information: 0 :     BIND cmd line arg ['foo] to position [0]
DEBUG: 2021-02-08 17:07:10.0199 ParameterBinding Information: 0 :     BIND cmd line arg [bar'] to position [1]
DEBUG: 2021-02-08 17:07:10.0199 ParameterBinding Information: 0 :     BIND cmd line arg [a,b,c] to position [2]
DEBUG: 2021-02-08 17:07:10.0200 ParameterBinding Information: 0 :     BIND cmd line arg ["one] to position [3]
DEBUG: 2021-02-08 17:07:10.0200 ParameterBinding Information: 0 :     BIND cmd line arg [two"] to position [4]
DEBUG: 2021-02-08 17:07:10.0200 ParameterBinding Information: 0 :     BIND cmd line arg ["a\] to position [5]
DEBUG: 2021-02-08 17:07:10.0200 ParameterBinding Information: 0 :     BIND cmd line arg [b\] to position [6]
DEBUG: 2021-02-08 17:07:10.0200 ParameterBinding Information: 0 :     BIND cmd line arg [c"] to position [7]
DEBUG: 2021-02-08 17:07:10.0261 ParameterBinding Information: 0 : CALLING BeginProcessing
Argument 1 <'foo>
Argument 2 <bar'>
Argument 3 <a,b,c>
Argument 4 <"one>
Argument 5 <two">
Argument 6 <"a\>
Argument 7 <b\>
Argument 8 <c">

Additional considerations for tracing

It may be desirable to add additional tracing which provides information on the parameters as they were provided. The tracing above is created at the point where the StartInfo object is populated, and it may be useful to see the parameter before it is altered by globbing, etc.

Current Implementation

When PowerShell starts a new native process it takes all the arguments provided and attempts to stitch together the various parts into a single string (which is assigned to the Arguments property of the StartInfo object). This is done with some problematic behavior; empty strings '' are explicitly stripped, embedded quotes and spaces are “lost” and require addition escaping.

New Behavior

I think we can do better and reduce the effort and internal complexity when calling native applications. Dotnet has added a new property to the StartInfo object called ArgumentList which allows you to provide the arguments to the command as a collection of strings, alleviating the need to stitch the arguments into a single string. We can take advantage of this new API to reduce the complexity of our code. However, we should maintain backward compatibility if we can, so rather than producing new, breaking behavior via an experimental feature, I suggest that we provide a new runtime behavior based on a PowerShell variable. This allows users to change the behavior without restarting the PowerShell process and can be used when desired. By not changing the default behavior we can provide users an easy way opt-in to the new behavior. Telemetry can be added if desired to capture the count of how many times the current way is used in comparison with this new implementation.

NB: This proposal will actually increase the internal complexity of our code because we’ll have 2 ways of calling native applications. Hopefully, this would be temporary and we can use the new APIs exclusively in the future and deprecate the current code.

Tools used

The following utility is used to echo all passed parameters given in the examples above. This does not rely on the CLR runtime, but may be compiled for all platforms.

#include <stdio.h>
int main(int argc, char *argv[])
{
    for(int i = 1; i < argc; i++) {
        printf("Argument %d <%s>\n", i, argv[i]);
    }
    return 0;
}

We can add this to the tools for build if needed (either by binaries checked in for each platform or build)

Related Links:

Pull Requests

• https://github.com/PowerShell/PowerShell/pull/14692 (the PR to implement the behaviors listed here)
• https://github.com/PowerShell/PowerShell/pull/13483

Issues

• https://github.com/PowerShell/PowerShell/issues/13068
• https://github.com/PowerShell/PowerShell/issues/13428
• https://github.com/PowerShell/PowerShell/issues/1995

[bergmeister]

By not changing the default behavior we can provide users an easy way opt-in to the new behavior

I suggest to rather have the new behavior by default and use the PowerShell variable rather for opting out of the new behavior. Especially with previews, it will allow people to discover whether they need to adapt their code or to give feedback that would allow tweaking of this new feature.

[mklement0]

I mean the new one being proposed in a different issue somewhere.

I assume you mean the aforementioned #13068 (“native operator”) - its purpose is different, requires you to apply a different shell’s syntax and, without using a (here-)string as enclosure, is subject to the same conceptual headaches as --% while generally making it hard to integrate PowerShell variables / expressions in a given call. In short: it is not meant to address the problem at hand, and it would so poorly - see https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-653319079

That said, a new call operator - to be used explicitly, in lieu of & (which also came up in the same thread, at https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-653526374) - might be a low-ceremony alternative that avoids the preference-variable scoping headaches.

Finding the right sigil combination (I don’t think a single character is an option), may be a challenge (&! was mentioned), and, of course, it would be a very visible reminder in every call that extra effort is needed to get argument-passing to act correctly.

I’m a sysadmin.

Kudos on the extraordinary depth of your programming knowledge (just to be very clear: I mean it).

“I think it should be X but what I do I know, I don’t run into this problem”.

Understood. I just wanted to complement that with a transpersonal perspective, to leave no doubt that many others do struggle with this.