Update debugger attributes to provide an optimal and effective debugging experience in PowerShell

DISCLAIMER: Unless you’ve spent some time with debuggers, and specifically with the PowerShell debugger, this may confuse you.

Summary of the new feature/enhancement

As a PowerShell user, I want the entire debugging experience focused on “Just My Code” (JMC) by default, so that I can debug that code effectively and optimally without getting lost in or distracted by other code used in PowerShell.

As a PowerShell user, I want scripts and/or modules installed from a gallery to be automatically recognized as “not my code”, so that my debugging experience does not debug into those scripts/modules by default.

As a PowerShell tool author, I want properly defined and properly functioning DebuggerHidden and DebuggerStepThrough attributes, so that I can leverage those attributes in my tools and provide users of my tool with an optimal debugging experience.

As a PowerShell user, I want an option to allow me to turn off “Just My Code” (JMC) debugging, so that I can debug into installed (but not internal) PowerShell code when I need to.

For reference: Debug user code with Just My Code.

Proposed technical implementation details

This issue is all about the following debugger attributes that can be used in any script, function, or script block:

• [System.Diagnostics.DebuggerHidden()]
• [System.Diagnostics.DebuggerStepThrough()]
• [System.Diagnostics.DebuggerNonUserCode()]

Since PowerShell is interpreted rather than compiled, we need to be more vigilant when it comes to debugger behavior so that scripters can debug their own code more effectively. This means preventing scripters from debugging code that is meant to be entirely internal (hidden from the end user) as well as preventing users from unintentionally debugging scripts or modules that are considered stable (these may include scripts that they installed, or scripts that they wrote themselves but that they consider fully debugged). Essentially, some code should be treated as much like a black box as an executable or binary module would. These attributes are key to making that happen, but up to this point, the debugger has not been properly implemented to provide an optimal debugging experience for users when those attributes are used.

Some background

If you would like to better understand the problems, try some of the scenarios listed below using PowerShell 7 preview 3 or earlier.

1. Scenario 1: Set a command breakpoint on Set-StrictMode.

   Open a new session, and then invoke the following:

   Set-PSBreakpoint -Command Set-StrictMode
   

   When you do this, you’ll immediately hit a breakpoint because PSReadline internally invokes PowerShell, and the PSConsoleHostReadLine function does not use a debugger attribute to tell PowerShell that they aren’t meant for end-user debugging.

   If you did this, the only way out is to remove your breakpoint and then quit the debugger.

2. Scenario 2: Set a command breakpoint on Set-StrictMode with PSReadline unloaded, and generate an error.

   Open a new session, and then invoke the following:

   Remove-Module PSReadline
   Set-PSBreakpoint -Command Set-StrictMode
   Get-Process -Id 12345678
   

   When you invoke these commands, the moment the error is raised you’ll hit a breakpoint inside the formatting code. This happens because the formatting code invokes nested script blocks, and at the moment these script blocks do not pick up on the DebuggerHidden attribute that is set on the parent script block.

   Note that in this and the previous example, I’m just using Set-StrictMode because it is an easy example to pick on. The point is that there are plenty of tools, including PowerShell itself, that invoke PowerShell in a way that is supposed to be internal (a black box to the end user), but that the debugger can still see and trigger breakpoints on. It all depends on the commands or variables that those tools invoke, and the level of effort the tool authors made in order to prevent their internal logic written in an interpreted language from being visible to the PowerShell debugger.

3. Scenario 3: Set a breakpoint on Get-Help in Visual Studio Code.

   Open Visual Studio Code, and in the embedded PowerShell terminal, invoke the following:

   Set-PSBreakpoint -Command Get-Help
   

   Now move your mouse around over a tab that has a PowerShell ps1 file open in it. Eventually you’ll probably see the terminal crash. This is because it tries to enter the debugger in a call to Get-Help, which is invoked in a mouse hover event handler. Either that, or something similar…I haven’t read through the code to see what is actually going on, but regardless, setting a breakpoint should not cause a crash.

4. Scenario 4: Set a breakpoint on something used inside of a Where-Object or ForEach-Object script block, when that command is invoked from a script block that is configured with DebuggerHidden.

   Open a new PowerShell session and invoke the following:

   Set-PSBreakpoint -Variable i -Mode ReadWrite
   & {
       [System.Diagnostics.DebuggerHidden()]
       param()
       $i = 0
       1..5 | ForEach-Object {$i += $_}
   }
   

   When you invoke that code, notice that the debugger does not stop on the line where $i is initialized, but that it does stop inside the ForEach-Object invocation where $i is changed, even though this is inside of a parent script block with the DebuggerHidden attribute. Support for the DebuggerHidden attribute in PowerShell was simply not implemented to handle scenarios like this, but it must be because otherwise it can be very difficult for tool builders to create truly internal code that is written in PowerShell.

5. Scenario 5: Set a breakpoint on something used inside of a script block defined in user scope, but pass that script block into a command that is configured with DebuggerHidden.

   Open a new PowerShell session and invoke the following:

   $sb = {$global:i += $_}
   Set-PSBreakpoint -Variable i -Mode Read
   $i = 0
   function Invoke-FiveTimes {
       [System.Diagnostics.DebuggerHidden()]
       param(
           [Parameter(Mandatory)]
           [ValidateNotNull()]
           [scriptblock]
           $ScriptBlock
       )
       1..5 | ForEach-Object $ScriptBlock
   }
   Invoke-FiveTimes -ScriptBlock $sb
   

   When you invoke that code, the debugger will stop on the line in the script block that you passed into the function each time it reads the value of $i. This may seem like the correct behavior, because the script block is defined outside of the function that uses the DebuggerHidden attribute, but that’s not true. DebuggerHidden is meant to act as bouncer to a black box – the debugger is simply not allowed in. On the other hand, DebuggerStepThrough is meant to allow a debugger to pass through a script block and then debug other commands or script blocks that are invoked that are defined outside of the script block that has the attribute.

6. Scenario 6: Stepping into a function that invokes a script block passed in as a parameter, when that function uses DebuggerStepThrough.

   Open a new PowerShell session and invoke the following:

   & {
       $sb = {
           Get-Process -Id $PID
       }
       function Test-StepInto {
           [CmdletBinding()]
           [System.Diagnostics.DebuggerStepThrough()]
           param(
               [scriptblock]$ScriptBlock
           )
           & $ScriptBlock
       }
       Wait-Debugger
       Test-StepInto -ScriptBlock $sb
   }
   

   When you invoke this script, you’ll be dropped into the debugger on the Test-StepInto command invocation. Invoke the debugger s command to step into the Test-StepInto function. The debugger is supposed to step through that function because of the DebuggerStepThrough attribute, and bring you into the script block that is invoked, but it doesn’t. Instead, it just bounces you out the door to the closing curly brace after the invocation.

Most of these scenarios use command breakpoints, but as you can see some also use variable breakpoints to have similar impacts if I choose the right variable names. A key point is that command breakpoints and variable breakpoints are very powerful debugging tools, and should be capable of bringing you very close to your issues in your scripts or modules when you want them to, but as long as other tools and PowerShell itself can get in the way, which they do when they use the same commands or variable names in their implementation, the usefulness of these breakpoints is greatly diminished. You can work around the breakpoint challenges by scoping these types of breakpoints to specific files (one breakpoint is created per file), but that requires much more work, especially if you’re just working from the command line, and it is more complicated if you’re working across multiple files in an automated solution.

For the last few scenarios, they show how stepping is not working the way it should when these attributes are defined. The entire reason for the existence of these attributes is efficient debugging, yet PowerShell is not handling them properly, and debugging is anything but efficient as a result.

Some solutions

Wouldn’t it be great if you could count on command breakpoints and variable breakpoints to hit breakpoints in your code, and not get caught up in code from any of the other PowerShell tools you have on your system?

As a tool author, wouldn’t it be great if you could more easily author your code and be able to count on it being treated like a black box when it is installed on other systems?

And when you are stepping through the debugger, wouldn’t it be great if the debugger properly avoided code that it shouldn’t?

The proposed solutions below are designed to solve those problems and make debugging much, much easier.

Defining the behavior provided by the Debugger* attributes

Current behavior

The current behavior of the debugger attributes isn’t very appropriate for an interpreted language. Here’s how these attributes work today:

DebuggerHidden (DH)

If you apply this attribute to a script block (or an entire script), the contents of the root scope of that script block will be hidden from the debugger. Breakpoints will not trigger within that script block, but they will trigger in script blocks defined and invoked within that script block, or in script blocks defined outside of that script block but invoked internally as parameters. Wait-Debugger, and ActionPreference.Break preferences work the same way. Users will not be able to step through the root scope of the script block using the debugger.

You can toggle the DH behavior on a script block today by changing the boolean value of a script block’s DebuggerHidden property.

DebuggerStepThrough (DST)

If you apply this attribute to a script block (or an entire script), the contents of the root scope of that script block will be hidden from the debugger. Breakpoints will not trigger within the root scope of that script block, but they will trigger in script blocks defined and invoked within that script block. Wait-Debugger, and ActionPreference.Break preferences work the same way. If you try to step into a script block that uses this attribute, you cannot. If, however, you are in the debugger on a breakpoint that is in a nested scope where the parent scope has the DST attribute, you can step out of that scope and step through the internals that are supposed to be hidden via DST.

You cannot toggle the DH behavior on a script block today by changing the boolean value of a script block’s DebuggerStepThrough property because that property is internal. This is unfortunate, and it would be better if that property was made public so that you could toggle it on a single function/script block if needed to do some debugging without changing the code.

DebuggerNonUserCode (DNUC)

This attribute is simply linked to the DST behavior, and functions the exact same way.

Desired behavior

To correct the issues identified above, along with other issues not specifically called out here, the debugger attribute behavior in PowerShell should be defined as follows:

DebuggerHidden (DH), aka the debugger bouncer)

Breakpoints: Breakpoints do not trigger within a script block that has the DH attribute, nor do they trigger in anything that is invoked from within that script block. This attribute acts as a bouncer for the PowerShell debugger, and simply does not let it come in and hang out, at all.

Step Into: You cannot step into a script block that has the DH attribute. Step into is simply treated as step over.

Step Out: N/A. Since the debugger is bounced, you can never step out to a DH script block.

Step Over: N/A. Since the debugger is bounced, you can never step over from within a DH script block.

Toggle behavior: Script blocks expose a DebuggerHidden property that can be changed to toggle this behavior without changing code. This is useful when you want to temporarily allow debugging for something that is normally hidden from the debugger, assuming you have access to the actual script block to toggle the flag.

Recommended Usage: Use this attribute in tooling on code that is meant to be internal, invisible to the PowerShell debugger. For example, formatting code in PowerShell itself, functions that are internal (not exported) from a module, or any calls made to PowerShell from a tool that are only meant for internal use and shouldn’t be visible to the debugger as users debug their own scripts.

DebuggerStepThrough (DST)

Breakpoints: Breakpoints do not trigger within a script block that has the DST attribute, nor do they trigger in any script block defined within and invoked from that script block, recursively. Breakpoints will trigger in commands or script blocks that are defined outside of that script block, if the attributes on those commands allow it.

Step Into: If you step into a script block that has the DST attribute, the debugger will step through that command to the first script block defined externally to that command that is visible to the debugger. This means if you pass in a script block as a parameter and then invoke it, the debugger can step into that script block.

Step Out from nested: If you are on a breakpoint in a scope that is nested under a script block that has the DST attribute and you step out, the debugger will step out of that script block, and out of the script block that has the DST attribute.

Step Over: If you are on a breakpoint in a scope that is nested under a script block that has the DST attribute and you step over, the debugger will step out of that script block, and through the script block that has the DST attribute until it steps out of that script block or into a script block that the debugger can step into.

Toggle behavior: Script blocks should expose their DebuggerStepThrough property as public so that it can be changed to toggle DST behavior without changing code. This is useful when you want to temporarily allow debugging for something that is normally hidden from the debugger. The DebugPx module has a command that makes this possible on single functions or an entire module as needed, which keeps debugging efficient and focused.

Recommended Usage: Use this attribute in tooling on code that is not internal but that is considered stable/debugged. For example, if you and/or your team create some scripts or modules and debug them, and you don’t want to go through that code with the debugger, use this attribute. Another example: if you publish stable releases of scripts or modules to the gallery for others to download and use and you don’t want users to debug them, set this attribute in the scripts or in the functions included in those modules so that your users don’t get confused by the debugger taking them into that stable, debugged code. Note that with the published module/script scenario, you could instead rely on Just My Code handling that for you.

DebuggerNonUserCode (DNUC)

This attribute is special, because nobody should ever have to set it in their code. Instead, PowerShell should automatically distinguish between user code and non-user code, taking a conservative approach so that it doesn’t make mistakes.

User code is a script or module that is written by an end user and needs support for debugging. Non-user code is a script or module or tool that is written by someone else and installed on a system for use as a tool. It generally doesn’t need to be debugged, because it is a tool, but users may want to debug it, especially if it is not a stable release.

Debugging-functionality-wise, DNUC should function just like DST; however, there are two key differences between the current functionality of this attribute today and how it should work going forward:

1. The automatic, dynamic application of that attribute to any file that comes from an installed tool (i.e. the debugger needs to recognize which files are non-user files, and treat them accordingly).
2. By default, PowerShell would be configured to only debug user code (“Just My Code”), but an option would have to be provided to allow users to debug non-user code when needed.

For the automatic, dynamic application of the DNUC attribute, scripts or modules installed using PowerShellGet should automatically be treated as DNUC. Code that is installed under specific directories (“Program Files”, “Program Files (x86)”, and “Windows” on Windows operating systems, or “/usr” or “/opt” on Linux or macOS – please comment on these directories) should also automatically be treated as DNUC. Everything else should be treated as user code. Tool builders that ship scripts that are not meant to be treated as user code but that install outside of these paths can manually apply the DNUC attribute (or one of the other attributes) to that code.

Another option would be to define an environment variable that identifies DNUC base paths, so that installers or users can extend it if they want to mark additional DNUC folders.

For the “Just My Code” option, that could be provided as an automatic PowerShell variable ($PSDebugJustMyCode) that is set to true by default but can be set to false to enable debugging of all non-user code. Additionally, I have seen tools show a “MyCode” flag on modules (libraries), and would like to add that as a session-specific read/write flag to modules and scripts (commands), so that instead of allowing debugging of all non-user code, scripters could toggle the flag on a specific script or module that they want to debug in that session.

Related questions

Are the listed DNUC paths correct/enough?

I’ve listed what makes sense to me for this feature. Are the paths listed not conservative enough? Are there other paths that should be included by default?

Should we have $env:PSNonUserCodePath?

Such an environment variable would allow users to configure specific paths where they install scripts that they don’t want to debug day-to-day.

What about classes?

Does PowerShell use these attributes with classes today? I haven’t dug into how these attributes work with classes/methods/properties in PowerShell at all yet, but from the outside I think the exact same behavior would apply, and the only decisions would be whether or not these attributes can be applied on classes, methods, properties, accessors, etc.

StepThrough as a debugger command?

Since the debugger needs to be designed to be able to step through a block of script, it would be helpful if users using the debugger could invoke a stepThrough command (t for short) to step through the current block of script into the first nested block defined outside of this block that the debugger can step into. That will enable faster debugging in scenarios where you temporarily stop inside of a command to look around, and then want to go through that command into other commands that it invokes.

Expose ScriptBlock.DebuggerStepThrough property as public?

DebuggerHidden is already public. If we were to decide to make just one of these properties public, I would have argued for DebuggerStepThrough instead, since that would actually be used on code where you can access that attribute. At any rate, I think DebuggerStepThrough should be public so that it can be modified without changing code, which allows commands like what I have in DebugPx do their work without mucking around with internals.

Should PSScriptAnalyzer have rules for debugger attributes?

In general, I think the majority of the community shouldn’t have to use debugger attributes or worry about debugger attributes, although tool builders will need to consider DebuggerHidden. Efficient debugging should be possible automatically if scripts/modules are distributed through a repository (which can be as simple as a file share).

That said, I think it would be useful to consider a few PSSA rules. Generally speaking, DebuggerNonUserCode should not be used in scripts. Users who use that in scripts should really consider DebuggerStepThrough instead. Script blocks should not have more than one debugger attribute associated with them, because there is no point to that (one would override another, so only one would be in effect). Those scenarios could be handled in PSSA rules if desired.

End result

The end result of these changes should result in debugger attributes behaving as follows:

	Step into	Breakpoints triggerable
DebuggerHidden	Treated as step over	Never
DebuggerStepThrough	Treated as step through 1	Never
DebuggerNonUserCode	Treated as step through 1	Depends on JMC settings 2

¹ t, stepThrough will be added as a new debugger options, allowing users to get into nested code more efficiently from the debugger. ² New option added to ScriptDebugger class exposed via a $PSDebugJustMyCode automatic variable, set by default to $true.

That’s enough of my thoughts on this for now.

The good news is that the outcome of this should help end users not have to think about any of this – they should simply be able to use the debugger to debug only their code. At the same time, developers will have options that allow them to dig in deeper with the debugger when needed.

If you care about debugging PowerShell, specifically in how to help others debug PowerShell more easily and more efficiently, please share your thoughts on this so that we can make the debugging experience as easy as possible for the PowerShell community.

[SteveL-MSFT]

@KirkMunro I appreciate you following the new proposed process by initiating discussion in an issue and driving towards consensus. However, we need more varied feedback from the community. I’ve tweeted this issue so hopefully that’ll encourage more diverse discussion on agreement on the issue and then agreement on a proposed solution. The PowerShell team is busy trying to complete commitments we’ve made to PS7 so it’s unlikely this will make it into PS7, but we can certainly spend time on this early in vNext.

[KirkMunro]

Anyone care to share thoughts on the intent of the current implementation of the debugger attributes, or thoughts about where I’m trying to take this?

There’s a pretty cool opportunity here to make the debugging experience much, much better and much more focused for all users, without much effort on the part of scripters.

Also, a few more things I believe are incorrect and would like to change:

1. If you set a command breakpoint on a command that happens to have the DebuggerHidden attribute, the debugger will not stop on the invocation of that command. I believe that behavior is wrong, because the attribute’s effect should apply to the script block where it is used, not to invocations outside of that script block.

2. If you set a command breakpoint on a function, when that breakpoint triggers, you are already inside of the function. I believe that behavior is also incorrect, because it’s inconsistent (command breakpoints for cmdlets break on the command invocations) and it causes the previous issue. No matter what type of command being invoked, a command breakpoint should enter the debugger on the invocation of that command when triggered.

~Lastly, on the notion of non-user code (DebuggerNonUserCode), there is another issue that causes pain for tool builders, where tools can be influenced by user preferences (#10334). User preferences such as $PSDefaultParameterValues and $*Preference variables should not influence internal PowerShell invocations inside of tools that users use. I was thinking today that an easy application of the DebuggerNonUserCode attribute on PowerShell invocations from within tools would be helpful in preventing user preferences from impacting code that is meant to be run as a black box.~ (Strike that, the thoughts in this paragraph are better covered in #10334).