HTTP/NTLM reauthentication no longer works in Active Scan

Describe the bug

After upgrading from ZAP 2.11.1 to 2.12.0, running an Active Scan against a node in the Site tree that requires Digest or NTLM authentication stopped working. The console log shows “java.lang.IllegalStateException: AuthScheme is null”:

java.lang.IllegalStateException: AuthScheme is null
        at org.apache.hc.core5.util.Asserts.notNull(Asserts.java:56) ~[?:?]
        at org.apache.hc.client5.http.impl.auth.HttpAuthenticator.updateAuthState(HttpAuthenticator.java:216) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.needAuthentication(ProtocolExec.java:294) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:207) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:158) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:394) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:297) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:103) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:298) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:266) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:222) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:193) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:57) ~[?:?]
        at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:478) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.sendAndReceive(Analyser.java:528) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.analyse(Analyser.java:190) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:381) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:384) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.start(Analyser.java:136) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.HostProcess.run(HostProcess.java:372) ~[zap-2.12.0.jar:2.12.0]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Steps to reproduce the behavior

1. Install ZAP “Core Cross Platform Package”.
2. Start zap.sh. If absent, install the “Spider” add-on then restart.
3. Double-click “Default Context”
4. Under “Include in Context”, click “Add…”, type “.*httpbin.*”, then click “Add”.
5. Under “Authentication”, change “Manual Authentication” to “HTTP/NTLM Authentication”. Set “Hostname” to “httpbin.org”.
6. Under “Users”, click “Add…”, set Username to “user” and password to “passwd” then click Add.
7. Under “Forced User”, make sure “user” is selected then click OK to close the dialog.
8. Click the “Forced User Mode” toolbar button to enable it.
9. On the Quick Start tab, click Automated Scan. Set “URL to attack” to “http://httpbin.org/digest-auth/auth/user/passwd”. Make sure “Use traditional spider” is selected then click Attack. Wait for the spider to complete.
10. In the Site tree, click the “GET:passwd” node. Verify the Request and Response tabs show the authentication having succeeded.
11. In the Site tree, right-click the “GET:passwd” node then click “Attack > Active Scan…”. Select “user” then click “Start Scan”.

Expected behavior

At least one successful 200 OK traffic message should be shown on the Active Scan tab. This worked fine in 2.11.1. In 2.12.0, active scan rules can’t get past the “AuthScheme is null” IllegalStateException.

Software versions

OWASP ZAP Version: 2.12.0

Installed Add-ons: [[id=bruteforce, version=12.0.0], [id=callhome, version=0.5.0], [id=commonlib, version=1.11.0], [id=database, version=0.1.0], [id=diff, version=12.0.0], [id=gettingStarted, version=14.0.0], [id=help, version=15.0.0], [id=invoke, version=12.0.0], [id=network, version=0.3.0], [id=onlineMenu, version=10.0.0], [id=pscanrules, version=44.0.0], [id=quickstart, version=35.0.0], [id=reports, version=0.16.0], [id=reveal, version=5.0.0], [id=spider, version=0.1.0], [id=tips, version=10.0.0]]

Screenshots

No response

Errors from the zap.log file

java.lang.IllegalStateException: AuthScheme is null
        at org.apache.hc.core5.util.Asserts.notNull(Asserts.java:56) ~[?:?]
        at org.apache.hc.client5.http.impl.auth.HttpAuthenticator.updateAuthState(HttpAuthenticator.java:216) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.needAuthentication(ProtocolExec.java:294) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:207) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapHttpRequestRetryExec.execute(ZapHttpRequestRetryExec.java:81) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.ZapInternalHttpClient.doExecute(ZapInternalHttpClient.java:158) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245) ~[?:?]
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl0(HttpSenderApache.java:394) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:297) ~[?:?]
        at org.zaproxy.addon.network.internal.client.apachev5.HttpSenderApache.sendImpl(HttpSenderApache.java:103) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAuthenticated(BaseHttpSender.java:298) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendNoRedirections(BaseHttpSender.java:266) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.send(BaseHttpSender.java:222) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:193) ~[?:?]
        at org.zaproxy.addon.network.internal.client.BaseHttpSender.sendAndReceive(BaseHttpSender.java:57) ~[?:?]
        at org.parosproxy.paros.network.HttpSender.sendAndReceive(HttpSender.java:478) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.sendAndReceive(Analyser.java:528) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.analyse(Analyser.java:190) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:381) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.inOrderAnalyse(Analyser.java:384) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.Analyser.start(Analyser.java:136) ~[zap-2.12.0.jar:2.12.0]
        at org.parosproxy.paros.core.scanner.HostProcess.run(HostProcess.java:372) ~[zap-2.12.0.jar:2.12.0]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Additional context

In the above reproduction steps, I demonstrate the issue with some public HTTP Digest endpoint. However, I am running into this same “AuthScheme is null” issue with various internal service endpoints within my organization including those that require NTLM authentication. Both Digest and NTLM authentication worked fine in ZAP 2.11.1.

Would you like to help fix this issue?

• Yes

[benken-parasoft]

The referenced PR should address the issues reported

Yes, it does! Your commit solves the remainder of the problems I was running into. I verified after building a new network add-on from thc202:network/reauth-fixes. I now see all the expected traffic and alerts. I also verified that the ZAP log shows no more runtime exceptions for any of my scans.

[benken-parasoft]

Configuring a proxy server with NTLM authentication against Active Directory (for example) can be tricky to setup. So, if you have a patch or pull request that are you need help verifying then I am happy to pull it in and build a new network add-on locally and retest using the proxy server in my environment.