Most WinRM operations can fail on low timeout

Pywinrm 0.2.0 lowered the default operation timeouts from 60s to 20s (and the read timeout to 30s) to allow for faster failures on dead hosts, etc. This generally isn’t a problem, but for newly-created (and possibly for heavily-loaded) instances on AWS, the lower timeout can expire before the initial shell gets created (and possibly in other places).

Rather than lowering the timeout, we could add polling for it to other places known to run long (starting with open_shell) similar to what we do during receive.

The failure manifests as a 500, eg:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/winrm.py", line 151, in _winrm_connect
    self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
  File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 132, in open_shell
    res = self.send_message(xmltodict.unparse(req))
  File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 207, in send_message
    return self.transport.send_message(message)
  File "/usr/local/lib/python2.7/dist-packages/winrm/transport.py", line 191, in send_message
    raise WinRMTransportError('http', error_message)
WinRMTransportError: (u'http', u'Bad HTTP response returned from server. Code 500')

See https://github.com/ansible/ansible/issues/16873 for original issue.

[silencev]

I was struggling with exception WinRMTransportError before, especially send WinRM command to the machine just bootup. After digging into the real error message that WinRM sent back. I list down every error happened in my environment and if it happened, I will retry again.

Illegal operation attempted on a registry key that has been marked for deletion.

similar issue also report here: https://github.com/test-kitchen/winrm-transport/issues/18

<f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2147943418" Machine="windows-host">
    <f:Message>
        <f:ProviderFault provider="Shell cmd plugin" path="%systemroot%\\system32\\winrscmd.dll">Illegal operation attempted on a registry key that has been marked for deletion. </f:ProviderFault>
    </f:Message>
</f:WSManFault>

Class not registered

dont know the root cause till now

<s:Detail>
    <f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2147746132" Machine="windows-host">
        <f:Message>
            <f:ProviderFault provider="Shell cmd plugin" path="%systemroot%\\system32\\winrscmd.dll">Class not registered </f:ProviderFault>
        </f:Message>
    </f:WSManFault>
</s:Detail>

socket read timeout

I guess the windows machine is busy and not able to response in a short given timeout

OperationTimeout

This error is handled by pywinrm itself here. however, sometime this error may only contains the error code 2150858793 but without the string _http://schemas.microsoft.com/wbem/wsman/1/windows/shell/Receive_ So I will use error code to capture this error and retry again

<s:Detail>
    <f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2150858793" Machine="windows-host">
        <f:Message>The WS-Management service cannot complete the operation within the time specified in OperationTimeout.  </f:Message>
        </f:WSManFault>
</s:Detail>

[twglomski]

Also, I was getting the error above intermittently after startup. Turns out a process on the Windows server was pegging the CPU and as a result the call was hanging (and if it didn’t get the error on first connection, every command would take a really long time). Just throwing this up here in case the intrepid Googler stumbles upon this.