id-challenge HTML element does not always contain a JSON value

Sometimes aws-google-auth fails with the following exception:

  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/__init__.py", line 197, in process_auth
    google_client.do_login()
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 259, in do_login
    sess = self.handle_sk(sess)
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 371, in handle_sk
    raise e
  File "/usr/local/lib/python3.7/site-packages/aws_google_auth/google.py", line 366, in handle_sk
    challenges = json.loads(challenges_txt)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I think the issue is because of the challenge HTML form having changed. During a failed run, it looks like this:

            <div class="LJtPoc" jsname="Ki8mld" jscontroller="qNel6e" jsaction="rcuQ6b:WYd">
                <form method="POST" id="challenge" action="/signin/challenge/sk/6" jsname="rzWj5" jscontroller="HNBfvc" jsaction="submit:zbvklb" jsshadow>
                    <content>
                        <input name="challengeId" type="hidden" id="challengeId" value="6">
                        <input name="challengeType" type="hidden" id="challengeType" value="2">
                        <input name="continue" type="hidden" value="https://accounts.google.com/o/saml2/initsso?idpid=REDACTED&amp;spid=REDACTED&amp;forceauthn=false&amp;from_login=1&amp;as=REDACTED">
                        <input name="scc" type="hidden" value="1">
                        <input name="sarp" type="hidden" value="1">
                        <input name="checkedDomains" type="hidden" value="youtube">
                        <input name="pstMsg" type="hidden" value="0">
                        <input name="TL" type="hidden" value="REDACTED">
                        <input type="hidden" name="gxf" id="gxf" value="REDACTED">
                        <div jsname="KrwUDc"><img jsname="TqVmm" class="JC07Dd" src="//ssl.gstatic.com/accounts/marc/gnubby_icon.png" alt="">
                            <div jsname="kwHene"></div>
                            <div class="EGmPD" jsname="BCqkPb">Insert your security key</div>
                            <div class="VnJmLc" jsname="NhJ5Dd">If your security key has a button, tap it.
                                <br>If it doesn't, remove and re-insert it.</div>
                            <div>
                                <input type="hidden" name="id-challenge" jsname="wCVnAe" value="REDACTED_STRING_NOT_JSON">
                                <input type="hidden" name="id-assertion" jsname="n6knUb">
                            </div>
                            <div jsname="C0oDBd" data-challenge-ui="%.@.null,null,&quot;TWO_STEP_VERIFICATION&quot;,6,null,&quot;SEND_SUCCESS&quot;,null,null,2,1,true,true,true,null,null,null,&quot;REDACTED&quot;,&quot;https://lh3.googleusercontent.com/-XdUIqdMkCWA/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3re6XM1OFpqpdEsscOf-CRg8KCNC_g/mo/photo.jpg&quot;,null,null,null,null,[]
,{&quot;1010&quot;:[2,false]
,&quot;5010&quot;:[null,null,null,&quot;https://accounts.google.com/signin/challenge/sk/6&quot;,null,[&quot;google.com&quot;,&quot;REDACTED_STRING&quot;,[[2,&quot;REDACTED_STRING&quot;]
]
,&quot;{\&quot;appid\&quot;:\&quot;https://www.gstatic.com/securitykey/origins.json\&quot;}&quot;]
]
}]
"></div>
                            <div class="ARshqb">
                                <input type="checkbox" name="TrustDevice" id="trustDevice" class="aCOJmf" checked><span>Don&#39;t ask again on this computer</span>
                                <div class="Bfmfyc" role="tooltip">
                                    <div class="x7qQqf"></div>
                                    <div class="hzC8Lb">For your convenience, keep this checked. On shared devices, additional precautions are recommended. <a href="https://support.google.com/accounts/?p=securesignin&amp;hl=en" target="_blank">Learn more</a></div>
                                </div>
                            </div>
                        </div>
                    </content>
                </form>
            </div>

Based on this it looks like it comes down to the value string of id-challenge no longer being a stringified JSON object, but instead a “random” string that looks like "...hUTYdnvUG6M25UzFVz...". I poked around but it wasn’t clear to me what the right way is to fix this, especially because

<div jsname="C0oDBd" data-challenge-ui="%.@.null,null,&quot;TWO_STEP_VERIFICATION&quot;,6,null,&quot;SEND_SUCCESS&quot;,null,null,2,1,true,true,true,null,null,null,&quot;[e-mail]&quot;,&quot;https://lh3.googleusercontent.com/-XdUIqdMkCWA/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3re6XM1OFpqpdEsscOf-CRg8KCNC_g/mo/photo.jpg&quot;,null,null,null,null,[]
,{&quot;1010&quot;:[2,false]
,&quot;5010&quot;:[null,null,null,&quot;https://accounts.google.com/signin/challenge/sk/6&quot;,null,[&quot;google.com&quot;,&quot;REDACTED_STRING&quot;,[[2,&quot;REDACTED_STRING&quot;]
]
,&quot;{\&quot;appid\&quot;:\&quot;https://www.gstatic.com/securitykey/origins.json\&quot;}&quot;]
]
}]
"></div>

looks like it has the relevant data but also looks like it isn’t simply JSON…

This has started happening approximately 5 days ago.

In case it is relevant, saving the html and opening it in a browser (Chrome) renders this page:

[sinkr]

Set up some kind of campaign / funding avenue and I’m sure a bunch of us will have no trouble donating the money required to buy a YubiKey.

[maxleonca]

same here, yubikey as well.