Tenant registration ends with Invalid Quote with TPM provided by QEMU

Is your an issue a feature request? If so, please raise it as an enhancement

Environment

• OS / version: CentosStream-9/Fedora-35
• Processor architecture: x86_64
• TPM Manufacturer: QEMU
• Keylime version: current latest upstream 2902fe8a7173f3350138f7dcfe8551f1fc3b697f

Description

System is a virtual system with TPM emulated by QEMU. IMA is configured in kernel (I am not using keylime IMA emulator). When registering a tenant it ends up in “operational_state”: “Invalid Quote” and verifier log contains: keylime.ima - ERROR - IMA measurement list does not match TPM PCR 0cf7680bac137381a86c8d467f7d92331 9d680d6ae5a63e6a493cbd2ef1efa4b

Expected behavior vs. actual behavior

system should be properly registered having state Get Quote.

Steps to reproduce problem

1. Have a system with qemu provided TPM
2. enable IMA in kernel
3. install keylime upstream bits
4. configure keylime and add a tenant

Adding brief test log below

# ./test.sh 

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Do the keylime setup
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ 17:10:09 ] :: [  BEGIN   ] :: Running 'rlImport "./test-helpers"'
:: [ 17:10:09 ] :: [   INFO   ] :: rlImport: Found './test-helpers' during upwards traversal
:: [ 17:10:09 ] :: [   INFO   ] :: rlImport: Will try to import ./test-helpers from /var/tmp/fmf_wrapper_git_cache/tests/https:__github.com_RedHat-SP-Security_keylime-tests/./Library/test-helpers/lib.sh
:: [ 17:10:09 ] :: [   INFO   ] :: found dependencies: ''
:: [ 17:10:09 ] :: [   PASS   ] :: Command 'rlImport "./test-helpers"' (Expected 0, got 0)
:: [ 17:10:09 ] :: [   INFO   ] :: using '/var/tmp/beakerlib-EKC5AC1/backup-limeConf' as backup destination
:: [ 17:10:09 ] :: [  BEGIN   ] :: Running 'sed -i 's/^require_ek_cert.*/require_ek_cert = False/' /etc/keylime.conf'
:: [ 17:10:09 ] :: [   PASS   ] :: Command 'sed -i 's/^require_ek_cert.*/require_ek_cert = False/' /etc/keylime.conf' (Expected 0, got 0)
:: [ 17:10:09 ] :: [  BEGIN   ] :: Running 'sed -i 's/^ca_implementation.*/ca_implementation = openssl/' /etc/keylime.conf'
:: [ 17:10:09 ] :: [   PASS   ] :: Command 'sed -i 's/^ca_implementation.*/ca_implementation = openssl/' /etc/keylime.conf' (Expected 0, got 0)
Redirecting to /bin/systemctl status tpm2-abrmd.service
Redirecting to /bin/systemctl start tpm2-abrmd.service
:: [ 17:10:09 ] :: [   LOG    ] :: rlServiceStart: Service tpm2-abrmd started successfully
:: [ 17:10:14 ] :: [  BEGIN   ] :: Running 'keylime_verifier 2>&1 >> /var/tmp/limeLib/limeLib-keylime-verifier.log &'
:: [ 17:10:14 ] :: [   PASS   ] :: Command 'keylime_verifier 2>&1 >> /var/tmp/limeLib/limeLib-keylime-verifier.log &' (Expected 0, got 0)
:: [ 17:10:14 ] :: [  BEGIN   ] :: Running 'limeWaitForVerifier'
:: [ 17:10:14 ] :: [   INFO   ] :: rlWaitForSocket: Waiting max 120s for socket `8881' to start listening
:: [ 17:10:15 ] :: [   INFO   ] :: rlWaitForSocket: Wait successful!
:: [ 17:10:15 ] :: [   PASS   ] :: Command 'limeWaitForVerifier' (Expected 0, got 0)
:: [ 17:10:15 ] :: [  BEGIN   ] :: Running 'keylime_registrar 2>&1 >> /var/tmp/limeLib/limeLib-keylime-registrar.log &'
:: [ 17:10:15 ] :: [   PASS   ] :: Command 'keylime_registrar 2>&1 >> /var/tmp/limeLib/limeLib-keylime-registrar.log &' (Expected 0, got 0)
:: [ 17:10:15 ] :: [  BEGIN   ] :: Running 'limeWaitForRegistrar'
:: [ 17:10:15 ] :: [   INFO   ] :: rlWaitForSocket: Waiting max 120s for socket `8891' to start listening
:: [ 17:10:16 ] :: [   INFO   ] :: rlWaitForSocket: Wait successful!
:: [ 17:10:16 ] :: [   PASS   ] :: Command 'limeWaitForRegistrar' (Expected 0, got 0)
:: [ 17:10:16 ] :: [  BEGIN   ] :: Running 'keylime_agent 2>&1 >> /var/tmp/limeLib/limeLib-keylime-agent.log &'
:: [ 17:10:16 ] :: [   PASS   ] :: Command 'keylime_agent 2>&1 >> /var/tmp/limeLib/limeLib-keylime-agent.log &' (Expected 0, got 0)
Writing allowlist to /var/tmp/fmf_wrapper_git_cache/tests/https:__github.com_RedHat-SP-Security_keylime-tests/functional/basic-attestation-on-localhost/allowlist.txt with sha256sum...
Creating allowlist for init ram disk
extracting /boot//initramfs-0-rescue-da769618583f4a5ba0144f95b04a2879.img
extracting /boot//initramfs-5.14.0-39.el9.x86_64.img
extracting /boot//initramfs-5.14.0-39.el9.x86_64kdump.img

gzip: stdin: not in gzip format
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Duration: 20s
::   Assertions: 8 good, 0 bad
::   RESULT: PASS (Do the keylime setup)


::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Add keylime tenant
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ 17:10:29 ] :: [  BEGIN   ] :: Running 'cat > script.expect <<_EOF
set timeout 20
spawn keylime_tenant -v 127.0.0.1 -t 127.0.0.1 -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000 --allowlist allowlist.txt --exclude excludelist.txt --include payload --cert default -c add
expect "Please enter the password to decrypt your keystore:"
send "keylime
"
expect eof
_EOF'
:: [ 17:10:29 ] :: [   PASS   ] :: Command 'cat > script.expect <<_EOF
set timeout 20
spawn keylime_tenant -v 127.0.0.1 -t 127.0.0.1 -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000 --allowlist allowlist.txt --exclude excludelist.txt --include payload --cert default -c add
expect "Please enter the password to decrypt your keystore:"
send "keylime
"
expect eof
_EOF' (Expected 0, got 0)
:: [ 17:10:29 ] :: [  BEGIN   ] :: Running 'expect script.expect'
spawn keylime_tenant -v 127.0.0.1 -t 127.0.0.1 -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000 --allowlist allowlist.txt --exclude excludelist.txt --include payload --cert default -c add
Using config file /etc/keylime.conf
2022-01-05 17:10:29.493 - keylime.tpm - INFO - TPM2-TOOLS Version: 5.0
2022-01-05 17:10:29.496 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca
2022-01-05 17:10:29.497 - keylime.registrar_client - WARNING - TLS is enabled.
2022-01-05 17:10:29.497 - keylime.registrar_client - INFO - Setting up client TLS...
/usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2022-01-05 17:10:29.573 - keylime.tenant - INFO - TPM PCR Mask from policy is 0x408000
2022-01-05 17:10:29.573 - keylime.tenant - INFO - TPM PCR Mask from policy is 0x808000
Please enter the password to decrypt your keystore: 
2022-01-05 17:10:29.675 - keylime.ca-util - INFO - Creating cert package for d432fbb3-d2f1-4a97-9ef7-75bd81c00000 in d432fbb3-d2f1-4a97-9ef7-75bd81c00000-pkg.zip
2022-01-05 17:10:29.754 - keylime.ca-util - INFO - Creating cert package for RevocationNotifier in RevocationNotifier-pkg.zip
/usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
<Response [200]>
/usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2022-01-05 17:10:30.698 - keylime.tpm - WARNING - PCR #0 in quote not found in tpm_policy, skipping.
2022-01-05 17:10:30.699 - keylime.tenant - WARNING - DANGER: EK cert checking is disabled and no additional checks on EKs have been specified with ek_check_script option. Keylime is not secure!!
2022-01-05 17:10:30.699 - keylime.tenant - INFO - Quote from 127.0.0.1 validated
:: [ 17:10:30 ] :: [   PASS   ] :: Command 'expect script.expect' (Expected 0, got 0)
/usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
:: [ 17:10:35 ] :: [  BEGIN   ] :: Running 'keylime_tenant -c cvlist'
Using config file /etc/keylime.conf
2022-01-05 17:10:36.181 - keylime.tpm - INFO - TPM2-TOOLS Version: 5.0
2022-01-05 17:10:36.184 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca
2022-01-05 17:10:36.184 - keylime.tenant - WARNING - Using default UUID d432fbb3-d2f1-4a97-9ef7-75bd81c00000
/usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2022-01-05 17:10:36.261 - keylime.tenant - INFO - From verifier 127.0.0.1 port 8881 retrieved: "{'code': 200, 'status': 'Success', 'results': {'uuids': [['d432fbb3-d2f1-4a97-9ef7-75bd81c00000']]}}"
:: [ 17:10:36 ] :: [   PASS   ] :: Command 'keylime_tenant -c cvlist' (Expected 0, got 0)
2022-01-05 17:10:36.261 - keylime.tenant - INFO - From verifier 127.0.0.1 port 8881 retrieved: "{'code': 200, 'status': 'Success', 'results': {'uuids': [['d432fbb3-d2f1-4a97-9ef7-75bd81c00000']]}}"
:: [ 17:10:36 ] :: [   PASS   ] :: File '/var/tmp/rlRun_LOG.AgyL1SAD' should contain '{'code': 200, 'status': 'Success', 'results': {'uuids':.*'d432fbb3-d2f1-4a97-9ef7-75bd81c00000'' 
:: [ 17:10:36 ] :: [  BEGIN   ] :: Running 'keylime_tenant -c status -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000'
Using config file /etc/keylime.conf
2022-01-05 17:10:36.780 - keylime.tpm - INFO - TPM2-TOOLS Version: 5.0
2022-01-05 17:10:36.783 - keylime.tenant - INFO - Setting up client TLS in /var/lib/keylime/cv_ca
2022-01-05 17:10:36.783 - keylime.registrar_client - WARNING - TLS is enabled.
2022-01-05 17:10:36.783 - keylime.registrar_client - INFO - Setting up client TLS...
/usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2022-01-05 17:10:36.859 - keylime.tenant - INFO - {"code": 200, "status": "Agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000 exists on registrar 127.0.0.1 port 8891.", "results": {"d432fbb3-d2f1-4a97-9ef7-75bd81c00000": {"aik_tpm": "ARgAAQALAAUAcgAAABAAFAALCAAAAAAAAQDTYljsAACsaGr/3ZYFpkaUyaSXX3ukGqRGkiY8WhBLsG0svlFVGTXL4b9IcNF6SWnwKwZKmJ73tb5rO1vg8HFu1MboQUlHPmb5/P6LpjOT2DZP5WZGBA18y2bzcJ7k3KMO8bolKbQtjl2toHVoBZ2QtwI+ggb3CMNzQEw+w5Q68/M/fefcukq06arlol1GHuhFicnBgmfaJd0bS3PrAtTFrIzOPLBcYjnDY95isDeDLTPFDiX0rVDg7+gC7p+QtltfQa0SkulH+ZrV5ueUyiaMYJHfKahHC9GgxNM1IPQv2lz/MeBonZyg9d68CBP79aU8AHRRPO95iXvnEqlP4KtN", "ek_tpm": "AToAAQALAAMAsgAgg3GXZ0SEs/gakMyNRqXXJP1S124GUgtk8qHaGzMUaaoABgCAAEMAEAgAAAAAAAEArptNXgABJ7wYSQw24GSsbmQBGHkAgKiq8mgq/5AJd4vpA0pYXTrjnptgGi7fWIvWRS4/PZEh9Tog6N3B0ytQCkqbHpHcXVWfaqpby/AYVSDm9nmEb4myMsC/PWPQK0Tn1wVGt2ueXnIOCcFKv6/iEJkYKfJdy1+ib37ihVmqBT9RslfS8a8Gp4zeIzPfuUXmPaft6LsXySMRyWmgDAhPe4iworAny3FGanErSjGYmyZQngUs6Pbbz6CjkmKLyKryQG+RWaOkQ8othj2nmRf7FyRVOPfmAzCmGXiqtKcG/ZazBrYjo63i9ZWOYng8RziPDjeagqoFQT+i+sgslBgUhw==", "ekcert": "MIIEGTCCAoGgAwIBAgIBBTANBgkqhkiG9w0BAQsFADAYMRYwFAYDVQQDEw1zd3RwbS1sb2NhbGNhMB4XDTE5MDYxOTEwNDIwN1oXDTI5MDYxNjEwNDIwN1owODE2MDQGA1UEAxMtdHBtLXBiZDM6ZmNhZGE3NmEtYzg2Ny00NmYxLWI3NWMtZTc4MDU0NGU4ODgyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArptNXgABJ7wYSQw24GSsbmQBGHkAgKiq8mgq/5AJd4vpA0pYXTrjnptgGi7fWIvWRS4/PZEh9Tog6N3B0ytQCkqbHpHcXVWfaqpby/AYVSDm9nmEb4myMsC/PWPQK0Tn1wVGt2ueXnIOCcFKv6/iEJkYKfJdy1+ib37ihVmqBT9RslfS8a8Gp4zeIzPfuUXmPaft6LsXySMRyWmgDAhPe4iworAny3FGanErSjGYmyZQngUs6Pbbz6CjkmKLyKryQG+RWaOkQ8othj2nmRf7FyRVOPfmAzCmGXiqtKcG/ZazBrYjo63i9ZWOYng8RziPDjeagqoFQT+i+sgslBgUhwIDAQABo4HNMIHKMBAGA1UdJQQJMAcGBWeBBQgBMFIGA1UdEQEB/wRIMEakRDBCMRYwFAYFZ4EFAgEMC2lkOjAwMDAxMDE0MRAwDgYFZ4EFAgIMBXN3dHBtMRYwFAYFZ4EFAgMMC2lkOjIwMTcwNjE5MAwGA1UdEwEB/wQCMAAwIgYDVR0JBBswGTAXBgVngQUCEDEOMAwMAzIuMAIBAAICAJYwHwYDVR0jBBgwFoAUC76nO8Z3SOYicOU/tWx3k5u6A4kwDwYDVR0PAQH/BAUDAwcgADANBgkqhkiG9w0BAQsFAAOCAYEArwBulIo+dnzXXo3PMt1x6aJpjPCfyr9Av0QVkXfr/5PRg5bYiMVPkzH2J6PCyNfOuYJJ9fU9eKIEpTENLvgS7p54SdV59xBl44JqbMoDM2BhYz7zRKgaovJaPEG+53JnO6IMptz6pT7BzhMhiwh5GTS4AvS/LzvNvKA/p8CvoC8auoIw7TtcRovh7Kpq2LuWMd1aH3BbEUHh9ervL1gHlJC+ImG/oD0B5J7x7NATVgmKZzP0XnTez+C4nnoKdtP/ZxR0LFhyrYeaEJ5S2BMlrllnwq7dhHoE1/9Ed0ClKkvUsQAOAtPoYtRy7yi9p1UTmdxFNPwXXu1Ie4Wt0uJtPbiNDWnCjD4SN9AfI+X1whrM3Ss/1eiNC1baSq8GfQJBqkTSNeKagadIiXHnJ5cp1PdIPQEdTGKRd47CSxDoZhqZu08YUZcKybr4fZ0cTYjBwidVak5S2x8JEJ2rdxMEAKDyKARjEAC0J/wuKgxK1biQOAXobEJl4AvrfZbZAvI2", "ip": "127.0.0.1", "port": 9002, "regcount": 1, "operational_state": "Registered"}}}
/usr/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
2022-01-05 17:10:36.950 - keylime.tenant - INFO - Agent Info:
{"d432fbb3-d2f1-4a97-9ef7-75bd81c00000": {"operational_state": "Invalid Quote", "v": "Kg2uD39aSy/2BwT2q5n7eYOJ5iz+ChPXZBYJkcgajco=", "ip": "127.0.0.1", "port": 9002, "tpm_policy": "{\"22\": [\"0000000000000000000000000000000000000001\", \"0000000000000000000000000000000000000000000000000000000000000001\", \"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001\", \"ffffffffffffffffffffffffffffffffffffffff\", \"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\", \"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\"], \"15\": [\"0000000000000000000000000000000000000000\", \"0000000000000000000000000000000000000000000000000000000000000000\", \"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\"], \"mask\": \"0x408400\"}", "vtpm_policy": "{\"23\": [\"ffffffffffffffffffffffffffffffffffffffff\", \"0000000000000000000000000000000000000000\"], \"15\": [\"0000000000000000000000000000000000000000\"], \"mask\": \"0x808000\"}", "meta_data": "{\"cert_serial\": 2, \"subject\": \"OU=53,O=MITLL,L=Lexington,ST=MA,CN=d432fbb3-d2f1-4a97-9ef7-75bd81c00000,C=US\"}", "allowlist_len": 6, "mb_refstate_len": 0, "accept_tpm_hash_algs": ["sha512", "sha384", "sha256", "sha1"], "accept_tpm_encryption_algs": ["ecc", "rsa"], "accept_tpm_signing_algs": ["ecschnorr", "rsassa"], "hash_alg": "sha256", "enc_alg": "rsa", "sign_alg": "rsassa", "verifier_id": "default", "verifier_ip": "127.0.0.1", "verifier_port": 8881, "severity_level": 6, "last_event_id": "ima.pcr_mismatch"}}
:: [ 17:10:37 ] :: [   PASS   ] :: Command 'keylime_tenant -c status -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000' (Expected 0, got 0)
:: [ 17:10:37 ] :: [   FAIL   ] :: File '/var/tmp/rlRun_LOG.HHUXpNXC' should contain '"operational_state": "Get Quote"' 
:: [ 17:10:37 ] :: [   FAIL   ] :: File /var/tmp/test_payload_file should exist 
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Duration: 8s
::   Assertions: 5 good, 2 bad
::   RESULT: FAIL (Add keylime tenant)

Relevant logs

kernel 5.14.0-39.el9.x86_64 I am not facing this issue when using keylime emulator instead of QEMU TPM

With git bisect we have identified the following commit as the first one introducing the issue. 80990a75d0444c9c4b854000c1482951cc130ad5 (“algorithms: extend Hash class to simplify computing hash values”) however with this commit the behavior is not exactly as described (there is a traceback due to missing codecs module and even after fixing the import there are other issues.) With the subsequent commit applied (517a7e89fd9ff9f93d831d21db7f285cb9d1b382) the behavior matches the reported one.

[sergio-correia]

Yeah, I got many entries like these:

10 0000000000000000000000000000000000000000 ima-sig sha1:0000000000000000000000000000000000000000 /var/tmp/restraintd/logs/137557067/task.log
10 0000000000000000000000000000000000000000 ima-sig sha1:0000000000000000000000000000000000000000 /var/tmp/restraintd/logs/137557067/harness.log

[THS-on]

Nice. Can you check if your tested IMA log contains entries like this (the hash is only zeros):

10 0000000000000000000000000000000000000000 ima-ng sha1:0000000000000000000000000000000000000000 /var/tmp/restraintd/logs/137573676/harness.log

If yes this I’ll make this a PR.