Changes to `web3.sha3`

What was wrong?

Currently, the implementation of web3.sha3 has some unexpected behavior and non ideal behavior.

1. It uses the web3_sha3 RPC endpoint which allows for different providers to return different values.
2. By default it expects the value to be hashed to be passed in as a hex encoded string.
3. If you specify any encoding besides hex, it treats it as if it was bytes.

How can it be fixed?

1. Use eth_utils.keccak to compute the hash.

This is more reliably going to produce the result that users expect as well as reducing RPC request overhead.

2. Change the default encoding to bytes.

This would be a breaking change so figuring out how to deprecate the current behavior while giving users the appropriate time to upgrade would be ideal. This could be done in a two step process.

We should have an explicit list of encodings that are supported and have logic for dealing with those appropriately. I suggest bytes, utf8, and hex.

[carver]

Went with hexstr. It grew on me.

[carver]

Encoding Background

There is a caveat to # 2 above, which is that if someone passes bytes, then the encoding is meaningless and should be ignored. Also, “encoding” as the keyword argument is confusing here, because it actually means a different thing for each of these options:

1. bytes (no action necessary)
2. utf8 (take this string of characters, and encode them using utf-8)
3. hex (take this hex representation of bytes, and decode them into their underlying bytes)

"txt"   == encode utf-8 ==>   b'\x74\x78\x74'   == encode hex str ==>   "0x747874"
"txt"   <== decode utf-8 ==   b'\x74\x78\x74'   <== decode hex str ==   "0x747874"

Also confusing is that python supports hex literals, which seems reasonable to support too, like: web3.sha3(0x74657874). The primitive here is int. Again, like bytes, if an int is passed, then “encoding” should be ignored.

The “encoding” here is only ever referring to the type of string we’re looking at. We can (and should) auto-detect all the rest. We’d be done, if it wasn’t for those meddling kids and their hex-string encodings.

New API

So what I’m ramping up to is a new API (the breaking change):

def sha3(primitive=None, text=None, hex=None)

My least favorite part is overwriting the builtin hex, but I really want a short syntax here. Even core python does stuff like this (see file kwarg in print).

So you could call any of:

web3.sha3(b'\x74\x78\x74')
web3.sha3(0x747874)
web3.sha3(hex="0x747874")
web3.sha3(text="txt")

This is concise, easy to read, and should cover all cases. It’s not terribly important that people understand that the text is encoded into utf8, because text in all clients should always use this same encoding.

Migration Plan

So the migration plan for web3.sha3 would be something like:

1. Leave encoding kwarg in place
2. Auto-detect bytes and int, and encode them appropriately, ignoring the ‘encoding’ kwarg
3. Enable using the new keywords hex and text
4. Issue a Deprecation warning for all sha3 calls, unless they are using the new functionality in 2 and 3
5. After some time (1 month?) raise an exception if users pass a string as the first positional arg
6. After even longer (3 months? certainly after requiring py3), remove the encoding keyword argument