High CPU use in crcmod during large datachannel buffer encoding

While encoding VP8 video from incoming video frames to a single remote peer, I see high CPU use (running on an i7 CPU). H264 encoding is not as bad, but majority of time in same place.

Image frame format is 640x512, RGB24 raw pixels

Using py-spy live profiling tool, looks like CRC computation is using a majority of the CPU time. Any ideas why?

(If CRC computation is done in Python, may be an easy optimization to do it in C/C++?)

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                                                                                                                                                                      
 62.00%  63.00%    2.88s     2.89s   _crc32r (crcmod/_crcfunpy.py)
  9.00%   9.00%   0.370s    0.370s   encode (aiortc/codecs/vpx.py)
  5.00%   5.00%   0.200s    0.200s   sendto (/usr/lib/python3.6/asyncio/selector_events.py)
  4.00%  76.00%   0.300s     3.73s   _transmit (aiortc/rtcsctptransport.py)
  2.00%   2.00%   0.050s    0.050s   recv_multipart (zmq/sugar/socket.py)
  2.00%   2.00%   0.030s    0.030s   _update_rto (aiortc/rtcsctptransport.py)
  2.00%  99.00%   0.100s     4.95s   _run_once (/usr/lib/python3.6/asyncio/base_events.py)
  2.00%   2.00%   0.030s    0.030s   __init__ (aiortc/rtcsctptransport.py)
  1.00%   1.00%   0.020s    0.020s   current_datetime (aiortc/clock.py)
...

Here’s an icecycle graph generated by py-spy

[jlaine]

I have just tried installing / uninstalling crcmod in various configurations, and the only condition under which I did not end up with the C extension is if python3.x-dev (matching the Python version) is not installed.

Note: if you ever installed crcmod before you installed python3.x-dev the wheel won’t get rebuilt at your next install, you will from then on be installing a cached version from $HOME/.cache/pip. So the full fix is:

• pip uninstall crcmod
• apt-get install python3-dev
• rm -rf $HOME/.cache/pip
• pip install crcmod

I have made a change which spews a warning if the C extension is not present, see 9430449cfa7f2f9f2b1d172e5d68a52342536d13

[inactivist]

I have made a change which spews a warning if the C extension is not present, see 9430449

Beautiful, thank you!