random video interrupts

Hi jlaine! Sometimes, when we are video streaming, the frames stops. we track the problem to the rtcrtpreceiver.py file and it seems some frames are None and so they don’t inserted in the queue. Do you have any ideas why this happens and what should we do to solve the problem? Is it a bug or an expected behavior? Here is the related section in rtcrtpreceiver.py module for convenience:

async def _handle_rtp_packet(self, packet):
     self.__log_debug('< %s', packet)
     if packet.payload_type in self._decoders:
         decoder = self._decoders[packet.payload_type]
         loop = asyncio.get_event_loop()

         # RTCP
         if self.__remote_ssrc is None:
             self.__remote_ssrc = packet.ssrc
             self.__remote_counter = LossCounter(packet.sequence_number)
         else:
             self.__remote_counter.add(packet.sequence_number)

         if self._kind == 'audio':
             # FIXME: audio should use the jitter buffer!
             audio_frame = await loop.run_in_executor(None, decoder.decode, packet.payload)
             await self._track._queue.put(audio_frame)
         else:
             # check if we have a complete video frame
             self._jitter_buffer.add(packet.payload, packet.sequence_number, packet.timestamp)
             payloads = []
             got_frame = False
             last_timestamp = None
             for count in range(self._jitter_buffer.capacity):
                 frame = self._jitter_buffer.peek(count)
                 if frame is None:
                     break
                 if last_timestamp is None:
                     last_timestamp = frame.timestamp
                 elif frame.timestamp != last_timestamp:
                     got_frame = True
                     break
                 payloads.append(frame.payload)

             if got_frame:
                 self._jitter_buffer.remove(count)
                 video_frames = await loop.run_in_executor(None, decoder.decode, payloads)
                 for video_frame in video_frames:
                     await self._track._queue.put(video_frame)

I think the important section is the last else block, where you are putting the video_frames to the queue. It creates the video frames only when got_frame is True and other frames get discarded. Our clients are android smart phones and we are testing in a local network.

Based on another issue we changed the buffer capacity from 32 to 64 and we got slightly better results. Do you think the buffer capacity is an important factor here?

[JRafiei]

Yes, sure! After testing with the latest version, the video quality has increased significantly and it is more stable. Sometimes the video get pixelated and stopped for a short time but it resumed to the normal state after that. Thanks a lot for your efforts. during testing the video call we get two new errors that may be informative.

Task exception was never retrieved
future: <Task finished coro=<RTCDtlsTransport.__run() done, defined at /usr/local/lib/python3.6/site-packages/aiortc-0.9.1-py3.6-linux-x86_64.egg/aiortc/rtcdtlstransport.py:365> exception=Error('replay check failed (bad index)',)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/aiortc-0.9.1-py3.6-linux-x86_64.egg/aiortc/rtcdtlstransport.py", line 368, in __run
await self._recv_next()
File "/usr/local/lib/python3.6/site-packages/aiortc-0.9.1-py3.6-linux-x86_64.egg/aiortc/rtcdtlstransport.py", line 427, in _recv_next
data = self._rx_srtp.unprotect(data)
File "/usr/local/lib/python3.6/site-packages/pylibsrtp/__init__.py", line 178, in unprotect
return self.__process(packet, lib.srtp_unprotect)
File "/usr/local/lib/python3.6/site-packages/pylibsrtp/__init__.py", line 198, in __process
_srtp_assert(func(self._srtp[0], self._cdata, len_p))
File "/usr/local/lib/python3.6/site-packages/pylibsrtp/__init__.py", line 52, in _srtp_assert
raise Error(ERRORS[rc])
pylibsrtp.Error: replay check failed (bad index)

and

Exception in callback Transaction.__retry()
handle: <TimerHandle when=1566464.094515949 Transaction.__retry()>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/asyncio/events.py", line 127, in _run
self._callback(*self._args)
File "/usr/local/lib/python3.6/site-packages/aioice-0.6.2-py3.6.egg/aioice/stun.py", line 257, in __retry
self.__future.set_exception(exceptions.TransactionTimeout())
asyncio.base_futures.InvalidStateError: invalid state

After these errors, video playback completely stops and the ice connection become disconnected and then failed.

[jlaine]

Ok can we please open separate tickets to track the new issues?