Discussion: Current state of IP Cameras (Secure Video & custom RTP implementation)

I wanted to start a discussion about the current state of IP cameras and some crazy ideas from my side 🙃

At the time of writing our implementation looks like the following

• We have reliable and working snapshot request (since v0.5.1 we even support bridging multiple cameras)
• Streaming video is pretty much sorted out and works most of the time. The RTP/RTCP protocol is probably not 100% spec conform but it somehow works (most projects using ffmpeg). The only thing which isn’t really supported is the reconfigureStream request, to dynamically change the bitrate according to the network condition (limitation of ffmpeg).
• It is possible managing support for one way audio (microphone camera -> HomeKit device). But most of the time audio is choppy and doesn’t really work reliably.
• Two way audio is pretty much unresolved. One big issue is that ffmpeg firstly doesn’t support incoming audio frames to be encoded and containered (in the same process) and secondly doesn’t support multiplexing RTP and RTCP on a single port, resulting in some hacky implementations using UDP proxies and the inability of RTCP packets arriving. Most of them really do not work very well.

Generally the RTP implementation is a key point for a reliable implementation of IP cameras. I though of starting a project trying to create a custom RTP implementation as required by the HAP spec. Meaning adhering to RFC 3550, RFC 3551, RFC 4585, RFC 5104, multiplexing RTP and RTCP defined in RFC 5761, payload formats for H.264 (RFC 6184) and Opus voice (RFC 7587) and securing the connection according to RFC 3711 and RFC 6188. Then implementing a video/audio pipeline using gstreamer for better bitrate control. I would focus my work mainly on a Raspberry Pi and it’s PiCam though keeping the API as abstract as possible.

I wanted to start this discussion to get some feedback from people with more experience, who maybe dissuade me and prevent me wasting my time lol 😅 I currently have a basic understanding of the fundamental RTP RFCs, though no concrete understanding for stuff like SRTP. I also don’t know how time intensive this would be, how motivated I can stay and even wether I can manage to invest that much time into this. I just wanted to get my crazy idea out and start a conversation with people, collecting some knowledge and/or experience.

[Supereg]

Some good news. The protocol for Secure-Video cameras is pretty much figured out, which means any HAP-NodeJS/homebridge plugin will soon be able to add support for secure-video.

For anyone interested, I created a little unofficial specification which should contain all known information (could be interesting for any other HAP implementations out there).

There are currently two steps missing (the second one is also a bit of sad news, which currently holds us back):

• First of all we still need to implement a proper API into HAP-NodeJS. I have some sketches here and I think we can abstract quite a lot (similar to the RemoteController). So nobody has to deal with HDS payloads and pretty much only media creation.
• And the second rather big thing: I couldn’t figure out how to create proper fragmented mp4, mostly because of time reasons. It seems to be doable cause it is very similar to that what HLS defines. So ffmpeg or gstreamer (which apple seems to favor for ADK) should somehow be able to create those segments. For anyone interested in contributing to this step, I added example recordings to the specification repo.

Have a nice day ~ Supereg

[iko3]

Thanks for all the efforts, but Is there any estimate on when Secure Video will be officially supported in the HAP-NodeJS library?