Faster face detection on video stream

• face_recognition version: 1.2.2
• Python version: 3.5
• Operating System: Ubuntu 16.04

Description

Is there any way to make the face detection part run faster? I’m running this for my rtsp camera stream and the video is very slow - each frame takes 2 seconds to process. I managed to accelerate my video stream to 3 FPS by using “cnn” face detection method, by resizing the processing frame by a factor of 0.33 and by moving all the processing function calls into the separate thread.

I measured time it takes to process one frame - and it is somewhere around 350 ms. But this is done in the separate thread (not the one that handles video display), so I don’t understand why does this slow my video so much.

I am running this program on Nvidia Jetson TX1.

What I Did

# The main cycle part
for face_location, name in zip(face_locations, face_names):
    top, right, bottom, left = face_location
    cv2.rectangle(img, (left * 3, top * 3), (right * 3, bottom * 3), (0, 0, 230), 3)
if obama_available:
    oth = Thread(target=obama_thread, args=[copy.deepcopy(img)])
    oth.daemon = True
    oth.start()

## in the other thread
def obama_thread(img):
    obama_available = False
    small_frame = cv2.resize(img, (0, 0), fx=0.33, fy=0.33)
    rgb_small_frame = small_frame[:, :, ::-1]
    face_locations_temp = face_recognition.face_locations(rgb_small_frame, 1, "cnn")
    face_locations.clear()
    for (top, right, bottom, left) in face_locations_temp:
        face_locations.append([top, right, bottom, left])
    obama_available = True

[KLelkov]

@VellalaVineethKumar Hey! The very first thing you need to do is to get rtsp link to your camera’s video stream. You can google how to get it for your exact camera module, or simply search it in its settings. In general, it looks like this rtsp://1. 10.2.0.10/live

You can check if your link is correct by opening it in VLC player (File -> play from URL). Or you can test with this free rtsp stream sample: rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov

Alright, after you got your link - you need to copy this example program: https://github.com/ageitgey/face_recognition/blob/master/examples/facerec_from_webcam_faster.py

Check lines 13-14

# Get a reference to webcam #0 (the default one)
video_capture = cv2.VideoCapture(0)

This is how you get a video stream from webcam. But this does NOT work for IP cameras. There’s the function you will need:

def open_cam_rtsp(uri, width, height, latency):
    gst_str = ('rtspsrc location={} latency={} ! '
               'rtph264depay ! h264parse ! omxh264dec ! '
               'nvvidconv ! '
              'video/x-raw, width=(int){}, height=(int){}, 
               'format=(string)BGRx ! '
               'videoconvert ! appsink').format(uri, latency, width, height)
    return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

Now you can open your camera stream by providing url to rtsp stream as

video_capture = open_cam_rtsp("rtsp://192.168.1.4/live1.sdp", 1280, 720, 200)

The rest of the code should be pretty straightforward. Here you load your known-people-faces. Notice that “obama.jpg” is actually a path to the image. In this case, the image is located in the same folder as the python script.

# Load a sample picture and learn how to recognize it.
obama_image = face_recognition.load_image_file("obama.jpg")

Let’s say you have a photo of your friend Steve in “known_faces/steve.png”. The code will be as follows:

steve_image = face_recognition.load_image_file("known_faces/steve.png")

[VellalaVineethKumar]

Thanks for your quick response! when i run the facerec_from_webcam_faster.py from examples python@cse-w1:~/face_recognition-master/examples$ python facerec_from_webcam_faster.py

and i tried to print the return string in the function: rtspsrc location=rtsp://admin:admin@10.2.3.177 latency=200 ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, width=(int)1280, height=(int)720format=(string)BGRx ! videoconvert ! appsink Traceback (most recent call last): File “facerec_from_webcam_faster.py”, line 56, in <module> small_frame = cv2.resize(frame, (0, 0), fx=0.25, fy=0.25) cv2.error: OpenCV(4.0.0) /io/opencv/modules/imgproc/src/resize.cpp:3784: error: (-215:Assertion failed) !ssize.empty() in function ‘resize’

my function call link=“rtsp://admin:admin@10.2.3.10” video_capture = open_cam_rtsp(link, 1280, 720, 200) and i get ret value as false in the VideoCapture() .

in the return function i removed cv2.CAP_GSTREAMER in the retuen statement as i got an error when i ran on my webcam #video_capture = cv2.VideoCapture(0, cv2.CAP_GSTREAMER)

please help!