Using OpenCV to use great circle plots for horizon detection
Now that we have a way to plot great circles on to equirectangular planes, how do we use that to detect the pitch and roll of the camera and consequently stabilise the video?
First, we need to draw the curves as opencv images:
# import the plotting module from the previous blog post import plot import numpy as np import cv2 import math def getCvCurve(roll, pitch, lineRes, resx): # we add 90 degrees to standardise the roll according to mainstream use roll = math.radians(90 + roll) pitch = math.radians(pitch) curve = plot.getCurve(roll,pitch,lineRes) resy = resx/2 ptsArr =  for i in range(0,len(curve)): x = normalise(curve[i].x, 'x', resx) y = normalise(curve[i].y, 'y', resy) ptsArr.append([x,y]) pts = np.array(ptsArr, dtype=np.int32) return pts def normalise (val, ax, res): if ax == "x": val = val + math.pi thispt = val / (2*math.pi) * res return thispt if ax == "y": val = val + math.pi/2 thispt = val / math.pi * res return thispt def showCurve(pts, resx): resy = resx/2 img = np.zeros((resy,resx), np.uint8) cv2.polylines(img, [pts], False,(255,255,255),1) cv2.imshow('dst_rt', img) cv2.waitKey(0)
We calculate the curves according to the roll and pitch provided. Then we normalise the curves such that when they are drawn, they appear from 0-2pi and 0-pi on the x and y axis respectively. Drawing these points as polylines using opencv generates an image like this.
10° roll, 15° pitch
Using a script, we can generate and store all the possible curves in low resolution. The module to do that is available here.
Once we have the curves, we can write another script to compare the curves with the horizon in the video. Comparing images for similarities is a well-documented procedure and I will leave it to the reader to write the code. The gist is below.
We can do the comparison through the following steps:
- Contrast out the horizon curve from the video frame. cv2.cvtColor and cv2.threshold functions from the OpenCV library are useful functions for this.
- Do a pixel comparison for this frame with each curve in your collection. The highest bitwise_and with the curve is the curve which matches the image and contains the roll and pitch value for that frame.
This is a very processor intensive process and can be extremely slow because of the sheer number of comparisons. Luckily, we can do some basic optimisations to speed up. For example, we know the horizon curve won’t change between frames by a very large amount so it is likely that a match will be within ±5 degrees roll/pitch. Also, it is likely that the video starts with [0,0] roll and pitch so the first frame comparison can be shortened as well.
Once we have the roll and pitch for each frame, it is a simple step of counter-rotating the [Open:Web]GL sphere on which the 360 video is being projected against the roll-pitch values gleaned from the process above. The result: smooth 360 Video where you can stabilise the POV to a great extent!
Till next time.