OpenCV 360 Video Stabilisation


Now that we have a way to plot great circles on to equirectangular planes, how do we use that to detect the pitch and roll of the camera and consequently stabilise the video?

Draw Curves

First, we need to draw the curves as opencv images:

# import the plotting module from the previous blog post
import plot
import numpy as np
import cv2
import math

def getCvCurve(roll, pitch, lineRes, resx):
	# we add 90 degrees to standardise the roll according to mainstream use
	roll = math.radians(90 + roll)
	pitch = math.radians(pitch)
	curve = plot.getCurve(roll,pitch,lineRes)
	resy = resx/2
	ptsArr = []

	for i in range(0,len(curve)):
		x = normalise(curve[i].x, 'x', resx)
		y = normalise(curve[i].y, 'y', resy)
		ptsArr.append([x,y])

	pts = np.array(ptsArr, dtype=np.int32)
	return pts
	
def normalise (val, ax, res):
	if ax == "x":
		val = val + math.pi
		thispt = val / (2*math.pi) * res
		return thispt
	if ax == "y":
		val = val + math.pi/2
		thispt = val / math.pi * res
		return thispt

def showCurve(pts, resx):
	resy = resx/2
	img = np.zeros((resy,resx), np.uint8)
	cv2.polylines(img, [pts], False,(255,255,255),1)
	cv2.imshow('dst_rt', img)
	cv2.waitKey(0)

We calculate the curves according to the roll and pitch provided. Then we normalise the curves such that when they are drawn, they appear from 0-2pi and 0-pi on the x and y axis respectively. Drawing these points as polylines using opencv generates an image like this.

OpenCV Plot

10° roll, 15° pitch

Using a script, we can generate and store all the possible curves in low resolution. The module to do that is available here.

Once we have the curves, we can write another script to compare the curves with the horizon in the video. Comparing images for similarities is a well-documented procedure and I will leave it to the reader to write the code. The gist is below.

We can do the comparison through the following steps:

  • Contrast out the horizon curve from the video frame. cv2.cvtColor and cv2.threshold functions from the OpenCV library are useful functions for this.
  • Do a pixel comparison for this frame with each curve in your collection. The highest bitwise_and with the curve is the curve which matches the image and contains the roll and pitch value for that frame.

Optimisations

This is a very processor intensive process and can be extremely slow because of the sheer number of comparisons. Luckily, we can do some basic optimisations to speed up. For example, we know the horizon curve won’t change between frames by a very large amount so it is likely that a match will be within +-5 degrees roll/pitch. Also, it is likely that the video starts with [0,0] roll and pitch so the first frame comparison can be shortened as well.

Stabilising

Once we have the roll and pitch for each frame, it is a simple step of counter-rotating the [Open:Web]GL sphere on which the 360 video is being projected against the roll-pitch values gleaned from the process above. The result: smooth 360 Video where you can stabilise the POV to a great extent!

Till next time.

-Taha


comments powered by Disqus