Using OpenCV to use great circle plots for horizon detection

By taha 2 comments

Now that we have a way to plot great circles on to equirectangular planes, how do we use that to detect the pitch and roll of the camera and consequently stabilise the video?

Draw Curves

First, we need to draw the curves as opencv images:

# import the plotting module from the previous blog post
import plot
import numpy as np
import cv2
import math

def getCvCurve(roll, pitch, lineRes, resx):
    # we add 90 degrees to standardise the roll according to mainstream use
    roll = math.radians(90 + roll)
    pitch = math.radians(pitch)
    curve = plot.getCurve(roll,pitch,lineRes)
    resy = resx/2
    ptsArr = []

    for i in range(0,len(curve)):
        x = normalise(curve[i].x, 'x', resx)
        y = normalise(curve[i].y, 'y', resy)
        ptsArr.append([x,y])

    pts = np.array(ptsArr, dtype=np.int32)
    return pts
    
def normalise (val, ax, res):
    if ax == "x":
        val = val + math.pi
        thispt = val / (2*math.pi) * res
        return thispt
    if ax == "y":
        val = val + math.pi/2
        thispt = val / math.pi * res
        return thispt

def showCurve(pts, resx):
    resy = resx/2
    img = np.zeros((resy,resx), np.uint8)
    cv2.polylines(img, [pts], False,(255,255,255),1)
    cv2.imshow('dst_rt', img)
    cv2.waitKey(0)

We calculate the curves according to the roll and pitch provided. Then we normalise the curves such that when they are drawn, they appear from 0-2pi and 0-pi on the x and y axis respectively. Drawing these points as polylines using opencv generates an image like this.

OpenCV Plot

10° roll, 15° pitch

Using a script, we can generate and store all the possible curves in low resolution. The module to do that is available here.

Once we have the curves, we can write another script to compare the curves with the horizon in the video. Comparing images for similarities is a well-documented procedure and I will leave it to the reader to write the code. The gist is below.

We can do the comparison through the following steps:

  • Contrast out the horizon curve from the video frame. cv2.cvtColor and cv2.threshold functions from the OpenCV library are useful functions for this.
  • Do a pixel comparison for this frame with each curve in your collection. The highest bitwise_and with the curve is the curve which matches the image and contains the roll and pitch value for that frame.

Optimisations

This is a very processor intensive process and can be extremely slow because of the sheer number of comparisons. Luckily, we can do some basic optimisations to speed up. For example, we know the horizon curve won’t change between frames by a very large amount so it is likely that a match will be within ±5 degrees roll/pitch. Also, it is likely that the video starts with [0,0] roll and pitch so the first frame comparison can be shortened as well.

Stabilising

Once we have the roll and pitch for each frame, it is a simple step of counter-rotating the [Open:Web]GL sphere on which the 360 video is being projected against the roll-pitch values gleaned from the process above. The result: smooth 360 Video where you can stabilise the POV to a great extent!

Till next time.

2 Comments

Justin

Apr 4, 2020, 2:20 am Reply

Hey there 🙂

Your wordpress site is very sleek – hope you don’t mind me asking what theme you’re
using? (and don’t mind if I steal it? :P)

I just launched my site –also built in wordpress like yours– but the theme slows (!) the site down quite a bit.

In case you have a minute, you can find it by searching for “royal cbd” on Google (would appreciate any feedback)
– it’s still in the works.

Keep up the good work– and hope you all take care of yourself during the coronavirus scare!

taha

Apr 4, 2020, 8:18 pm Reply

Hello,

I am using the theme Lessons found here with some small modifications: https://wpxon.com/

You are welcome to use it of course (if the theme allows commercial use) since I don’t own the theme myself 🙂

Leave a Reply