Track hand rotational angle - Leap + Javascript

In this tutorial, we are going to teach you how you can track your hand rotation using Leap Motion Controller. Leap Motion Controller is a device that allows you to detect your hand motions and gestures, and perform any desired action on them. It provides SDK in Java, Javascript, Python, C++, and Unreal Engine. We will be using Javascript in this tutorial.

You can buy the Leap Motion Controller from their official website. After buying, connect the device with your computer using a USB or USB-C port. Then download the SDK from here and do the installation.

If you prefer the video tutorial:

From the above video, you can see what we are going to create. So you will need a Doctor Strange time-spell video and some images. You will also need the Leap JS library to connect with Leap Motion Controller. You can find all 3 in the source files below.

Setup the Project

The following code helps you to reverse a video and concatenate 2 videos using FFmpeg:

// reverse a video
// ffmpeg -i Pentagramm2_green.mp4 -vf reverse reversed.mp4

// concatenate 2 videos
// ffmpeg -i Pentagramm2_green.mp4 -i reversed.mp4 -filter_complex "[0:v] [0:a] [1:v] concat=n=2:v=1:a=1 [v] [a]" -map "[v]" -map "[a]" output.mp4

Then create a video tag for the time-spell video.

<video src="output.mp4" muted id="time-spell"></video>

After that, create a div to show the image and text below it.

<div style="position: absolute; right: 50px; top: 50px;">
    <img src="images/2021.JPG" id="image" />
    <h1 id="title">2021</h1>
</div>

Now include the Leap JS library in your project.

<script src="leap-0.6.4.min.js"></script>

Then apply some CSS styles to display the video, image, and text properly.

<style>
    /* remove margin and set background color as black */
    body {
        margin: 0px;
        background-color: black;
    }

    /* set width of time spell and its position */
    #time-spell {
        width: 1200px;
        position: absolute;
        left: -275px;
        top: 30px;
    }

    /* set width and height of image and fit inside container */
    #image {
        width: 800px;
        height: 600px;
        object-fit: contain;
    }

    /* set color, position, alignment, font size and background color of image text */
    #title {
        color: white;
        position: relative;
        bottom: 129px;
        text-align: center;
        font-size: 70px;
        background-color: rgba(0, 0, 0, 0.7);
    }
</style>

Then create an array of images and their text in Javascript.

<script>

    // list of images and their text
    var images = [{
        "src": "images/2021.JPG",
        "title": "2021"
    }];

    // index is out of bound right now
    var index = images.length;

</script>

After that, initialize some variables that we will be using throughout the tutorial.

// total duration of time spell video
var totalDuration = 40;

// state of time spell (back or forward)
var state = "";

// previous state of time spell
var previousState = "";

// hand rotation from Leap Motion Controller
var handRotation = "";

// is time spell start rotating
var isMediaPlaying = false;

// video tag
var media = document.getElementById("time-spell");

Leap motion controller connection

Now is the time to connect our Javascript app with Leap Motion Controller.

// setup Leap loop with frame callback function
var controllerOptions = {};

Leap.loop(controllerOptions, function(frame) {
    // get all hands in frame
    var hands = frame.hands;

    // if there is any hand
    if (hands.length > 0) {

        // get the first hand
        var hand = hands[0];

        // ignore if the hand is left
        if (hand.isLeft) {
            return false;
        }

        // get direction of hand (x,y,z co-ordinates)
        var axis = hand.direction;

        // get x-axis
        axis = axis[0];

        if (axis > 0 && axis < 0.9) {
            // if hand is rotated to right
            handRotation = "right";
        } else if (axis > -0.09 && axis < -0.01) {
            // if hand is in center (no rotation)
            handRotation = "center";

            media.pause();
        } else if (axis < 0 && axis < -0.5) {
            // if hand is rotated to left
            handRotation = "left";
        }
    } else {
        // if there is no hand in the frame, simply pause the time spell
        if (media != null) {
            media.pause();
        }
    }
});

For explanation, comments have been added with each line. Now we need to call a function every 0.5 seconds and rotate the time spell to the left if the hand is rotated to left, and right if the hand is rotated to right. Also, it will call a function to change the image as per hand rotation.

// called every 0.5 seconds
setInterval(function () {
    
    // if hand is rotating to right
    if (handRotation == "right") {
        
        // only called once to rotate time spell
        if (state == "") {
            // rotate to right
            mediaBackward();
        }

        // if time spell starts rotating
        if (isMediaPlaying) {
            // show next image
            nextImage();
        }

        // make it empty to give a delay to show next image
        handRotation = "";
    } else if (handRotation == "left") {

        // only called once to rotate time spell
        if (state == "") {
            // rotate to left
            mediaForward();
        }

        // if time spell starts rotating
        if (isMediaPlaying) {
            // show previous image
            previousImage();
        }

        // make it empty to give a delay to show previous image
        handRotation = "";
    } else if (handRotation == "center") {
        // pause the time spell, when hand is in center of Leap Motion Controller
        media.pause();
    }
}, 500);

Now we just need to create these 4 functions called from the above code.

// when hand is rotated right
function mediaBackward() {

    // "back" means reverse
    state = "back";

    // starts rotating time spell
    media.play();

    // current time of video
    var currentTime = media.currentTime;

    // if video stops playing
    if (currentTime <= 0) {
        // starts from middle to move right
        currentTime = totalDuration / 2;
    }

    // this condition becomes true only when the
    // time spell was rotating forward (left side)
    if (previousState != state) {
        // change direction of time spell
        var newTime = totalDuration - currentTime;
        media.currentTime = newTime;
    }
}

// when hand is rotated left
function mediaForward() {

    // "forward" means original video
    state = "forward";

    // starts rotating time spell
    media.play();

    // current time of video
    var currentTime = media.currentTime;

    // if video stops playing
    if (currentTime <= 0) {
        // starts from zero to move left
        currentTime = totalDuration;
    }

    // this condition becomes true only when the
    // time spell was rotating backward (right side)
    if (previousState != state) {
        // change direction of time spell
        var newTime = totalDuration - currentTime;
        media.currentTime = newTime;
    }
}

// show next image from images array
function nextImage() {
    // increment the index
    index++;

    // check if the index is out of bound
    if (index >= images.length || index < 0) {
        return false;
    }

    // get source of image
    var src = images[index].src;

    // get text of image
    var title = images[index].title;

    // update image
    document.getElementById("image").setAttribute("src", src);

    // update title
    document.getElementById("title").innerHTML = title;
}

// show previous image from images array
function previousImage() {
    // decrement the index
    index--;

    // check if the index is out of bound
    if (index < 0 || index >= images.length) {
        return false;
    }

    // get source of image
    var src = images[index].src;

    // get text of image
    var title = images[index].title;

    // update image
    document.getElementById("image").setAttribute("src", src);

    // update title
    document.getElementById("title").innerHTML = title;
}

Finally, 2 event listeners are added to check when the time spell is rotating and when is paused. Based on that, we will set our variables to change images.

// called when the time spell starts rotating
media.onplaying = function () {
    // set the variable to true
    setTimeout(function () {
        isMediaPlaying = true;
    }, 100);
};

// called when the time spell is paused
media.onpause = function () {
    // get the previous state
    previousState = state;

    // set the current state as empty
    state = "";
    
    // set the variable to false
    isMediaPlaying = false;
};

Following this tutorial, you will be able to track hand rotations using Leap motion controller and Javascript.

Download source code:

[wpdm_package id=’1271′]

Track hand rotational angle – Leap + Javascript

Setup the Project

Leap motion controller connection

Download source code:

Published by adnanafzal565