Track hand rotational angle – Leap + Javascript
In this tutorial, we are going to teach you how you can track your hand rotation using Leap Motion Controller. Leap Motion Controller is a device that allows you to detect your hand motions and gestures, and perform any desired action on them. It provides SDK in Java, Javascript, Python, C++, and Unreal Engine. We will be using Javascript in this tutorial.
You can buy the Leap Motion Controller from their official website. After buying, connect the device with your computer using a USB or USB-C port. Then download the SDK from here and do the installation.
If you prefer the video tutorial:
From the above video, you can see what we are going to create. So you will need a Doctor Strange time-spell video and some images. You will also need the Leap JS library to connect with Leap Motion Controller. You can find all 3 in the source files below.
Setup the Project
The following code helps you to reverse a video and concatenate 2 videos using FFmpeg:
// reverse a video
// ffmpeg -i Pentagramm2_green.mp4 -vf reverse reversed.mp4
// concatenate 2 videos
// ffmpeg -i Pentagramm2_green.mp4 -i reversed.mp4 -filter_complex "[0:v] [0:a] [1:v] concat=n=2:v=1:a=1 [v] [a]" -map "[v]" -map "[a]" output.mp4
Then create a video tag for the time-spell video.
<video src="output.mp4" muted id="time-spell"></video>
After that, create a div to show the image and text below it.
<div style="position: absolute; right: 50px; top: 50px;">
<img src="images/2021.JPG" id="image" />
<h1 id="title">2021</h1>
</div>
Now include the Leap JS library in your project.
<script src="leap-0.6.4.min.js"></script>
Then apply some CSS styles to display the video, image, and text properly.
<style>
/* remove margin and set background color as black */
body {
margin: 0px;
background-color: black;
}
/* set width of time spell and its position */
#time-spell {
width: 1200px;
position: absolute;
left: -275px;
top: 30px;
}
/* set width and height of image and fit inside container */
#image {
width: 800px;
height: 600px;
object-fit: contain;
}
/* set color, position, alignment, font size and background color of image text */
#title {
color: white;
position: relative;
bottom: 129px;
text-align: center;
font-size: 70px;
background-color: rgba(0, 0, 0, 0.7);
}
</style>
Then create an array of images and their text in Javascript.
<script>
// list of images and their text
var images = [{
"src": "images/2021.JPG",
"title": "2021"
}];
// index is out of bound right now
var index = images.length;
</script>
After that, initialize some variables that we will be using throughout the tutorial.
// total duration of time spell video
var totalDuration = 40;
// state of time spell (back or forward)
var state = "";
// previous state of time spell
var previousState = "";
// hand rotation from Leap Motion Controller
var handRotation = "";
// is time spell start rotating
var isMediaPlaying = false;
// video tag
var media = document.getElementById("time-spell");
Leap motion controller connection
Now is the time to connect our Javascript app with Leap Motion Controller.
// setup Leap loop with frame callback function
var controllerOptions = {};
Leap.loop(controllerOptions, function(frame) {
// get all hands in frame
var hands = frame.hands;
// if there is any hand
if (hands.length > 0) {
// get the first hand
var hand = hands[0];
// ignore if the hand is left
if (hand.isLeft) {
return false;
}
// get direction of hand (x,y,z co-ordinates)
var axis = hand.direction;
// get x-axis
axis = axis[0];
if (axis > 0 && axis < 0.9) {
// if hand is rotated to right
handRotation = "right";
} else if (axis > -0.09 && axis < -0.01) {
// if hand is in center (no rotation)
handRotation = "center";
media.pause();
} else if (axis < 0 && axis < -0.5) {
// if hand is rotated to left
handRotation = "left";
}
} else {
// if there is no hand in the frame, simply pause the time spell
if (media != null) {
media.pause();
}
}
});
For explanation, comments have been added with each line. Now we need to call a function every 0.5 seconds and rotate the time spell to the left if the hand is rotated to left, and right if the hand is rotated to right. Also, it will call a function to change the image as per hand rotation.
// called every 0.5 seconds
setInterval(function () {
// if hand is rotating to right
if (handRotation == "right") {
// only called once to rotate time spell
if (state == "") {
// rotate to right
mediaBackward();
}
// if time spell starts rotating
if (isMediaPlaying) {
// show next image
nextImage();
}
// make it empty to give a delay to show next image
handRotation = "";
} else if (handRotation == "left") {
// only called once to rotate time spell
if (state == "") {
// rotate to left
mediaForward();
}
// if time spell starts rotating
if (isMediaPlaying) {
// show previous image
previousImage();
}
// make it empty to give a delay to show previous image
handRotation = "";
} else if (handRotation == "center") {
// pause the time spell, when hand is in center of Leap Motion Controller
media.pause();
}
}, 500);
Now we just need to create these 4 functions called from the above code.
// when hand is rotated right
function mediaBackward() {
// "back" means reverse
state = "back";
// starts rotating time spell
media.play();
// current time of video
var currentTime = media.currentTime;
// if video stops playing
if (currentTime <= 0) {
// starts from middle to move right
currentTime = totalDuration / 2;
}
// this condition becomes true only when the
// time spell was rotating forward (left side)
if (previousState != state) {
// change direction of time spell
var newTime = totalDuration - currentTime;
media.currentTime = newTime;
}
}
// when hand is rotated left
function mediaForward() {
// "forward" means original video
state = "forward";
// starts rotating time spell
media.play();
// current time of video
var currentTime = media.currentTime;
// if video stops playing
if (currentTime <= 0) {
// starts from zero to move left
currentTime = totalDuration;
}
// this condition becomes true only when the
// time spell was rotating backward (right side)
if (previousState != state) {
// change direction of time spell
var newTime = totalDuration - currentTime;
media.currentTime = newTime;
}
}
// show next image from images array
function nextImage() {
// increment the index
index++;
// check if the index is out of bound
if (index >= images.length || index < 0) {
return false;
}
// get source of image
var src = images[index].src;
// get text of image
var title = images[index].title;
// update image
document.getElementById("image").setAttribute("src", src);
// update title
document.getElementById("title").innerHTML = title;
}
// show previous image from images array
function previousImage() {
// decrement the index
index--;
// check if the index is out of bound
if (index < 0 || index >= images.length) {
return false;
}
// get source of image
var src = images[index].src;
// get text of image
var title = images[index].title;
// update image
document.getElementById("image").setAttribute("src", src);
// update title
document.getElementById("title").innerHTML = title;
}
Finally, 2 event listeners are added to check when the time spell is rotating and when is paused. Based on that, we will set our variables to change images.
// called when the time spell starts rotating
media.onplaying = function () {
// set the variable to true
setTimeout(function () {
isMediaPlaying = true;
}, 100);
};
// called when the time spell is paused
media.onpause = function () {
// get the previous state
previousState = state;
// set the current state as empty
state = "";
// set the variable to false
isMediaPlaying = false;
};
Following this tutorial, you will be able to track hand rotations using Leap motion controller and Javascript.
Download source code:
[wpdm_package id=’1271′]
