Machine learning trains a model against right and wrong examples so it can recognize patterns and perform tasks such as prediction. A hand-tracking model, for example, learns to recognize images of hands. Training may happen in Python with TensorFlow, while the trained model runs in a browser through TensorFlow.js. section 15.4
A browser-based hand-tracking feature can be assembled by:
- Loading a Handtrack.js model with parameters such as horizontal flipping, detection limits, overlap thresholds, and prediction confidence. section 15.4
- Passing video frames to the model and receiving hand-location predictions. section 15.4
- Hiding the model and tracking logic behind a Web Component that exposes hand locations to the page. section 15.4
Hand tracking for a Web Harp
A hand-tracking component can sit over a Web Harp interface, use the camera and display canvas, and report detected points through an event. The harp can then respond to hand movement without exposing the underlying model or TensorFlow dependencies. section 15.4
Listing 15.20. Hand-tracking Web Component
import Video from './video.js';
export default class HandTracker extends Video { #1
static get HAND_LOCATION() { return 'onHandLocation'; }
constructor() {
super();
const modelParams = {
flipHorizontal: true, // flip e.g for video
maxNumBoxes: 20, // maximum number of boxes to detect
iouThreshold: 0.5, // ioU threshold for non-max suppression
scoreThreshold: 0.6, // confidence threshold for predictions.
};
handTrack.load(modelParams) #2
.then(lmodel => {
this._model = lmodel;
});
}
runDetection() {
if (!this._model) { return; }
this._model.detect(this.dom.video).then(predictions => {
const pts = [];
for (let c = 0; c < predictions.length; c++) {
const centerpoint = {};
centerpoint.x = (predictions[c].bbox[0] +
(predictions[c].bbox[2] / 2));
centerpoint.y = (predictions[c].bbox[1] +
(predictions[c].bbox[3] / 2));
pts.push(centerpoint);
}
this._model.renderPredictions(predictions, this.dom.canvas,
this.canvasContext, this.dom.video);
const ce = new CustomEvent( HandTracker.HAND_LOCATION,
{ detail: { points: pts }, bubbles: true, composed: true });
this.dispatchEvent(ce); #3
});
}
init() { #4
super.init();
handTrack.startVideo(this.dom.video).then((status) => {
this.onResize();
console.log(this.visibleVideoRect)
if (status) { this.runDetection(); }
});
}
getCurrentFrameData(mode, noredraw) { #5
this.runDetection();
}
}
if (!customElements.get('wcia-handtracker')) {
customElements.define('wcia-handtracker', HandTracker);
}
Inspect the component that loads the model, runs detection on video, and turns predictions into hand-location data. section 15.4
