flugmaschine/hand-tracking-drawing repository overview

⁠Hand Tracking Drawing

Draw in the air with your hand, then view the drawing as a rotatable 3D object. Hand tracking uses the MediaPipe Hand Landmarker. Rendering and the UI use OpenCV.

⁠Image

• Tag: flugmaschine/hand-tracking-drawing:latest
• Platform: linux/amd64
• Base: python:3.12-slim

⁠Run

This is a camera and GUI application. It needs a Linux host with a webcam and an X11 display. It cannot reach the camera or screen on Docker Desktop for macOS or Windows.

xhost +local:docker

docker run --rm \
  --device /dev/video0:/dev/video0 \
  -e DISPLAY=$DISPLAY \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  --network host \
  flugmaschine/hand-tracking-drawing:latest

Saved drawings go to /app/output inside the container. Mount a folder to keep them:

  -v $(pwd)/output:/app/output

⁠Logic

• One index finger - draw
• Index + middle - move cursor without drawing
• Fist (hold) - grab the drawing; move the fist to rotate it in 3D
• Two fists - scale (canvas in 2D, object in 3D)
• Open palm - release, back to drawing
• OK (hold) - clear the canvas
• Pinky (hold) - undo the last stroke
• Middle finger - shows :(( for a second
• Both palms waving - say goodbye, closes the app

Keys: S save, C clear, U undo, 1..6 color, [ / ] thickness, SPACE 2D/3D, R reset view, K skeleton, D debug, H help, Q quit.

⁠3D Drawing

The 3D Drawing mode uses the YOLO depth model (models/yolo26s-depth.pt), which returns a per pixel depth map in meters. Every drawn point takes its Z from the metric depth under the fingertip, so moving the hand closer or further really draws into the scene: the stroke is a 3D curve in space, not a flat picture placed on a board. The first drawn point sets the zero plane, everything after that is relative to it. Depth runs in a background thread, so tracking and drawing stay at camera speed.

Raw monocular depth flickers, so the pen does not follow it directly: the global scale drift of the map is cancelled against the static background, the fingertip depth goes through a one euro filter with a noise deadband and a speed limit, and the newest points of a stroke are low passed along the curve. A circle drawn without moving the hand comes out flat, while a deliberate push of 25 cm is kept in full.

By default strokes are colored by a depth gradient - warm near the camera, cool further away. 1..6 switch to a solid color, G goes back to the gradient.

Gestures and keys are the same as in 2D, with two additions:

• Two ring fingers (hold) - toggle depth colors for the whole frame, drawing keeps working
• Open palm (hold) - reset the view

In 3D the fist rotates the drawing itself and two fists scale it, SPACE toggles depth colors, R resets the view.

by Salimli Ayzek (Салимли Айзек): https://mathematiclove.github.io⁠/my-cv⁠