L. Vigneswar Reddy, M. K. Dinesh Sekar, M. Pavan Kumar Reddy, Mrs. S. Lavanya
Abstract
This paper presents an innovative Gesture Art Board system that enables intuitive human-computer interaction through real- time hand gesture recognition. Traditional digital whiteboards require physical hardware such as styluses, touchscreens, or mice, imposing significant cost and accessibility barriers in educational and professional settings. The proposed system eliminates these dependencies by leveraging computer vision and machine learning techniques to interpret natural hand movements captured through a standard webcam. The system supports four core interaction modes: freehand drawing, color palette selection, content navigation, and canvas erasing, each triggered by distinct hand gestures. The MediaPipe Hands framework is used for real-time 21-point hand landmark detection, while OpenCV handles video frame processing and canvas rendering. A Random Forest classifier trained on gesture landmark features achieves 91% recognition accuracy at 24 FPS on commodity hardware. A React.js frontend provides a responsive single-page interface connected to a Node.js/Express REST API backend and a Python/Flask machine learning inference microservice. Experimental evaluation demonstrates superior classification performance across all four gesture classes, outperforming baseline models including Logistic Regression, SVM, Decision Tree, XGBoost, and MLP. The system improves accessibility, user engagement, and interaction efficiency across educational, professional, and creative application domains.
Keywords
Gesture Recognition, Gesture Art Board, Hand Tracking, MediaPipe, OpenCV, Computer Vision, Machine Learning, Digital Whiteboard, Human-Computer Interaction, Real-Time Processing, Random Forest, React.js, Canvas Rendering, Landmark Detection
Conclusion
This paper presented the Gesture Art Board, a complete gesture-driven digital canvas system that enables touchless freehand drawing, colour selection, erasure, and navigation using only natural hand gestures captured through a standard webcam. The system integrates the MediaPipe Hands framework for real-time 21-point hand landmark detection, a trained Random Forest classifier for four-class gesture recognition, and a modern three-tier web architecture comprising a React.js frontend, Node.js/Express middleware, and Python/Flask ML inference microservice. Experimental evaluation on a 63,000-instance gesture dataset demonstrated a gesture recognition accuracy of 91.00% and an F1- score of 90.77%, outperforming five baseline classifiers across all evaluation metrics. The system sustains an end-to-end interaction latency of 38.2 ms at 24 FPS on CPU-only hardware, meeting real-time interaction requirements without specialised computational resources. The gesture validity screening pipeline reduced spurious canvas events by 87%, significantly improving the quality of the drawing experience. Informal usability evaluation confirmed high user satisfaction across all primary interaction modes. The following future enhancements are planned for the system:
1)Cloud Deployment: Migration to AWS or Azure withauto-scaling ML inference endpoints, HIPAA/GDPR-compliant data handling, and multi-region CDN delivery for globally distributed users.
2)Mobile Integration: Native mobile SDK wrapping the gesturerecognition pipeline using TensorFlow Lite for on-device inference on Android and iOS devices without server round-trips.
3)Expanded Gesture Vocabulary: Adding support for pinch-to- zoom, two-handed rotation, three-finger undo/redo, and wrist-roll brush size adjustment, expanding the interaction vocabulary to ten or more gestures.
4)Deep Learning Models: Investigation of lightweight CNNarchitectures (MobileNetV3, EfficientNet-Lite) and temporal gesture models (LSTM, Transformer) for improved accuracy on ambiguous gesture transitions without sacrificing real-time performance.
5)Collaborative Multi-User Canvas: Real-time shared canvasfunctionality using WebRTC peer-to-peer data channels, enabling geographically distributed users to draw simultaneously on a shared virtual whiteboard with per-user colour coding.
6)Accessibility Features: Adaptations for users with limited handmobility, including gesture simplification modes, single- hand operation, and eye-gaze integration via webcam-based gaze tracking as a complementary input modality.
The Gesture Art Board system represents a meaningful contribution toward democratising digital art creation and interactive whiteboard technology through accessible, hardware-free, intelligent gesture- based interaction. By combining state-of-the-art computer vision with a modern web stack, the system demonstrates that high-quality touchless interaction is achievable on commodity hardware, opening new possibilities for inclusive digital creativity in educational, professional, and creative domains.
References
[1]F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang,and
M. Grundmann, “MediaPipe Hands: On-device real-time hand tracking,” in Proc. ECCV Workshop on Computer Vision for Augmented and Virtual Reality, Glasgow, UK, Aug. 2020.
[2]X. Liu, Y. Chen, and R. Patel, “Touchscreen-based hand tracking for remotewhiteboard interaction in distance education,” IEEE Trans. Multimedia, vol. 26, no. 3, pp. 112–124, Mar. 2024.
[3]M. Oudah, A. Al-Naji, and J. Chahl, “Hand gesture recognition based oncomputer vision: A review of techniques,” Journal of Imaging, vol. 6, no. 8,
p. 73, Aug. 2020.
[4]A. Singh, P. Sharma, and R. Gupta, “Air Canvas: A real-time virtual drawingsystem using hand gestures and OpenCV,” in Proc. Int. Conf. Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, Mar. 2021, pp. 1–6.
[5]L. Chen, W. Wang, and Z. Li, “Real-time hand tracking whiteboard systemusing MediaPipe and OpenCV,” in Proc. ICCV Workshop on Interactive AI Systems, Paris, France, Oct. 2023, pp. 45–52.
[6]T. Zhang, S. Kumar, and H. Liu, “Gesture-based virtual whiteboard using deepconvolutional neural networks and temporal attention,” IEEE Access, vol. 12, pp. 6543–6558, Jan. 2024. [7]G. Jocher, A. Chaurasia, and J. Qiu, “YOLOv8 for real-time object detectionand gesture recognition,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, Jun. 2023.
[8]F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
[9]N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002.
[10]L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–
32, Oct. 2001.
How to Cite This Paper
L. Vigneswar Reddy, M. K. Dinesh Sekar, M. Pavan Kumar Reddy, Mrs. S. Lavanya (2026). Gesture Art Board: Advancing Digital Expression Through Hand Gesture Recognition Technology. International Journal of Computer Techniques, 13(3). ISSN: 2394-2231.
