YoloMobileServo - Game Object Detection & Automation

A comprehensive YOLOv8-based object detection pipeline for training custom models and automating game interaction via servo control. Built for efficient real-time inference on game footage with automated labeling workflows.

Features

✨ Complete ML Workflow

• 📹 Video-to-frames extraction and preprocessing
• 🏷️ Interactive web-based manual labeling (Streamlit)
• 🤖 Automatic object classification using template matching (SSIM)
• 🎓 Incremental YOLOv8 model training with pre-trained weights
• 📊 Dataset quality review and cleaning tools

✨ Inference & Automation

• 🎥 Real-time camera-based object detection
• 🎮 Automated game control via servo/serial interface
• 🎛️ Manual servo control UI (Streamlit)
• 📍 Automatic Region of Interest (ROI) detection via pattern matching

✨ Key Technical Highlights

• Robust background subtraction for contour detection
• SSIM-based template matching for class identification (handles lighting variations)
• Multithreaded batch processing for fast labeling
• Hardware-aware coordinate transformations
• GPU-optimized inference pipeline

Project Structure

Codes/
├── README.md                          # This file
├── AGENTS.md                          # AI agent guide for codebase
├── requirements.txt                   # Python dependencies
│
├── Training Pipeline
│   ├── yolov8-train.py               # YOLOv8 model training script
│   └── yolo_datasets/
│       └── yolomobileservo2.yaml     # Dataset configuration
│
├── Labeling Pipeline
│   ├── video-to-frames.py            # Extract MP4 → PNG frames
│   ├── video-labeler.py              # Interactive labeling UI (Streamlit)
│   ├── auto-labeler.py               # Batch auto-labeling reference
│   ├── dataset-cleaner.py            # QA tool for labeled data
│   └── gray_roi.png                  # ROI template for pattern matching
│
├── Inference & Control
│   ├── yolov8-camera-detect.py       # Real-time camera inference demo
│   ├── auto-gamer.py                 # Full automation pipeline
│   └── serial-servo.py               # Manual servo control UI
│
└── Utilities
    ├── dataset-cleanup.py            # Dataset maintenance
    └── autodelete-unlabeled.py       # Batch cleanup utility

Quick Start

Prerequisites

• Python 3.8+
• GPU with CUDA support (recommended for training; CPU inference supported)
• OpenCV 4.6.0+
• PyTorch with torchvision

Installation

1. Clone the repository

   git clone https://github.com/yourusername/YoloMobileServo.git
   cd YoloMobileServo/Codes

2. Create virtual environment

   python -m venv venv
   # On Windows:
   venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install dependencies

   pip install -r requirements.txt
   pip install ultralytics  # YOLOv8 (not in requirements.txt)

4. Configure paths (Windows-specific in this version)

   • Paths are currently hardcoded in the scripts.
   • If you want a configurable setup, update the path constants directly in the relevant files.

Typical Workflow

1. Label Training Data

# Extract video frames
python video-to-frames.py

# Launch interactive labeler
streamlit run video-labeler.py

# Review and clean labels
streamlit run dataset-cleaner.py

2. Train Model

python yolov8-train.py
# Check results in runs/detect/train_<timestamp>/

3. Run Inference

# Demo: Live camera detection
python yolov8-camera-detect.py

# Automation: Detection + servo control
python auto-gamer.py

# Manual servo control
streamlit run serial-servo.py

Key Concepts

Object Classes

["Player", "Heart", "SmallTree", "BigTree", "Rock", "SnowPile", "EndScreen"]

The "None" class (index 7) is assigned when detected objects don't exceed 85% confidence threshold.

Data Format

Uses standard YOLO format with normalized center coordinates:

<class_idx> <center_x> <center_y> <width> <height>
# Example: 0 0.5 0.5 0.2 0.3 (class 0 at center, 20% width, 30% height)

Configuration Tuning

Key parameters in processing files (adjust for your setup):

• frame_delta_threshold - Foreground detection sensitivity
• contour_min_area / contour_max_area - Object size filtering
• object_class_match_min_accuracy - Classification confidence (default 0.85)
• thread_num - Batch processing parallelization

Note: These are tuned for 1920×1080 resolution. Scale proportionally for different resolutions.

ROI (Region of Interest) Detection

The system uses pattern matching against gray_roi.png to detect the playable game area, making it robust to screen position changes. If game graphics change significantly, regenerate this template.

Deployment Notes

GPU/CPU Inference

• Training requires GPU (device=0 in yolov8-train.py)
• Inference works on GPU/CPU (specify in model.predict)
• Streamlit apps run on CPU (suitable for labeling, not real-time inference)

Serial/Servo Control

• Use auto-gamer.py for automated gameplay
• Uses baudrate 115200 for serial communication
• Servo state encoded as: left*2 + right*1 → {0,1,2,3}
• Verify COM port before running: python -c "from serial.tools.list_ports import comports; print([cp.device for cp in comports()])"

Arduino / Firmware (ESP8266)

This repository includes firmware for an ESP8266-based controller located at arduino/MobileServo.ino.

Key points about MobileServo.ino:

• The sketch sets up a small WiFi web UI for initial servo calibration (servos attached to GPIO pins used by the board).
• After calibration you press "Start" in the web UI and the board switches to serial control mode.
• Serial parameters: 115200 baud, ASCII integer values representing state (0..3). The mapping is left*2 + right*1 (same as above).
• On serial input the board sets two servo positions according to the received integer:
  • 0 → left down, right down
  • 1 → left down, right up
  • 2 → left up, right down
  • 3 → left up, right up

Uploading the firmware

1. Open arduino/MobileServo.ino in the Arduino IDE.
2. Select the correct board (e.g. "NodeMCU 1.0 (ESP-12E Module)" or similar) and the correct COM port.
3. Click Upload.

Replace COM3 with your device port and nodemcuv2 with the appropriate fqbn for your board.

Notes & Troubleshooting

• Ensure the correct USB-serial drivers are installed (e.g. CP210x, CH340, FTDI depending on your board).
• If using the WiFi calibration UI, open the serial monitor at 115200 baud to see the board IP address; then open the printed URL in your browser to access the UI.
• The firmware expects a short ASCII integer (0..3) followed by optional whitespace on serial — the included serial-servo.py Streamlit app sends the same format.

Model Updates

After training, update model paths in inference scripts:

model_path = "runs/detect/train_XXXX_epoch_finished/weights/best.pt"

Troubleshooting

ROI Detection Fails

• Check gray_roi.png exists in working directory
• If game graphics changed, capture and regenerate the ROI template

Contours Too Noisy

• Increase frame_delta_threshold (current: 8)
• Increase contour_min_area (current: 1000 px²)

Serial Connection Not Found

• Verify device COM port with device manager
• Ensure USB drivers installed (CH340, FTDI, etc. depending on hardware)

Out of Memory During Training

• Reduce batch size or training dataset size
• Use smaller model variant

Performance Benchmarks

(Expected performance on typical hardware)

• Model Training: ~90 epochs on GPU ≈ 4-6 hours
• Inference: 30-60 FPS at 1920×1080 on NVIDIA GPU
• Labeling: 100 frames → ~5-10 minutes with 4 worker threads

License

This project is licensed under the MIT License - see LICENSE file for details.

Additional Resources

• AGENTS.md - Comprehensive guide for AI agents and developers
• Ultralytics YOLOv8 Docs: https://docs.ultralytics.com/
• OpenCV Documentation: https://docs.opencv.org/
• Streamlit Docs: https://docs.streamlit.io/