Read this in other languages: English, 中文.

OpenVLA Lightweight Version

Lightweight version of OpenVLA, based on the llava-next framework

Features:

1.The main model is only 0.5B, using qwen2-0.5B.

2.Does not use RLDS format datasets, instead fine-tuned using mllm format.

3.Does not occupy inherent tokens of the LLM model, instead using an additional 271 tokens to represent actions.

How to use

Installation

Follow the steps below to install the OpenVLA-Qwen lightweight version.

1. Clone the repository

Clone the repository to your local machine:

git clone https://github.com/IP127000/openVLA-Qwen2-0.5B.git
cd openVLA-Qwen

2. Create a new conda environment and activate it:

conda create -n vla-qwen python=3.10 -y
conda activate vla-qwen

3. Install the required dependencies

pip install --upgrade pip
pip install -e ".[train]"

4. Download the weights from huggingface

cd openVLA-Qwen2-0.5B
mkdir models
cd models
git clone https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-ov

5. Change the file path in scripts/train/finetune_ov_vla.sh

PREV_STAGE_CHECKPOINT="models/llava-onevision-qwen2-0.5b-ov" 
--data_path scripts/train/vla.yaml
--image_folder /images \

6. Change the data path in scripts/train/vla.yaml

json_path: /root/data/vla_llava.json

7. start to train

cd /openVLA-Qwen2-0.5B
./scripts/train/finetune_ov_vla.sh

Acknowledgements

This project makes use of the following open-source projects, and I would like to express my gratitude to their creators:

• OpenVLA - For providing the core structure and functionality that inspired OpenVLA-Qwen.
• LLaVA_OpenVLA - For contribution to data preprocessing techniques.