InstaDA: Augmenting Instance Segmentation Data with Dual-Agent System

Xianbao Hou^{1, 2, ∗}, Yonghao He^{2, ∗, ‡}, Zeyd Boukhers³, John See⁴, Hu Su⁵, Wei Sui^{2, †}, Cong Yang^{1, †}

¹ Soochow University, ² D-Robotics, ³ Fraunhofer Institute for Applied Information Technology, ⁴ Heriot-Watt University Malaysia, ⁵ Institute of Automation, Chinese Academy of Sciences

^∗ Equal Contribution, ^† Corresponding Author, ^‡ Project Lead

Abstract

Acquiring high-quality instance segmentation data is challenging due to the labor-intensive nature of the annotation process and significant class imbalances within datasets. Recent studies have utilized the integration of Copy-Paste and diffusion models to create more diverse datasets. However, these studies often lack deep collaboration between large language models (LLMs) and diffusion models, and underutilize the rich information within the existing training data. To address these limitations, we propose InstaDA, a novel, training-free Dual-Agent system designed to augment instance segmentation datasets. First, we introduce a Text-Agent (T-Agent) that enhances data diversity through collaboration between LLMs and diffusion models. This agent features a novel Prompt Rethink mechanism, which iteratively refines prompts based on the generated images. This process not only fosters collaboration but also increases image utilization and optimizes the prompts themselves. Additionally, we present an Image-Agent (I-Agent) aimed at enriching the overall data distribution. This agent augments the training set by generating new instances conditioned on the training images. To ensure practicality and efficiency, both agents operate as independent and automated workflows, enhancing usability. Experiments conducted on the LVIS 1.0 validation set indicate that InstaDA achieves significant improvements, with an increase of +4.0 in box average precision (AP) and +3.3 in mask AP compared to the baseline. Furthermore, it outperforms the leading model, DiverGen, by +0.3 in box AP and +0.1 in mask AP, with a notable +0.7 gain in box AP on common categories and mask AP gains of +0.2 on common categories and +0.5 on frequent categories.

Overview

• An Overview of InstaDA Pipeline.

Features

• Automated Data Generation Pipeline: Synthesizes high-fidelity data to resolve training data scarcity.

TODOs

• Release the paper on arXiv.
• Release model weights.
• Release the complete code.

Contact

If you have any questions about this paper or code, feel free to email me at xbhou2024@stu.suda.edu.cn.

Acknowledgements

Our work is based on ComfyUI, ControlNet, FLUX, SAM, we appreciate their outstanding contributions. In addition, we are extremely grateful to DiverGen and XPaste for their outstanding contributions in the field of generative data augmentation.

Citation

@article{hou2025instada,
  title={InstaDA: Augmenting Instance Segmentation Data with Dual-Agent System},
  author={Hou, Xianbao and He, Yonghao and Boukhers, Zeyd and See, John and Su, Hu and Sui, Wei and Yang, Cong},
  journal={arXiv preprint arXiv:2509.02973},
  year={2025}
}