Multimodal AI framework for Indonesian butterfly classification using vision-language models and RAG-based reasoning in green engineering applications

Open

Nisa Dwi Septiyanti, Muhammad Irfan Luthfi

2025 E3S Web of Conferences Vol. 645 Conference paper Cited by 0 SDG 15SDG 17 Quartile

Abstract

Biodiversity loss in ecologically rich regions such as Indonesia underscores the need for sustainable, scalable species monitoring systems. While prior studies have explored deep learning and vision-language models for biological classification, most focus on generic benchmarks or high-resource environments, often lacking structured, domain-specific output. To address this gap, this study proposes a lightweight multimodal AI framework that classifies Indonesian butterfly species using vision-language reasoning and retrieval-augmented generation. The motivation lies in enabling accurate and interpretable ecological monitoring in resource-constrained settings. The system accepts image input via a mobile-responsive interface, processes it through GPT-4 Vision, and outputs six structured attributes: English name, Indonesian name, scientific name, butterfly family, population location, and endangered level. A total of 120 classification sessions were conducted using curated images of both Indonesian and non- Indonesian butterflies. Results show an overall accuracy of 85%, with high field completeness (mean: 4.58 out of 6), consistent reasoning across image quality levels, and low hallucination and latency rates. These findings confirm the system's viability for near-real-time classification and ecological reporting. The framework supports sustainable AI deployment for biodiversity conservation and offers a replicable model for domain-specific species monitoring in developing regions. © The Authors, published by EDP Sciences, 2025.

Affiliations

Department of Information Technology Education, Faculty of Engineering, Universitas Negeri Surabaya, Surabaya, 60231, Indonesia; Graduate Institute of Network Learning Technology, National Central University, Taoyuan, 320314, Taiwan; Department of Electronics and Informatics Engineering Education, Faculty of Engineering, Universitas Negeri Yogyakarta, Yogyakarta, 55281, Indonesia

Research at a Glance

Premium content — register to unlock

Research at a Glance

Register to unlock

Topics & SDG Alignment

Premium content — register to unlock

Topics & SDG Alignment

Register to unlock

Collaboration

Premium content — register to unlock

Collaboration

Register to unlock

Author Profile (Selected)

Premium content — register to unlock

Author Profile (Selected)

Register to unlock

References Overview

Premium content — register to unlock

References Overview

Register to unlock

Journal & Source

Premium content — register to unlock

Journal & Source

Register to unlock

Metadata & Integrity

Premium content — register to unlock

Metadata & Integrity

Register to unlock