Sri Yamtinah, Antuni Wiyarsi, Hayuni Retno Widarti, Ari Syahidul Shidiq, Dimas Gilang Ramadhani
Integrating artificial intelligence (AI) into educational assessments represents a paradigm shift, especially in STEM subjects such as chemistry, which require complex problem-solving and written feedback. This study focuses on the effects of fine-tuning on each of the four AI models—Gemini 1.5, Gemini 1.0, BERT, and XLNet—and their performance on chemistry tasks. The models were implemented using three evaluation methods: Seq2Seq for Sodium Reaction Grading, a Regression Task for Overall Grading, and Attention-Based Grading for Key Steps Grading. The fine-tuning process significantly improved the models' accuracy, precision, and stability. Gemini 1.5 outperformed the other models across all three metrics, with accuracy increasing from 80 % to 89.5 % and TPR from 0.73 to 0.93, whereas Gemini 1.0 achieved TPR gains from 0.69 to 0.89. BERT and XLNet, which had substantially lower baselines, also showed significant improvements, particularly in identifying fundamental steps in the evaluation. These advances highlight the critical role of fine-tuning in refining AI model output to align with expert grading standards, ensuring accuracy and reliability in assessment. The results confirm that fine-tuning is essential in preparing AI models for teaching applications, particularly for complex tasks such as chemistry evaluation, thus enabling scalable solutions. The findings of this study provide further justification for the wider adoption of fine-tuned AI models to improve the reliability, scalability, and effectiveness of grading systems in STEM education. © 2025
Chemistry Education Study Program, Faculty of Teacher Training and Education, Universitas Sebelas Maret, Indonesia; Chemistry Education, Faculty of Mathematics and Natural Science, Universitas Negeri Semarang, Indonesia; Chemistry Education, Faculty of Mathematics and Natural Science, Universitas Negeri Yogyakarta, Indonesia; Chemistry Education, Faculty of Mathematics and Natural Science, Universitas Negeri Malang, Indonesia
