Handaru Jati, Yuniar Indrihapsari, Susilo Romadhon, Pradana Setialana, Danang Wijaya, Satya Adhiyaksa Ardy
Large language models (LLMs) are increasingly used to support educational assessment, yet institutional adoption requires balancing performance with governance constraints related to auditability, data control, and deployability. We compare a closed-source model (GPT-4o) and an open-weights model (LLaMA 3.1) on two expert validated assessment formats across five academic domains (Science, History, Literature, Technology, and Social Studies). Experiment 1 evaluates multiple choice question (MCQ) option selection using 148 items, five prompting strategies (Raw, Brief Instruction, Long Instruction, Chain-of-Thought, and Question-Answer Prompt Generation), and five independent repetitions per condition (7,400 total model calls) to estimate both accuracy and run-to-run stability under a strict A to E output constraint. Experiment 2 evaluates 147 one sentence short answer items under a fixed instruction prompt. In the multiple choice questions (MCQs), the accuracy and stability of the results for GPT-4o were higher compared to those of LLaMA 3.1, and the incorporation of the additional prompt structure was associated with higher stability compared to the average improvement in correctness. In the short answer questions, the performance of GPT-4o was slightly higher compared to that of LLaMA 3.1 for ROUGE-L and METEOR, and these results mostly corresponded to the results obtained for the semantic similarity questions and the evaluation rubric. The results of the study are important for understanding the accuracy-stability trade-off for the two language generators and for supporting the evaluation approach that considers the correctness of the results for MCQs, their stability for multiple runs, and the validation of the results for short answer questions using the evaluation rubric and the results for the semantic similarity questions. © 2026, Penerbit UTHM. All rights reserved.
Department of Electronics and Informatics Engineering Education, Faculty of Engineering, Information Technology Studi Program, Universitas Negeri Yogyakarta, Karangmalang Number 1, Sleman, 55281, Indonesia; Department of Computer Science and Information Engineering, National Central University, Taoyuan, Taiwan; Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
