Implementasi Local-First RAG dengan Hybrid Retrieval IndoBERT dan BM25 untuk Pendukung Keputusan Akademik

Abstract

The need for rapid and accurate access to academic information for educational institution stakeholders, such as campus management and academic advisors, still relies on manual administrative processes, thereby leading to operational inefficiencies. This study develops a locally hosted, open-source Retrieval-Augmented Generation (RAG) system as an automated solution for academic information services while alleviating administrative burdens. The system integrates the Mistral-7B-Instruct LLM with a hybrid search approach within the Elasticsearch ecosystem, combining IndoBERT-based dense retrieval for narrative academic guideline documents and BM25-based sparse retrieval for structured student data. Evaluation was conducted using ROUGE-1, ROUGE-2, and ROUGE-L metrics against 60 test data points generated by Claude Sonnet 4.6. The system successfully answered 58 out of 60 queries, achieving a ROUGE-L f1-score of 0.29. An asymmetrical pattern was observed, where recall values were consistently higher than precision across all metrics, which indicates the impact of the language generation capacity gap between Mistral 7B and the reference model. An average input prompt length ranging from 1,270 to 1,380 tokens contributed to an average latency of 30 seconds per query, representing the primary contemporary challenge of the system. This research is expected to serve as a baseline for developing open-source RAG systems within Indonesian language domains, specifically in the context of higher education academic administration.