I am feeling lucky again?

Building an RAG Application which sometimes I rarely testing it other than “It’s good enought, oh it’s relate and the list seems good” became a vague metric and harder to judge how performance is our system.

Evaluation metric became a tremendous help for me to measure and optimize the system. While doing it stumbling a metric MRR (Mean Reciprocal Rank) give me a light bulb “Hmm it’s similar with the google I am feeling lucky ya”

Embedding & Reranking Model

Google releasing a new embedding model called Gemma Embedding 2, the introduction is quite interesting that it says it can run even on the edge device (smartphone), and the performance is quite good even on the smallest model. Now I am reflecting “what is quite good even means?”, so let’s deep dive by testing it first.

Running the model on Mac

The daily driver for me to testing things is on macbook air m4 with a 16GB of RAM. We run the model using MLX mlx-community/embeddinggemma-2-6bit text only. It shows a good enough result in here.

Backend	bs=1	bs=16	bs=64
MLX 6-bit	56	67	68

Not only embedding model, we do run a small reranking model qwen3-reranker-0.6b which will be used later after the vector search.

Choosing the dataset

Not so much for choosing the dataset, am telling ai to select subset of data from huggingface mteb/MIRACLRetrievalHardNegatives for Indonesia corpus which is totaled ~168k and it use ~9,884 passages.

Simple Benchmark RAG

As I mention when looking for RAG benchmark something clicked when seeing MRR and remembering I am feeling lucky on google search.

One of the benchmark is MRR (Mean Reciprocal Rank) where it only count the rank 1 relevance in position. For example if Rank 1 placed on 1st then 1/1=1 as the maximum value for the MRR is 1, so the value is 1/n where n is the position of rank 1.

Other benchmark that we use is Recall & nDCG which can represent do relevant passage showing up in the top k? and how well the relevant passage ranked among k. Below are the illustration how we do calculate each metric.

Running the Eval

Now finding out how well the embedding2 with reranking perform which should be able to run on edge devices. Oh we are using sqlite as vector database as the goal is quickly testing how well the vector search evaluation. We use k=30 and do rerank on them and calculate on the 10 result.

	nDCG@10	MRR@10	Recall@10
Vector search only	0.7288	0.7715	0.8608
+ Qwen3-Reranker-0.6B on top 30	0.7918	0.8188	0.9048

The vector search using a embedding model returning a quite good score (Too lazy to compare to other model haha) but I feel that it’s already quite good. By adding a small amount of component which is reranker it boost the result ~9% which is quite nice :)

Conclusion

The importance of evaluation not only for doing it for the sake of measuring, but each measurement giving us a insight when we choose it? how well it perform? and which measurement are relevance to our use case.

The focus on MRR could became an optimal choice where our system need to make sure the first choice is relevant, and that’s it what only important. While other measurement shine in other use case too.