[Feature Request] ROUGE Score Drops Due to <think> Tags in LLM Outputs (Internal Reasoning)

[BaconTomatoDeluxe]

Is your feature request related to a problem? Please describe.
Recent LLMs such as Qwen 3 and DeepSeek R1 use tags to include intermediate reasoning steps in their responses. However, this reasoning content is also included in the generated output used for evaluation, which leads to unfairly low ROUGE scores.

Describe the solution you'd like
I’d like the evaluation step to exclude content wrapped in ... tags. This would allow evaluation to focus only on the final answer intended for users, improving the fairness and accuracy of ROUGE-based scoring.

Describe alternatives you've considered
Right now, I'm using a simple workaround by modifying the generation output via regex filtering. Specifically, I patched the following line in:

.venv/lib/python3.10/site-packages/autorag/nodes/generator/llama_index_llm.py, line 84:

generated_texts = list(map(lambda x: re.sub(r'<think>.*?</think>\s*', '', x.text, flags=re.DOTALL).strip(), results))

This strips out the blocks before evaluation. While it works for now, I'm not sure if this is the best long-term solution.

[vkehfdl1]

Hello @BaconTomatoDeluxe
Looks like great feature to add! And thank you for suggesting workaround.
Maybe it can be great option to add in the rouge score metric.

I need token variation via model. Not every model use token isn't it? Some model can use or etc.
If I can get that kind of list, it will be very helpful.

[BaconTomatoDeluxe]

Thanks again for the quick response.
I also really appreciated your point about how the tag names might differ depending on the model.

Since I wanted to avoid including internal reasoning in the output altogether, I dug into the code to understand why the thinking part might still be showing up. So far, I’ve only looked into this in the context of Ollama.
While inspecting the llama_index integration, I noticed that Ollama’s response includes both a content field (representing the actual answer) and a thinking field (which seems to represent intermediate reasoning steps).

Ideally, only the content should be surfaced as the model’s final answer. I’m still not entirely sure why that happens — it could be due to some normalization step I’m missing on my end.

https://ollama.com/blog/thinking

[vkehfdl1]

@BaconTomatoDeluxe I think each model integrations have different strategies to deal with think tokens. (like huggingface, vllm, ollama, etc.)
Let's implement your workaround as a metric option first, and build an better version later if we can find a universal interface for classifying thinkpad and real answer of model.