mem0/docs/components/evaluation.mdx

---
title: 🔬 Evaluation
---

## Overview

We provide out-of-the-box evaluation methods for your datasets. You can use them to evaluate your models and compare them with other models.

Currently, we provide the following evaluation methods:

<CardGroup cols={3}>
    <Card title="Context Relevancy" href="#context_relevancy"></Card>
    <Card title="Answer Relevancy" href="#answer_relevancy"></Card>
    <Card title="Groundedness" href="#groundedness"></Card>
    <Card title="Custom" href="#custom"></Card>
</CardGroup>

More evaluation metrics are coming soon! 🏗️

## Usage

We have found that the best way to evaluate datasets is with the help of OpenAI's `gpt-4` model. Hence, we require you to set `OPENAI_API_KEY` as an environment variable. If you don't want to set it, you can pass it in the config argument of the respective evaluation class, as shown in the examples later below.

<Accordion title="We will assume the following dataset for the examples below">
<CodeGroup>
```python main.py
from embedchain.utils.eval import EvalData

data = [
    {
        "question": "What is the net worth of Elon Musk?",
        "contexts": [
            """Elon Musk PROFILEElon MuskCEO, ...""",
            """a Twitter poll on whether the journalists' ...""",
            """2016 and run by Jared Birchall.[335]...""",
        ],
        "answer": "As of the information provided, Elon Musk's net worth is $241.6 billion.",
    },
    {
        "question": "which companies does Elon Musk own?",
        "contexts": [
            """of December 2023[update], ...""",
            """ThielCofounderView ProfileTeslaHolds ...""",
            """Elon Musk PROFILEElon MuskCEO, ...""",
        ],
        "answer": "Elon Musk owns several companies, including Tesla, SpaceX, Neuralink, and The Boring Company.",
    },
]

dataset = []

for d in data:
    dataset.append(EvalData(question=d["question"], contexts=d["contexts"], answer=d["answer"]))
```
</CodeGroup>
</Accordion>

## Context Relevancy <a id="context_relevancy"></a>

Context relevancy is a metric to determine how relevant the context is to the question. We use OpenAI's `gpt-4` model to determine the relevancy of the context.
We achieve this by prompting the model with the question and the context and asking it to return relevant sentences from the context. We then use the following formula to determine the score:

context_relevance_score = (# of relevant sentences in context) $$\div$$ (total # of sentences in context)

You can run the context relevancy evaluation with the following simple code:

```python
from embedchain.eval.metrics import ContextRelevance
metric = ContextRelevance()
score = metric.evaluate(dataset)    # dataset from above
print(score)
# 0.27975528364849833
```

In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `ContextRelevanceConfig` class. 

### ContextRelevanceConfig

<ParamField path="model" type="str" optional>
    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
</ParamField>
<ParamField path="api_key" type="str" optional>
    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
</ParamField>
<ParamField path="language" type="str" optional>
    The language of the dataset being evaluated. We need this to determine the understand the context provided in the dataset. Defaults to `en`.
</ParamField>
<ParamField path="prompt" type="str" optional>
    The prompt to extract the relevant sentences from the context. Defaults to `CONTEXT_RELEVANCY_PROMPT`, which can be found at `embedchain.config.eval.base` path.
</ParamField>

```python
openai_api_key = "sk-xxx"
metric = ContextRelevance(config=ContextRelevanceConfig(model='gpt-4', api_key=openai_api_key, language="en"))
print(metric.evaluate(dataset))
```


## Answer Relevancy <a id="answer_relevancy"></a>

Answer relevancy is a metric to determine how relevant the answer is to the question. We use OpenAI's `gpt-4` model to determine the relevancy of the answer.
We achieve this by prompting the model with the answer and asking it to generate questions from the answer. We then use the cosine similarity between the generated questions and the original question to determine the score.

answer_relevancy_score = mean(cosine_similarity(generated_questions, original_question))

You can run the answer relevancy evaluation with the following simple code:

```python
from embedchain.eval.metrics import AnswerRelevance
metric = AnswerRelevance()
score = metric.evaluate(dataset)    # dataset from above
print(score)
# 0.9505334177461916
```

In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `AnswerRelevanceConfig` class.

### AnswerRelevanceConfig

<ParamField path="model" type="str" optional>
    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
</ParamField>
<ParamField path="embedder" type="str" optional>
    The embedder to use for embedding the text. Defaults to `text-embedding-ada-002`. We only support openai's embedders for now.
</ParamField>
<ParamField path="api_key" type="str" optional>
    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
</ParamField>
<ParamField path="num_gen_questions" type="int" optional>
    The number of questions to generate for each answer. We use the generated questions to compare the similarity with the original question to determine the score. Defaults to `1`.
</ParamField>
<ParamField path="prompt" type="str" optional>
    The prompt to extract the `num_gen_questions` number of questions from the provided answer. Defaults to `ANSWER_RELEVANCY_PROMPT`, which can be found at `embedchain.config.eval.base` path.
</ParamField>

```python
openai_api_key = "sk-xxx"
metric = AnswerRelevance(config=AnswerRelevanceConfig(model='gpt-4',
                                                      embedder="text-embedding-ada-002",
                                                      api_key=openai_api_key,
                                                      num_gen_questions=2))
print(metric.evaluate(dataset))
```

## Groundedness <a id="groundedness"></a>

Groundedness is a metric to determine how grounded the answer is to the context. We use OpenAI's `gpt-4` model to determine the groundedness of the answer.
We achieve this by prompting the model with the answer and asking it to generate claims from the answer. We then again prompt the model with the context and the generated claims to determine the verdict on the claims. We then use the following formula to determine the score:

groundedness_score = (sum of all verdicts) $$\div$$ (total # of claims)

You can run the groundedness evaluation with the following simple code:

```python
from embedchain.eval.metrics import Groundedness
metric = Groundedness()
score = metric.evaluate(dataset)    # dataset from above
print(score)
# 1.0
```

In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `GroundednessConfig` class.

### GroundednessConfig

<ParamField path="model" type="str" optional>
    The model to use for the evaluation. Defaults to `gpt-4`. We only support openai's models for now.
</ParamField>
<ParamField path="api_key" type="str" optional>
    The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.
</ParamField>
<ParamField path="answer_claims_prompt" type="str" optional>
    The prompt to extract the claims from the provided answer. Defaults to `GROUNDEDNESS_ANSWER_CLAIMS_PROMPT`, which can be found at `embedchain.config.eval.base` path.
</ParamField>
<ParamField path="claims_inference_prompt" type="str" optional>
    The prompt to get verdicts on the claims from the answer from the given context. Defaults to `GROUNDEDNESS_CLAIMS_INFERENCE_PROMPT`, which can be found at `embedchain.config.eval.base` path.
</ParamField>

```python
openai_api_key = "sk-xxx"
metric = Groundedness(config=GroundednessConfig(model='gpt-4',
                                                api_key=openai_api_key))
print(metric.evaluate(dataset))
```

## Custom <a id="custom"></a>

You can also create your own evaluation metric by extending the `BaseMetric` class. You can find the source code for the existing metrics at `embedchain.eval.metrics` path.

<Note>
You must provide the `name` of your custom metric in the `__init__` method of your class. This name will be used to identify your metric in the evaluation report.
</Note>

```python
from embedchain.eval.metrics import BaseMetric
from embedchain.utils.eval import EvalData
from embedchain.config.base_config import BaseConfig
from typing import Optional

class CustomMetric(BaseMetric):
    def __init__(self, config: Optional[BaseConfig] = None):
        super().__init__(name="custom_metric")

    def evaluate(self, dataset: list[EvalData]):
        score = 0.0
        # write your evaluation logic here
        return score
```