Évaluer des applications RAG

Créez et évaluez des applications RAG à l’aide de Weave et de juges LLM

La génération augmentée par récupération (RAG) est une méthode courante pour créer des applications d’IA générative ayant accès à des bases de connaissances personnalisées.

​

Ce que vous apprendrez :

Ce guide vous montre comment :

• Créer une base de connaissances
• Créer une application RAG avec une étape de récupération qui identifie les documents pertinents
• Suivre les étapes de récupération avec Weave
• Évaluer des applications RAG à l’aide d’un juge LLM pour mesurer la précision du contexte
• Définir des fonctions de score personnalisées

​

Prérequis

• Un compte W&B
• Python 3.8+ ou Node.js 18+
• Packages requis :
  • Python: pip install weave openai
  • TypeScript: npm install weave openai
• Une clé API OpenAI configurée comme variable d’environnement

​

Créer une base de connaissances

Commencez par calculer les embeddings des articles. En général, vous ne le faites qu’une seule fois pour vos articles, puis vous stockez les embeddings et les métadonnées dans une base de données. Mais ici, par souci de simplicité, cette opération est effectuée à chaque exécution du script.

Python

from openai import OpenAI
import weave
from weave import Model
import numpy as np
import json
import asyncio

articles = [
    "Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial “tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too,” one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.",
    "Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.",
    "Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities",
    "Rivian and Lucid shares plunge after weak EV earnings reports Shares of electric vehicle makers Rivian and Lucid fell Thursday after the companies reported stagnant production in their fourth-quarter earnings after the bell Wednesday. Rivian shares sank about 25 percent, and Lucids stock dropped around 17 percent. Rivian forecast it will make 57,000 vehicles in 2024, slightly less than the 57,232 vehicles it produced in 2023. Lucid said it expects to make 9,000 vehicles in 2024, more than the 8,428 vehicles it made in 2023.",
    "Mauritius blocks Norwegian cruise ship over fears of a potential cholera outbreak Local authorities on Sunday denied permission for the Norwegian Dawn ship, which has 2,184 passengers and 1,026 crew on board, to access the Mauritius capital of Port Louis, citing “potential health risks.” The Mauritius Ports Authority said Sunday that samples were taken from at least 15 passengers on board the cruise ship. A spokesperson for the U.S.-headquartered Norwegian Cruise Line Holdings said Sunday that 'a small number of guests experienced mild symptoms of a stomach-related illness' during Norwegian Dawns South Africa voyage.",
    "Intuitive Machines lands on the moon in historic first for a U.S. company Intuitive Machines Nova-C cargo lander, named Odysseus after the mythological Greek hero, is the first U.S. spacecraft to soft land on the lunar surface since 1972. Intuitive Machines is the first company to pull off a moon landing — government agencies have carried out all previously successful missions. The company's stock surged in extended trading Thursday, after falling 11 percent in regular trading.",
    "Lunar landing photos: Intuitive Machines Odysseus sends back first images from the moon Intuitive Machines cargo moon lander Odysseus returned its first images from the surface. Company executives believe the lander caught its landing gear sideways on the moon's surface while touching down and tipped over. Despite resting on its side, the company's historic IM-1 mission is still operating on the moon.",
]

def docs_to_embeddings(docs: list) -> list:
    openai = OpenAI()
    document_embeddings = []
    for doc in docs:
        response = (
            openai.embeddings.create(input=doc, model="text-embedding-3-small")
            .data[0]
            .embedding
        )
        document_embeddings.append(response)
    return document_embeddings

article_embeddings = docs_to_embeddings(articles) # Remarque : cette opération ne serait généralement effectuée qu'une seule fois avec vos articles, les embeddings et métadonnées étant ensuite stockés dans une base de données

TypeScript

require('dotenv').config();
import { OpenAI } from 'openai';
import * as weave from 'weave';

interface Article {
    text: string;
    embedding?: number[];
}

const articles: Article[] = [
    { 
        text: `Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too, one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.`
    },
    { 
        text: `Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.`
    },
    { 
        text: `Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities`
    }
];

function cosineSimilarity(a: number[], b: number[]): number {
    const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
    const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
    const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
    return dotProduct / (magnitudeA * magnitudeB);
}

const docsToEmbeddings = weave.op(async function(docs: Article[]): Promise<Article[]> {
    const openai = new OpenAI();
    const enrichedDocs = await Promise.all(docs.map(async (doc) => {
        const response = await openai.embeddings.create({
            input: doc.text,
            model: "text-embedding-3-small"
        });
        return {
            ...doc,
            embedding: response.data[0].embedding
        };
    }));
    return enrichedDocs;
});

​

Créer une application RAG

Ensuite, encapsulez la fonction de récupération get_most_relevant_document avec le décorateur weave.op(), puis créez une classe Model. Appelez weave.init('<team-name>/rag-quickstart') pour commencer à suivre toutes les entrées et sorties de vos fonctions afin de pouvoir les inspecter plus tard. Si vous ne spécifiez pas de nom de Team, la sortie est enregistrée dans votre Team ou entité W&B par défaut.

Python

from openai import OpenAI
import weave
from weave import Model
import numpy as np
import asyncio

@weave.op()
def get_most_relevant_document(query):
    openai = OpenAI()
    query_embedding = (
        openai.embeddings.create(input=query, model="text-embedding-3-small")
        .data[0]
        .embedding
    )
    similarities = [
        np.dot(query_embedding, doc_emb)
        / (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb))
        for doc_emb in article_embeddings
    ]
    # Obtenir l'index du document le plus similaire
    most_relevant_doc_index = np.argmax(similarities)
    return articles[most_relevant_doc_index]

class RAGModel(Model):
    system_message: str
    model_name: str = "gpt-3.5-turbo-1106"

    @weave.op()
    def predict(self, question: str) -> dict: # remarque : `question` sera utilisé plus tard pour sélectionner des données dans nos lignes d'évaluation
        from openai import OpenAI
        context = get_most_relevant_document(question)
        client = OpenAI()
        query = f"""Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know."
        Context:
        \"\"\"
        {context}
        \"\"\"
        Question: {question}"""
        response = client.chat.completions.create(
            model=self.model_name,
            messages=[
                {"role": "system", "content": self.system_message},
                {"role": "user", "content": query},
            ],
            temperature=0.0,
            response_format={"type": "text"},
        )
        answer = response.choices[0].message.content
        return {'answer': answer, 'context': context}

# Définissez vos noms de Team et de projet
weave.init('<team-name>/rag-quickstart')
model = RAGModel(
    system_message="You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source."
)
model.predict("What significant result was reported about Zealand Pharma's obesity trial?")

TypeScript

class RAGModel {
    private openai: OpenAI;
    private systemMessage: string;
    private modelName: string;
    private articleEmbeddings: Article[];

    constructor(config: {
        systemMessage: string;
        modelName?: string;
        articleEmbeddings: Article[];
    }) {
        this.openai = new OpenAI();
        this.systemMessage = config.systemMessage;
        this.modelName = config.modelName || "gpt-3.5-turbo-1106";
        this.articleEmbeddings = config.articleEmbeddings;
        this.predict = weave.op(this, this.predict);
    }

    async predict(question: string): Promise<{
        answer: string;
        context: string;
    }> {
        const context = await this.getMostRelevantDocument(question);
        
        const response = await this.openai.chat.completions.create({
            model: this.modelName,
            messages: [
                { role: "system", content: this.systemMessage },
                { role: "user", content: `Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know."
                    Context:
                    """
                    ${context}
                    """
                    Question: ${question}` }
            ],
            temperature: 0
        });

        return {
            answer: response.choices[0].message.content || "",
            context
        };
    }
}

​

Évaluer avec un juge LLM

Lorsqu’il n’existe pas de moyen simple d’évaluer votre application, une approche consiste à utiliser un LLM pour en évaluer certains aspects. Voici un exemple d’utilisation d’un juge LLM pour essayer de mesurer la précision du contexte, en lui demandant de vérifier si le contexte a été utile pour parvenir à la réponse fournie. Ce prompt a été adapté à partir du framework RAGAS.

​

Définir une fonction de score

Comme dans le tutoriel Créer un pipeline d’Évaluation, définissez un ensemble de lignes d’exemple pour tester votre application, ainsi qu’une fonction de score. La fonction de score prend une ligne et l’évalue. Les arguments d’entrée doivent correspondre aux clés de votre ligne ; ici, question provient donc du dictionnaire de la ligne. output est la sortie du modèle. L’entrée du modèle est récupérée depuis l’exemple en fonction de son argument d’entrée ; ici aussi, il s’agit de question. Cet exemple utilise des fonctions async pour s’exécuter rapidement en parallèle. Si vous avez besoin d’une brève introduction à l’async, vous pouvez en trouver une ici.

Python

from openai import OpenAI
import weave
import asyncio

@weave.op()
async def context_precision_score(question, output):
    context_precision_prompt = """Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output.
    Output in only valid JSON format.

    question: {question}
    context: {context}
    answer: {answer}
    verdict: """
    client = OpenAI()

    prompt = context_precision_prompt.format(
        question=question,
        context=output['context'],
        answer=output['answer'],
    )

    response = client.chat.completions.create(
        model="gpt-4-turbo-preview",
        messages=[{"role": "user", "content": prompt}],
        response_format={ "type": "json_object" }
    )
    response_message = response.choices[0].message
    response = json.loads(response_message.content)
    return {
        "verdict": int(response["verdict"]) == 1,
    }

questions = [
    {"question": "What significant result was reported about Zealand Pharma's obesity trial?"},
    {"question": "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?"},
    {"question": "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?"},
    {"question": "What were Rivian and Lucid's vehicle production forecasts for 2024?"},
    {"question": "Why was the Norwegian Dawn cruise ship denied access to Mauritius?"},
    {"question": "Which company achieved the first U.S. moon landing since 1972?"},
    {"question": "What issue did Intuitive Machines' lunar lander encounter upon landing on the moon?"}
]
evaluation = weave.Evaluation(dataset=questions, scorers=[context_precision_score])
asyncio.run(evaluation.evaluate(model)) # remarque : vous devrez définir un modèle à évaluer

TypeScript

const contextPrecisionScore = weave.op(async function(args: {
    datasetRow: QuestionRow;
    modelOutput: { answer: string; context: string; }
}): Promise<ScorerResult> {
    const openai = new OpenAI();
    
    const prompt = `Given question, answer and context verify if the context was useful...`;

    const response = await openai.chat.completions.create({
        model: "gpt-4-turbo-preview",
        messages: [{ role: "user", content: prompt }],
        response_format: { type: "json_object" }
    });

    const result = JSON.parse(response.choices[0].message.content || "{}");
    return {
        verdict: parseInt(result.verdict) === 1
    };
});

const evaluation = new weave.Evaluation({
    dataset: createQuestionDataset(),
    scorers: [contextPrecisionScore]
});

await evaluation.evaluate({
    model: weave.op((args: { datasetRow: QuestionRow }) => 
        model.predict(args.datasetRow.question)
    )
});

​

Facultatif : définir une classe Scorer

Dans certaines applications, vous pouvez vouloir créer des classes d’évaluation personnalisées — par exemple lorsqu’il faut créer une classe LLMJudge standardisée avec des paramètres spécifiques (p. ex. modèle de chat, prompt), une logique de score propre à chaque ligne et un calcul spécifique du score agrégé. Weave propose une liste de classes Scorer prêtes à l’emploi et permet aussi de créer facilement un Scorer personnalisé. L’exemple suivant montre comment créer une class CorrectnessLLMJudge(Scorer) personnalisée.

Dans les grandes lignes, les étapes pour créer un Scorer personnalisé sont assez simples :

1. Définissez une classe personnalisée qui hérite de weave.flow.scorer.Scorer
2. Redéfinissez la fonction score et ajoutez @weave.op() si vous voulez suivre chaque appel de la fonction
   • cette fonction doit définir un argument output dans lequel la prédiction du modèle sera transmise. Définissez-le avec le type Optional[dict] au cas où le modèle renverrait "None".
   • les autres arguments peuvent être de type Any ou dict, ou correspondre à des colonnes spécifiques du jeu de données utilisé pour évaluer le modèle avec la classe weave.Evaluate — ils doivent avoir exactement les mêmes noms que les noms de colonnes ou les clés d’une ligne après son passage à preprocess_model_input, si celle-ci est utilisée.
3. Facultatif : Redéfinissez la fonction summarize pour personnaliser le calcul du score agrégé. Par défaut, Weave utilise la fonction weave.flow.scorer.auto_summarize si vous ne définissez pas de fonction personnalisée.
   • cette fonction doit avoir un décorateur @weave.op().

Python

from weave import Scorer

class CorrectnessLLMJudge(Scorer):
    prompt: str
    model_name: str
    device: str

    @weave.op()
    async def score(self, output: Optional[dict], query: str, answer: str) -> Any:
        """Évalue l’exactitude des prédictions en comparant pred, query et target.
        Args:
            - output: le dict fourni par le modèle évalué
            - query: la question posée, telle que définie dans le jeu de données
            - answer: la réponse cible, telle que définie dans le jeu de données
        Returns:
            - dict unique {nom de métrique: valeur d’évaluation unique}"""

        # get_model est défini comme un récupérateur de modèle générique selon les paramètres fournis (OpenAI,HF...)
        eval_model = get_model(
            model_name = self.model_name,
            prompt = self.prompt
            device = self.device,
        )
        # évaluation asynchrone pour accélérer l’évaluation - cela n’a pas besoin d’être asynchrone
        grade = await eval_model.async_predict(
            {
                "query": query,
                "answer": answer,
                "result": output.get("result"),
            }
        )
        # analyse de la sortie - cela pourrait être fait de manière plus robuste avec pydantic
        evaluation = "incorrect" not in grade["text"].strip().lower()

        # le nom de colonne affiché dans Weave
        return {"correct": evaluation}

    @weave.op()
    def summarize(self, score_rows: list) -> Optional[dict]:
        """Agrège tous les scores calculés pour chaque ligne par la fonction de score.
        Args:
            - score_rows: une liste de dicts. Chaque dict contient des métriques et des scores
        Returns:
            - dict imbriqué avec la même structure que l’entrée"""

        # si rien n’est fourni, la fonction weave.flow.scorer.auto_summarize est utilisée
        # return auto_summarize(score_rows)

        valid_data = [x.get("correct") for x in score_rows if x.get("correct") is not None]
        count_true = list(valid_data).count(True)
        int_data = [int(x) for x in valid_data]

        sample_mean = np.mean(int_data) if int_data else 0
        sample_variance = np.var(int_data) if int_data else 0
        sample_error = np.sqrt(sample_variance / len(int_data)) if int_data else 0

        # la couche supplémentaire "correct" n’est pas nécessaire, mais ajoute de la structure dans l’UI
        return {
            "correct": {
                "true_count": count_true,
                "true_fraction": sample_mean,
                "stderr": sample_error,
            }
        }

TypeScript

Cette fonctionnalité n’est pas encore disponible en TypeScript.

Pour l’utiliser comme scorer, vous devez l’initialiser et le passer à l’argument scorers dans votre `Evaluation comme ceci :

Python

evaluation = weave.Evaluation(dataset=questions, scorers=[CorrectnessLLMJudge()])

TypeScript

Cette fonctionnalité n'est pas encore disponible en TypeScript.

​

Récapitulons

Pour obtenir le même résultat avec vos applications RAG :

• Encapsulez les appels LLM et les fonctions d’étape de récupération avec weave.op()
• (facultatif) Créez une sous-classe Model avec une fonction predict et les détails de votre application
• Rassemblez des exemples pour l’évaluation
• Créez des fonctions de score qui évaluent un exemple
• Utilisez la classe Evaluation pour exécuter des évaluations sur vos exemples

REMARQUE : Il arrive que l’exécution asynchrone des évaluations atteigne la limite de débit des modèles d’OpenAI, Anthropic, etc. Pour l’éviter, vous pouvez définir une variable d’environnement afin de limiter le nombre de workers parallèles, par exemple WEAVE_PARALLELISM=3.

Voici le code dans son intégralité.

Python

from openai import OpenAI
import weave
from weave import Model
import numpy as np
import json
import asyncio

# Exemples à utiliser pour les évaluations
articles = [
    "Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial “tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too,” one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.",
    "Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.",
    "Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if it's stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities",
    "Rivian and Lucid shares plunge after weak EV earnings reports Shares of electric vehicle makers Rivian and Lucid fell Thursday after the companies reported stagnant production in their fourth-quarter earnings after the bell Wednesday. Rivian shares sank about 25 percent, and Lucids stock dropped around 17 percent. Rivian forecast it will make 57,000 vehicles in 2024, slightly less than the 57,232 vehicles it produced in 2023. Lucid said it expects to make 9,000 vehicles in 2024, more than the 8,428 vehicles it made in 2023.",
    "Mauritius blocks Norwegian cruise ship over fears of a potential cholera outbreak Local authorities on Sunday denied permission for the Norwegian Dawn ship, which has 2,184 passengers and 1,026 crew on board, to access the Mauritius capital of Port Louis, citing “potential health risks.” The Mauritius Ports Authority said Sunday that samples were taken from at least 15 passengers on board the cruise ship. A spokesperson for the U.S.-headquartered Norwegian Cruise Line Holdings said Sunday that 'a small number of guests experienced mild symptoms of a stomach-related illness' during Norwegian Dawns South Africa voyage.",
    "Intuitive Machines lands on the moon in historic first for a U.S. company Intuitive Machines Nova-C cargo lander, named Odysseus after the mythological Greek hero, is the first U.S. spacecraft to soft land on the lunar surface since 1972. Intuitive Machines is the first company to pull off a moon landing — government agencies have carried out all previously successful missions. The company's stock surged in extended trading Thursday, after falling 11 percent in regular trading.",
    "Lunar landing photos: Intuitive Machines Odysseus sends back first images from the moon Intuitive Machines cargo moon lander Odysseus returned its first images from the surface. Company executives believe the lander caught its landing gear sideways on the surface of the moon while touching down and tipped over. Despite resting on its side, the company's historic IM-1 mission is still operating on the moon.",
]

def docs_to_embeddings(docs: list) -> list:
    openai = OpenAI()
    document_embeddings = []
    for doc in docs:
        response = (
            openai.embeddings.create(input=doc, model="text-embedding-3-small")
            .data[0]
            .embedding
        )
        document_embeddings.append(response)
    return document_embeddings

article_embeddings = docs_to_embeddings(articles) # Remarque : cette opération ne serait généralement effectuée qu'une seule fois avec vos articles, les embeddings et métadonnées étant ensuite stockés dans une base de données

# Ajouter un décorateur à l'étape de récupération
@weave.op()
def get_most_relevant_document(query):
    openai = OpenAI()
    query_embedding = (
        openai.embeddings.create(input=query, model="text-embedding-3-small")
        .data[0]
        .embedding
    )
    similarities = [
        np.dot(query_embedding, doc_emb)
        / (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb))
        for doc_emb in article_embeddings
    ]
    # Obtenir l'index du document le plus similaire
    most_relevant_doc_index = np.argmax(similarities)
    return articles[most_relevant_doc_index]

# Créer une sous-classe Model avec les détails de l'application, ainsi qu'une fonction predict qui produit une réponse
class RAGModel(Model):
    system_message: str
    model_name: str = "gpt-3.5-turbo-1106"

    @weave.op()
    def predict(self, question: str) -> dict: # remarque : `question` sera utilisé plus tard pour sélectionner des données dans nos lignes d'évaluation
        from openai import OpenAI
        context = get_most_relevant_document(question)
        client = OpenAI()
        query = f"""Utilisez les informations suivantes pour répondre à la question ci-dessous. Si la réponse est introuvable, écrivez « Je ne sais pas. »
        Context:
        \"\"\"
        {context}
        \"\"\"
        Question: {question}"""
        response = client.chat.completions.create(
            model=self.model_name,
            messages=[
                {"role": "system", "content": self.system_message},
                {"role": "user", "content": query},
            ],
            temperature=0.0,
            response_format={"type": "text"},
        )
        answer = response.choices[0].message.content
        return {'answer': answer, 'context': context}

# Définir les noms de votre équipe et de votre projet
weave.init('<team-name>/rag-quickstart')
model = RAGModel(
    system_message="Vous êtes un expert en finance et répondez aux questions relatives à la finance, aux services financiers et aux marchés financiers. Lorsque vous répondez en vous basant sur des informations fournies, veillez à citer la source."
)

# Voici notre fonction de scoring qui utilise notre question et la sortie pour produire un score
@weave.op()
async def context_precision_score(question, output):
    context_precision_prompt = """Étant donné la question, la réponse et le contexte, vérifiez si le contexte a été utile pour parvenir à la réponse donnée. Donnez un verdict « 1 » si utile et « 0 » sinon, avec une sortie JSON.
    Sortie en format JSON valide uniquement.

    question: {question}
    context: {context}
    answer: {answer}
    verdict: """
    client = OpenAI()

    prompt = context_precision_prompt.format(
        question=question,
        context=output['context'],
        answer=output['answer'],
    )

    response = client.chat.completions.create(
        model="gpt-4-turbo-preview",
        messages=[{"role": "user", "content": prompt}],
        response_format={ "type": "json_object" }
    )
    response_message = response.choices[0].message
    response = json.loads(response_message.content)
    return {
        "verdict": int(response["verdict"]) == 1,
    }

questions = [
    {"question": "Quel résultat significatif a été rapporté concernant l'essai sur l'obésité de Zealand Pharma ?"},
    {"question": "De combien les niveaux de trésorerie de Berkshire Hathaway ont-ils augmenté au quatrième trimestre ?"},
    {"question": "Quel est l'objectif de l'intégration par Highmark Health des technologies de Google Cloud et d'Epic Systems ?"},
    {"question": "Quelles étaient les prévisions de production de véhicules de Rivian et Lucid pour 2024 ?"},
    {"question": "Pourquoi le navire de croisière Norwegian Dawn s'est-il vu refuser l'accès à l'île Maurice ?"},
    {"question": "Quelle entreprise a réalisé le premier alunissage américain depuis 1972 ?"},
    {"question": "Quel problème le module lunaire d'Intuitive Machines a-t-il rencontré lors de son atterrissage sur la lune ?"}
]

# Définir un objet Evaluation et lui passer des exemples de questions ainsi que des fonctions de scoring
evaluation = weave.Evaluation(dataset=questions, scorers=[context_precision_score])
asyncio.run(evaluation.evaluate(model))

TypeScript

require('dotenv').config();
import { OpenAI } from 'openai';
import * as weave from 'weave';

interface Article {
    text: string;
    embedding?: number[];
}

const articles: Article[] = [
    { 
        text: `Novo Nordisk and Eli Lilly rival soars 32 percent after promising weight loss drug results Shares of Denmarks Zealand Pharma shot 32 percent higher in morning trade, after results showed success in its liver disease treatment survodutide, which is also on trial as a drug to treat obesity. The trial tells us that the 6mg dose is safe, which is the top dose used in the ongoing [Phase 3] obesity trial too, one analyst said in a note. The results come amid feverish investor interest in drugs that can be used for weight loss.`
    },
    { 
        text: `Berkshire shares jump after big profit gain as Buffetts conglomerate nears $1 trillion valuation Berkshire Hathaway shares rose on Monday after Warren Buffetts conglomerate posted strong earnings for the fourth quarter over the weekend. Berkshires Class A and B shares jumped more than 1.5%, each. Class A shares are higher by more than 17% this year, while Class B has gained more than 18%. Berkshire was last valued at $930.1 billion, up from $905.5 billion where it closed on Friday, according to FactSet. Berkshire on Saturday posted fourth-quarter operating earnings of $8.481 billion, about 28 percent higher than the $6.625 billion from the year-ago period, driven by big gains in its insurance business. Operating earnings refers to profits from businesses across insurance, railroads and utilities. Meanwhile, Berkshires cash levels also swelled to record levels. The conglomerate held $167.6 billion in cash in the fourth quarter, surpassing the $157.2 billion record the conglomerate held in the prior quarter.`
    },
    { 
        text: `Highmark Health says its combining tech from Google and Epic to give doctors easier access to information Highmark Health announced it is integrating technology from Google Cloud and the health-care software company Epic Systems. The integration aims to make it easier for both payers and providers to access key information they need, even if its stored across multiple points and formats, the company said. Highmark is the parent company of a health plan with 7 million members, a provider network of 14 hospitals and other entities`
    }
];

function cosineSimilarity(a: number[], b: number[]): number {
    const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
    const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
    const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
    return dotProduct / (magnitudeA * magnitudeB);
}

const docsToEmbeddings = weave.op(async function(docs: Article[]): Promise<Article[]> {
    const openai = new OpenAI();
    const enrichedDocs = await Promise.all(docs.map(async (doc) => {
        const response = await openai.embeddings.create({
            input: doc.text,
            model: "text-embedding-3-small"
        });
        return {
            ...doc,
            embedding: response.data[0].embedding
        };
    }));
    return enrichedDocs;
});

class RAGModel {
    private openai: OpenAI;
    private systemMessage: string;
    private modelName: string;
    private articleEmbeddings: Article[];

    constructor(config: {
        systemMessage: string;
        modelName?: string;
        articleEmbeddings: Article[];
    }) {
        this.openai = new OpenAI();
        this.systemMessage = config.systemMessage;
        this.modelName = config.modelName || "gpt-3.5-turbo-1106";
        this.articleEmbeddings = config.articleEmbeddings;
        this.predict = weave.op(this, this.predict);
    }

    private async getMostRelevantDocument(query: string): Promise<string> {
        const queryEmbedding = await this.openai.embeddings.create({
            input: query,
            model: "text-embedding-3-small"
        });

        const similarities = this.articleEmbeddings.map(doc => {
            if (!doc.embedding) return 0;
            return cosineSimilarity(queryEmbedding.data[0].embedding, doc.embedding);
        });

        const mostRelevantIndex = similarities.indexOf(Math.max(...similarities));
        return this.articleEmbeddings[mostRelevantIndex].text;
    }

    async predict(question: string): Promise<{
        answer: string;
        context: string;
    }> {
        const context = await this.getMostRelevantDocument(question);
        
        const response = await this.openai.chat.completions.create({
            model: this.modelName,
            messages: [
                { role: "system", content: this.systemMessage },
                { 
                    role: "user", 
                    content: `Use the following information to answer the subsequent question. If the answer cannot be found, write "I don't know."
                    Context:
                    """
                    ${context}
                    """
                    Question: ${question}`
                }
            ],
            temperature: 0
        });

        return {
            answer: response.choices[0].message.content || "",
            context
        };
    }
}

interface ScorerResult {
    verdict: boolean;
}

interface QuestionRow {
    question: string;
}

function createQuestionDataset(): weave.Dataset<QuestionRow> {
    return new weave.Dataset<QuestionRow>({
        id: 'rag-questions',
        rows: [
            { question: "What significant result was reported about Zealand Pharma's obesity trial?" },
            { question: "How much did Berkshire Hathaway's cash levels increase in the fourth quarter?" },
            { question: "What is the goal of Highmark Health's integration of Google Cloud and Epic Systems technology?" }
        ]
    });
}

const contextPrecisionScore = weave.op(async function(args: {
    datasetRow: QuestionRow;
    modelOutput: { answer: string; context: string; }
}): Promise<ScorerResult> {
    const openai = new OpenAI();
    
    const prompt = `Given question, answer and context verify if the context was useful in arriving at the given answer. Give verdict as "1" if useful and "0" if not with json output.
    Output in only valid JSON format.

    question: ${args.datasetRow.question}
    context: ${args.modelOutput.context}
    answer: ${args.modelOutput.answer}
    verdict: `;

    const response = await openai.chat.completions.create({
        model: "gpt-4-turbo-preview",
        messages: [{ role: "user", content: prompt }],
        response_format: { type: "json_object" }
    });

    const result = JSON.parse(response.choices[0].message.content || "{}");
    return {
        verdict: parseInt(result.verdict) === 1
    };
});

async function main() {
    # Définissez les noms de votre équipe et de votre projet
    await weave.init('<team-name>/rag-quickstart');
    
    const articleEmbeddings = await docsToEmbeddings(articles);
    
    const model = new RAGModel({
        systemMessage: "You are an expert in finance and answer questions related to finance, financial services, and financial markets. When responding based on provided information, be sure to cite the source.",
        articleEmbeddings
    });

    const evaluation = new weave.Evaluation({
        dataset: createQuestionDataset(),
        scorers: [contextPrecisionScore]
    });

    const results = await evaluation.evaluate({
        model: weave.op((args: { datasetRow: QuestionRow }) => 
            model.predict(args.datasetRow.question)
        )
    });
    
    console.log('Résultats de l\'évaluation :', results);
}

if (require.main === module) {
    main().catch(console.error);
}

​

Conclusion

Ce tutoriel vous a montré comment intégrer l’observabilité à différentes étapes de vos applications, comme l’étape de récupération dans cet exemple. Vous avez également appris à créer des fonctions de score plus complexes, comme un juge LLM, pour évaluer automatiquement les réponses de l’application.

​

Prochaines étapes

Consultez le cours RAG++ pour approfondir les techniques pratiques de RAG destinées aux ingénieurs. Vous y découvrirez des solutions prêtes pour la Production proposées par Weights & Biases, Cohere et Weaviate afin d’optimiser les performances, de réduire les coûts et d’améliorer la précision et la pertinence de vos applications.