Getting Began with MLFlow for LLM Analysis

MLflow is a strong open-source platform for managing the machine studying lifecycle. Whereas it’s historically used for monitoring mannequin experiments, logging parameters, and managing deployments, MLflow has just lately launched help for evaluating Massive Language Fashions (LLMs).

On this tutorial, we discover easy methods to use MLflow to guage the efficiency of an LLM—in our case, Google’s Gemini mannequin—on a set of fact-based prompts. We’ll generate responses to fact-based prompts utilizing Gemini and assess their high quality utilizing a wide range of metrics supported immediately by MLflow.

Organising the dependencies

For this tutorial, we’ll be utilizing each the OpenAI and Gemini APIs. MLflow’s built-in generative AI analysis metrics at present depend on OpenAI fashions (e.g., GPT-4) to behave as judges for metrics like reply similarity or faithfulness, so an OpenAI API secret’s required. You possibly can get hold of:

Putting in the libraries

pip set up mlflow openai pandas google-genai

Setting the OpenAI and Google API Keys as surroundings variable

import os
from getpass import getpass

os.environ["OPENAI_API_KEY"] = getpass('Enter OpenAI API Key:')
os.environ["GOOGLE_API_KEY"] = getpass('Enter Google API Key:')

Getting ready Analysis Knowledge and Fetching Outputs from Gemini

import mlflow
import openai
import os
import pandas as pd
from google import genai

Creating the analysis knowledge

On this step, we outline a small analysis dataset containing factual prompts together with their appropriate floor fact solutions. These prompts span subjects equivalent to science, well being, internet improvement, and programming. This structured format permits us to objectively examine the Gemini-generated responses towards recognized appropriate solutions utilizing varied analysis metrics in MLflow.

eval_data = pd.DataFrame(
    {
        "inputs": [
            "Who developed the theory of general relativity?",
            "What are the primary functions of the liver in the human body?",
            "Explain what HTTP status code 404 means.",
            "What is the boiling point of water at sea level in Celsius?",
            "Name the largest planet in our solar system.",
            "What programming language is primarily used for developing iOS apps?",
        ],
        "ground_truth": [
            "Albert Einstein developed the theory of general relativity.",
            "The liver helps in detoxification, protein synthesis, and production of biochemicals necessary for digestion.",
            "HTTP 404 means 'Not Found' -- the server can't find the requested resource.",
            "The boiling point of water at sea level is 100 degrees Celsius.",
            "Jupiter is the largest planet in our solar system.",
            "Swift is the primary programming language used for iOS app development."
        ]
    }
)

eval_data

Getting Gemini Responses

This code block defines a helper perform gemini_completion() that sends a immediate to the Gemini 1.5 Flash mannequin utilizing the Google Generative AI SDK and returns the generated response as plain textual content. We then apply this perform to every immediate in our analysis dataset to generate the mannequin’s predictions, storing them in a brand new “predictions” column. These predictions will later be evaluated towards the bottom fact solutions

consumer = genai.Shopper()
def gemini_completion(immediate: str) -> str:
    response = consumer.fashions.generate_content(
        mannequin="gemini-1.5-flash",
        contents=immediate
    )
    return response.textual content.strip()

eval_data["predictions"] = eval_data["inputs"].apply(gemini_completion)
eval_data

Evaluating Gemini Outputs with MLflow

On this step, we provoke an MLflow run to guage the responses generated by the Gemini mannequin towards a set of factual ground-truth solutions. We use the mlflow.consider() technique with 4 light-weight metrics: answer_similarity (measuring semantic similarity between the mannequin’s output and the bottom fact), exact_match (checking for word-for-word matches), latency (monitoring response technology time), and token_count (logging the variety of output tokens).

It’s necessary to notice that the answer_similarity metric internally makes use of OpenAI’s GPT mannequin to evaluate the semantic closeness between solutions, which is why entry to the OpenAI API is required. This setup gives an environment friendly strategy to assess LLM outputs with out counting on customized analysis logic. The ultimate analysis outcomes are printed and likewise saved to a CSV file for later inspection or visualization.

mlflow.set_tracking_uri("mlruns")
mlflow.set_experiment("Gemini Easy Metrics Eval")

with mlflow.start_run():
    outcomes = mlflow.consider(
        model_type="question-answering",
        knowledge=eval_data,
        predictions="predictions",
        targets="ground_truth",
        extra_metrics=[
          mlflow.metrics.genai.answer_similarity(),
          mlflow.metrics.exact_match(),
          mlflow.metrics.latency(),
          mlflow.metrics.token_count()
      ]
    )
    print("Aggregated Metrics:")
    print(outcomes.metrics)

    # Save detailed desk
    outcomes.tables["eval_results_table"].to_csv("gemini_eval_results.csv", index=False)

To view the detailed outcomes of our analysis, we load the saved CSV file right into a DataFrame and modify the show settings to make sure full visibility of every response. This enables us to examine particular person prompts, Gemini-generated predictions, floor fact solutions, and the related metric scores with out truncation, which is particularly useful in pocket book environments like Colab or Jupyter.

outcomes = pd.read_csv('gemini_eval_results.csv')
pd.set_option('show.max_colwidth', None)
outcomes

Take a look at the Codes right here. All credit score for this analysis goes to the researchers of this undertaking. Additionally, be at liberty to comply with us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our E-newsletter.