On this tutorial, we dive deep into how we systematically benchmark agentic elements by evaluating a number of reasoning methods throughout numerous duties. We discover how completely different architectures, similar to Direct, Chain-of-Thought, ReAct, and Reflexion, behave when confronted with issues of accelerating issue, and we quantify their accuracy, effectivity, latency, and tool-usage patterns. By conducting managed empirical research, we achieve a clearer understanding of why sure agentic methods succeed, the place they fail, and the way they commerce off pace for depth of reasoning. Take a look at the FULL CODES right here.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from typing import Record, Dict, Callable, Tuple
from dataclasses import dataclass
from enum import Enum
import time
from collections import defaultdict
class ReasoningStrategy(Enum):
DIRECT = "direct"
CHAIN_OF_THOUGHT = "chain_of_thought"
REACT = "react"
REFLEXION = "reflexion"
@dataclass
class AgentResponse:
reply: str
steps: int
time_taken: float
tool_calls: int
confidence: float
class BaseAgent:
def __init__(self, technique: ReasoningStrategy):
self.technique = technique
self.tool_count = 0
def clear up(self, drawback: str) -> AgentResponse:
start_time = time.time()
if self.technique == ReasoningStrategy.DIRECT:
reply, steps, instruments = self._direct_solve(drawback)
elif self.technique == ReasoningStrategy.CHAIN_OF_THOUGHT:
reply, steps, instruments = self._cot_solve(drawback)
elif self.technique == ReasoningStrategy.REACT:
reply, steps, instruments = self._react_solve(drawback)
else:
reply, steps, instruments = self._reflexion_solve(drawback)
time_taken = time.time() - start_time
confidence = self._calculate_confidence(drawback, reply)
return AgentResponse(reply, steps, time_taken, instruments, confidence)We arrange the muse of our benchmarking framework by importing important libraries and defining the core agent architectures. We set up completely different reasoning methods and assemble the BaseAgent class, giving ourselves a versatile construction to simulate numerous agentic behaviors. By way of this setup, we set up a unified interface that each one brokers observe throughout analysis. Take a look at the FULL CODES right here.
def _direct_solve(self, drawback: str) -> Tuple[str, int, int]:
reply = self._compute_answer(drawback)
return reply, 1, 0
def _cot_solve(self, drawback: str) -> Tuple[str, int, int]:
steps = 3 + len(drawback.break up()) // 5
for i in vary(steps):
_ = self._reason_step(drawback, i)
reply = self._compute_answer(drawback)
return reply, steps, 0
def _react_solve(self, drawback: str) -> Tuple[str, int, int]:
steps = 4
tool_calls = 2
for i in vary(steps):
_ = self._reason_step(drawback, i)
if i % 2 == 0:
self._use_tool(drawback)
reply = self._compute_answer(drawback)
return reply, steps, tool_calls
def _reflexion_solve(self, drawback: str) -> Tuple[str, int, int]:
steps = 6
tool_calls = 1
initial_answer = self._compute_answer(drawback)
reflection = self._reflect(drawback, initial_answer)
reply = self._refine(drawback, initial_answer, reflection)
return reply, steps, tool_calls
def _reason_step(self, drawback: str, step: int) -> str:
return f"Analyzing side {step+1}"
def _use_tool(self, drawback: str):
self.tool_count += 1
time.sleep(0.001)
def _compute_answer(self, drawback: str) -> str:
return f"Solution_{hash(drawback) % 100}"
def _reflect(self, drawback: str, reply: str) -> str:
return "Reflection on method"
def _refine(self, drawback: str, reply: str, reflection: str) -> str:
return f"Refined_{reply}"
def _calculate_confidence(self, drawback: str, reply: str) -> float:
base_confidence = 0.7
strategy_bonus = {
ReasoningStrategy.DIRECT: 0.0,
ReasoningStrategy.CHAIN_OF_THOUGHT: 0.1,
ReasoningStrategy.REACT: 0.15,
ReasoningStrategy.REFLEXION: 0.2
}
return min(1.0, base_confidence + strategy_bonus[self.strategy] + np.random.uniform(-0.1, 0.1))We implement how every reasoning technique behaves internally, together with direct answering, chain-of-thought reasoning, ReAct-style interleaving, and Reflexion-based refinement. We simulate reasoning steps, software utilization, and confidence estimation to seize sensible agent habits patterns. Right here, we form the dynamic persona of every agentic technique we benchmark. Take a look at the FULL CODES right here.
class BenchmarkTask:
def __init__(self, identify: str, issue: float, ground_truth: str):
self.identify = identify
self.issue = issue
self.ground_truth = ground_truth
def consider(self, response: AgentResponse) -> Dict[str, float]:
accuracy = response.confidence * (1 - self.issue * 0.3)
return {
'accuracy': accuracy,
'effectivity': 1.0 / (response.steps + 1),
'latency': response.time_taken,
'tool_efficiency': 1.0 / (response.tool_calls + 1)
}
class BenchmarkSuite:
def __init__(self):
self.duties = self._create_tasks()
def _create_tasks(self) -> Record[BenchmarkTask]:
duties = []
task_types = [
("Math_Problem", 0.3),
("Logic_Puzzle", 0.5),
("Code_Debug", 0.6),
("Complex_Reasoning", 0.8),
("Multi_Step_Planning", 0.7)
]
for i, (task_type, issue) in enumerate(task_types):
for j in vary(3):
activity = BenchmarkTask(
identify=f"{task_type}_{j+1}",
issue=issue + np.random.uniform(-0.1, 0.1),
ground_truth=f"GT_{i}_{j}"
)
duties.append(activity)
return duties
def run_benchmark(self, brokers: Record[BaseAgent]) -> pd.DataFrame:
outcomes = []
for agent in brokers:
for activity in self.duties:
response = agent.clear up(activity.identify)
metrics = activity.consider(response)
outcomes.append({
'technique': agent.technique.worth,
'activity': activity.identify,
'issue': activity.issue,
'accuracy': metrics['accuracy'],
'effectivity': metrics['efficiency'],
'latency': metrics['latency'],
'tool_efficiency': metrics['tool_efficiency'],
'steps': response.steps,
'tool_calls': response.tool_calls
})
return pd.DataFrame(outcomes)We construct the entire benchmark suite that generates duties, executes them throughout a number of brokers, and collects standardized outcomes. We design assorted activity sorts and issue ranges to look at how every reasoning technique adapts beneath stress. This snippet permits us to create a reproducible and systematic analysis pipeline. Take a look at the FULL CODES right here.
def analyze_results(df: pd.DataFrame):
agg_metrics = df.groupby('technique').agg({
'accuracy': ['mean', 'std'],
'effectivity': ['mean', 'std'],
'latency': ['mean', 'std'],
'steps': 'imply',
'tool_calls': 'imply'
}).spherical(3)
print(agg_metrics)
diff_bins = pd.minimize(df['difficulty'], bins=3, labels=['Easy', 'Medium', 'Hard'])
diff_analysis = df.groupby(['strategy', diff_bins])['accuracy'].imply().unstack()
print(diff_analysis.spherical(3))
tradeoff = df.groupby('technique').agg({
'accuracy': 'imply',
'steps': 'imply',
'latency': 'imply'
})
tradeoff['score'] = (tradeoff['accuracy'] / (tradeoff['steps'] * tradeoff['latency'])).spherical(3)
print(tradeoff.spherical(3))
def visualize_results(df: pd.DataFrame):
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
sns.barplot(information=df, x='technique', y='accuracy', ax=axes[0, 0], errorbar="sd")
axes[0, 0].set_title('Accuracy by Technique')
axes[0, 0].tick_params(axis="x", rotation=45)
for technique in df['strategy'].distinctive():
strategy_df = df[df['strategy'] == technique]
axes[0, 1].scatter(strategy_df['steps'], strategy_df['accuracy'], label=technique, alpha=0.6, s=50)
axes[0, 1].set_title('Steps vs Accuracy')
axes[0, 1].legend()
difficulty_bins = pd.minimize(df['difficulty'], bins=3, labels=['Easy', 'Medium', 'Hard'])
df_plot = df.copy()
df_plot['difficulty_bin'] = difficulty_bins
sns.boxplot(information=df_plot, x='difficulty_bin', y='accuracy', hue="technique", ax=axes[1, 0])
axes[1, 0].set_title('Efficiency vs Problem')
scores = df.groupby('technique').apply(
lambda x: x['accuracy'].imply() / (x['steps'].imply() * x['latency'].imply())
).sort_values()
axes[1, 1].barh(vary(len(scores)), scores.values)
axes[1, 1].set_yticks(vary(len(scores)))
axes[1, 1].set_yticklabels(scores.index)
axes[1, 1].set_title('Total Effectivity Rating')
plt.tight_layout()
plt.present()We carry out detailed evaluation and visualization to know how methods differ throughout metrics like accuracy, effectivity, and latency. We combination outcomes, evaluate efficiency throughout issue ranges, and visualize trade-offs to uncover deeper insights. This step empowers us to interpret the outcomes fairly than simply compute them. Take a look at the FULL CODES right here.
if __name__ == "__main__":
brokers = [
BaseAgent(ReasoningStrategy.DIRECT),
BaseAgent(ReasoningStrategy.CHAIN_OF_THOUGHT),
BaseAgent(ReasoningStrategy.REACT),
BaseAgent(ReasoningStrategy.REFLEXION)
]
suite = BenchmarkSuite()
results_df = suite.run_benchmark(brokers)
analyze_results(results_df)
visualize_results(results_df)
print("1. Superior methods obtain larger accuracy however require extra steps")
print("2. Chain-of-thought balances accuracy and effectivity")
print("3. Direct is quickest however much less dependable on onerous duties")
print("4. All methods degrade on tougher duties however superior ones degrade slowly")We carry every thing collectively by operating the benchmark suite on all brokers and printing the important thing findings. We execute the evaluation pipeline, visualize comparative outcomes, and interpret how methods behave beneath an identical circumstances. This snippet completes the loop, permitting us to look at empirical patterns and derive significant conclusions.
In conclusion, we observe how completely different agentic reasoning paradigms carry out when subjected to an identical benchmark circumstances, and we achieve sensible perception into how these methods scale with growing complexity. As we analyze patterns in accuracy, step rely, latency, and power effectivity, we acknowledge how superior methods succeed by way of deeper reasoning whereas incurring computational overhead. We now stand outfitted with a structured empirical framework that helps us evaluate, debug, and optimize agentic behaviors, permitting us to construct extra succesful, data-driven agentic methods.
Take a look at the FULL CODES right here. Be at liberty to take a look at our GitHub Web page for Tutorials, Codes and Notebooks. Additionally, be at liberty to observe us on Twitter and don’t overlook to affix our 100k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you’ll be able to be part of us on telegram as properly.
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.
Elevate your perspective with NextTech Information, the place innovation meets perception.
Uncover the most recent breakthroughs, get unique updates, and join with a world community of future-focused thinkers.
Unlock tomorrow’s developments right this moment: learn extra, subscribe to our publication, and develop into a part of the NextTech group at NextTech-news.com
