1

Honeycomb

Honeycomb provides an observability platform for engineering teams that covers distributed tracing, log analytics, time series metrics, frontend observability, and telemetry pipelines. It also supports observability for AI agents and LLM applications, including tracing multi-agent conversations and LLM cost tracking.

N/A

freemium

United States

2

Fiddler AI

Fiddler AI provides a platform for monitoring and ensuring the integrity of AI models, offering explainable AI and observability tools to manage the lifecycle of ML and LLM applications effectively. It integrates with major cloud providers.

-1

freemium

United States

3

Promptfoo

Promptfoo offers a platform for robust LLM application security testing, enabling developers to run tailored vulnerability scans and receive actionable insights for software protection.

-1

freemium

United States

4

Llm-Stats

LLM Stats operates an independent AI benchmarking platform that ranks over 300 language models by intelligence, speed, and price using a composite score from public benchmarks and live API metrics. It also offers leaderboards, model comparison tools, pricing data, arenas, and a data API with benchmark scores and metadata.

N/A

freemium

United States

5

Deepchecks Monitoring

Deepchecks offers comprehensive evaluation and monitoring tools for LLM applications, leveraging open-source ML testing packages to automate quality control, compliance checks, and risk mitigation, enhancing the reliability and performance of AI systems.

-2

freemium

Israel

6

Inference.net

Inference.net provides a platform for deploying, monitoring, tracing, evaluating, and fine-tuning open-source and custom AI models, including a marketplace-style spot market for underutilized data center GPU capacity. It also offers an OpenAI-compatible gateway, LLM observability, continuous evaluation, and fine-tuning workflows, with SOC 2 Type II certification.

N/A

freemium

United States

7

TruEra

TruEra provides AI-driven solutions for machine learning model monitoring, testing, and quality management, ensuring reliable and trustworthy AI implementation across industries such as banking and manufacturing.

-3

unknown

United States

8

Pydantic

Pydantic provides an end-to-end AI engineering stack that includes Pydantic AI for building agents, Pydantic Logfire for observability, Pydantic Evals, and an AI Gateway. It also maintains the Pydantic data validation library for Python, which validates data using type hints and has a Rust-based core.

N/A

freemium

United States

9

Scade.pro

Scade.pro offers a no-code platform that simplifies AI app development by providing access to over 1,500 models and tools. It reduces complexity with a unified API, allowing easy integration across web and mobile platforms without coding skills.

-4

freemium

Unknown

10

Guardrails

Guardrails AI provides an AI reliability platform and open-source Python framework for building, governing, and scaling production generative AI applications, with input/output guards that detect and mitigate risks. It also offers Snowglobe for simulating synthetic datasets and AI chat scenarios to test and evaluate AI systems before deployment.

N/A

unknown

United States

11

Respan

Respan provides an LLM engineering platform that combines a unified API gateway to over 1,000 models with observability, automated evaluations, and prompt optimization. It also offers tracing, scoring, and monitoring of cost, latency, errors, and tokens with alerts.

N/A

freemium

United States

12

LMSpeed

LMSpeed provides a web platform for comparing and testing LLM API providers and models, aggregating per-token pricing from over 100 providers and running speed benchmarks on latency and throughput. It also offers model benchmark lookups, free API key listings, and provider leaderboards.

N/A

freemium

Unknown

13

MLflow

MLflow is an open-source platform for managing the machine learning and generative AI lifecycle, including experiment tracking, model evaluation, model registry, and deployment. It also provides observability, evaluation, prompt management, and an AI Gateway for LLM applications and agents.

N/A

free

United States

14

Deepeval

DeepEval provides an open-source LLM evaluation framework for testing and benchmarking LLM applications. It offers over 50 research-backed metrics including LLM-as-a-judge, RAG, agent, multi-turn, voice, safety, and multimodal metrics, and runs Pytest-style assertions locally or in CI/CD pipelines.

N/A

unknown

Unknown

15

Mabyduck

Mabyduck provides a self-serve platform for running subjective quality evaluation studies on AI-generated audio, video, images, and interactive experiences. It also offers configurable pairwise comparisons, surveys, rubrics, leaderboards, rater pools, and an API for integrating evaluations into existing pipelines.

N/A

freemium

United Kingdom

16

Superwise

SUPERWISE offers AI governance and monitoring solutions, ensuring compliance and operational efficiency for enterprises across various industries.

-10

unknown

Israel

17

Twilix

Confident AI offers a platform to evaluate and benchmark LLM applications through advanced observability and synthetic dataset generation. It utilizes metrics proven to match human evaluation and offers real-time feedback on performance drift and regressions.

-10

freemium

Unknown

18

AgentRigor™

AIWORKX builds AI datasets, verifies AI models and data quality, and provides software testing and certification services. It also offers products including AgentRigor for evaluating AI agent reliability, DeviceProbe for test automation, blackolive for data processing, ADQ for data quality verification, and Omni AI for building conversational AI agents.

N/A

unknown

South Korea

19

Traceloop

Traceloop provides LLM observability and evaluation, turning logs into insights. It monitors prompts, responses, latency, and quality metrics like faithfulness and relevance. One line of code enables tracking, custom evaluators, and automated checks in CI/CD or real-time, supporting OpenTelemetry and 20+ providers.

-11

unknown

United States

20

Whitecircle

White Circle provides an AI control layer that lets companies test, monitor, and enforce policies on AI systems in real time, checking inputs, outputs, and tool calls against custom policies. It also offers guardrails, analytics, and red-teaming for production AI applications.

N/A

unknown

France

21

RagaAI

RagaAI Catalyst is a platform for evaluating and monitoring AI workflows, offering tools for observability, debugging, and performance enhancement, ensuring robust AI deployments.

-11

freemium

United States

22

Agent Insight

Agent Insight evaluates AI and human agents against custom quality standards, by role, across every conversation. It provides recommendations to improve AI agent performance and tracks metrics defined by the business. This enables continuous improvement of customer service and sales operations.

N/A

paid

United States

23

Agentops

AgentOps is a developer platform designed to test and debug AI agents, enhancing reliability through tools like time travel debugging and cost tracking. It integrates with leading cloud services, ensuring seamless deployment and management.

-12

freemium

United States

24

LLM Reference

LLM Reference is a reference and benchmarking platform that helps technology leaders and engineers evaluate AI language models, providers, and tools. It aggregates model benchmarks, pricing data, provider information, comparisons, and curated recommendations, enabling users to compare models for use cases such as coding, agents, research, writing, long context tasks, tool use, image generation, video generation, and other AI workloads.

-15

free

United States

25

ellamind GmbH

ellamind GmbH develops AI evaluation and agent software for regulated industries including health insurance, banking, and public administration, with its main product elluminate offering offline criteria-based scoring, experiment comparison, and EU AI Act compliance reporting. It also builds AI agents and offers an Accessibility Agent for BITV and BFSG accessibility testing.

N/A

unknown

Germany