1

Honeycomb

Honeycomb provides an observability platform for engineering teams to follow their code into production, covering distributed tracing, log analytics, time series metrics, frontend observability, and telemetry pipelines. It is built on an event-based, high-cardinality data model with OpenTelemetry-native ingestion, and includes features such as BubbleUp, SLOs, Service Map, and anomaly detection. The platform also supports observability for AI agents and LLM applications, including tracing multi-agent conversations, LLM cost tracking, and connecting coding agents to telemetry through the Honeycomb MCP server.

N/A

freemium

United States

2

Fiddler AI

Fiddler AI provides a platform for monitoring and ensuring the integrity of AI models, offering explainable AI and observability tools to manage the lifecycle of ML and LLM applications effectively. It integrates with major cloud providers.

-1

freemium

United States

3

Promptfoo

Promptfoo offers a platform for robust LLM application security testing, enabling developers to run tailored vulnerability scans and receive actionable insights for software protection.

-1

freemium

United States

4

Llm-Stats

LLM Stats operates an independent AI benchmarking and leaderboard platform that ranks 300+ language models by intelligence, speed, and price using a composite LLM Stats Score derived from public benchmarks and live API metrics. It offers leaderboards, model comparison, pricing data, arenas, and a data API/MCP with benchmark scores, rankings, and metadata for developers, researchers, and enterprise teams. The site also provides custom evaluation services for frontier models, including coding, long-context reasoning, and finance benchmarks.

N/A

freemium

United States

5

Deepchecks Monitoring

Deepchecks offers comprehensive evaluation and monitoring tools for LLM applications, leveraging open-source ML testing packages to automate quality control, compliance checks, and risk mitigation, enhancing the reliability and performance of AI systems.

-2

freemium

Israel

6

Inference.net

Inference.net (Inference R&D, Inc.) provides a platform for deploying, monitoring, tracing, evaluating, and fine-tuning open-source and custom AI models, including a marketplace-style spot market for underutilized data center GPU capacity. It targets AI-native engineering teams that want to run inference on models such as Kimi, GLM, MiniMax, and GPT-OSS, or train task-specific models on their own production traces. Key capabilities include an OpenAI-compatible gateway, LLM observability, continuous evaluation, and fine-tuning workflows, with SOC 2 Type II certification.

N/A

freemium

United States

7

TruEra

TruEra provides AI-driven solutions for machine learning model monitoring, testing, and quality management, ensuring reliable and trustworthy AI implementation across industries such as banking and manufacturing.

-3

unknown

United States

8

Pydantic

Pydantic provides an end-to-end AI engineering stack that includes Pydantic AI for building agents, Pydantic Logfire for observability and tracing, Pydantic Evals, and an AI Gateway. It also maintains the Pydantic data validation library for Python, which validates data using type hints and has a Rust-based core. The products target Python and TypeScript developers and engineering teams building and operating AI applications in production.

N/A

freemium

United States

9

Scade.pro

Scade.pro offers a no-code platform that simplifies AI app development by providing access to over 1,500 models and tools. It reduces complexity with a unified API, allowing easy integration across web and mobile platforms without coding skills.

-4

freemium

Unknown

10

Guardrails

Guardrails AI provides an AI reliability platform and open-source Python framework for building, governing, and scaling production generative AI applications across LLMs and deployment environments. Its Guardrails framework runs input/output guards that detect, quantify, and mitigate risks, and helps generate structured data from LLMs, while Guardrails Hub offers pre-built validators. The company also offers Snowglobe for simulating synthetic datasets and AI chat scenarios to test and evaluate AI systems before deployment.

N/A

unknown

United States

11

Respan

Respan (formerly Keywords AI) provides an LLM engineering platform that combines a unified API gateway to 1,000+ models with observability, automated evaluations, and prompt optimization for teams building AI agents. Users can trace every LLM call, tool run, and agent turn, score outputs with LLM judges, code checks, or human review, and monitor cost, latency, errors, and tokens with alerts. It is aimed at startups and enterprise teams running AI agents in production, with SOC 2, ISO 27001, GDPR, and HIPAA compliance options.

N/A

freemium

United States

12

LMSpeed

LMSpeed is a web platform for comparing and testing LLM API providers and models. It aggregates per-token pricing from 100+ providers, runs five-round speed benchmarks measuring first-token latency and output throughput, and audits OpenAI-compatible endpoints for model identity, prompt injection, and error leakage. It targets AI developers and teams choosing an API endpoint, and also offers model benchmark lookups, free API key listings, and provider leaderboards.

N/A

freemium

Unknown

13

MLflow

MLflow is an open-source platform for managing the machine learning and generative AI lifecycle, including experiment tracking, model evaluation, model registry, and deployment. It also provides observability, evaluation, prompt management, and an AI Gateway for LLM applications and agents. It is used by data scientists, ML engineers, and AI developers, and is governed by the Linux Foundation under an Apache 2.0 license.

N/A

free

United States

14

Deepeval

DeepEval is an open-source LLM evaluation framework, built and maintained by Confident AI, for testing and benchmarking LLM applications. It provides 50+ research-backed metrics (including LLM-as-a-judge, RAG, agent, multi-turn, voice, safety, and multimodal metrics) and runs Pytest-style assertions locally or in CI/CD pipelines. It targets AI engineers, data scientists, QAs, and PMs evaluating agents, chatbots, RAG pipelines, and MCP systems, and integrates with frameworks and model providers such as LangChain, OpenAI, Anthropic, and Gemini.

N/A

unknown

Unknown

15

Mabyduck

Mabyduck is a self-serve platform for running subjective quality evaluation studies on AI-generated audio, video, images, and interactive experiences. It is aimed at AI researchers, academics, startups, and enterprises, offering configurable pairwise comparisons, surveys, rubrics, leaderboards, rater pools, and an API for integrating evaluations into existing pipelines. Key capabilities include RLHF data collection, regression tests, embedded experiments, and top-K selection strategies for pairwise experiments.

N/A

freemium

United Kingdom

16

Superwise

SUPERWISE offers AI governance and monitoring solutions, ensuring compliance and operational efficiency for enterprises across various industries.

-10

unknown

Israel

17

Twilix

Confident AI offers a platform to evaluate and benchmark LLM applications through advanced observability and synthetic dataset generation. It utilizes metrics proven to match human evaluation and offers real-time feedback on performance drift and regressions.

-10

freemium

Unknown

18

AgentRigor™

AIWORKX (formerly Testworks) is a Korean AI data and software testing company that builds AI datasets, verifies AI models and data quality, and provides software testing and certification services. It offers products including AgentRigor for evaluating AI agent reliability, DeviceProbe for test automation and remote board farm management, blackolive for data processing and management, ADQ for data quality verification, and Omni AI, an LLMOps ecosystem for building conversational AI agents. It serves enterprise, manufacturing, public-sector, and finance customers, and operates as a social enterprise employing people from underrepresented groups in AI data work.

N/A

unknown

South Korea

19

Traceloop

Traceloop provides LLM observability and evaluation, turning logs into insights. It monitors prompts, responses, latency, and quality metrics like faithfulness and relevance. One line of code enables tracking, custom evaluators, and automated checks in CI/CD or real-time, supporting OpenTelemetry and 20+ providers.

-11

unknown

United States

20

Whitecircle

White Circle provides an AI control layer that lets companies test, monitor, and enforce policies on AI systems in real time. Its platform checks inputs, outputs, and tool calls against custom policies, with guardrails, analytics, and red-teaming for production AI applications. It is aimed at AI teams and enterprises deploying generative AI and agents.

N/A

unknown

France

21

RagaAI

RagaAI Catalyst is a platform for evaluating and monitoring AI workflows, offering tools for observability, debugging, and performance enhancement, ensuring robust AI deployments.

-11

freemium

United States

22

Agent Insight

Agent Insight evaluates AI and human agents against custom quality standards, by role, across every conversation. It provides recommendations to improve AI agent performance and tracks metrics defined by the business. This enables continuous improvement of customer service and sales operations.

N/A

paid

United States

23

Agentops

AgentOps is a developer platform designed to test and debug AI agents, enhancing reliability through tools like time travel debugging and cost tracking. It integrates with leading cloud services, ensuring seamless deployment and management.

-12

freemium

United States

24

LLM Reference

LLM Reference is a reference and benchmarking platform that helps technology leaders and engineers evaluate AI language models, providers, and tools. It aggregates model benchmarks, pricing data, provider information, comparisons, and curated recommendations, enabling users to compare models for use cases such as coding, agents, research, writing, long context tasks, tool use, image generation, video generation, and other AI workloads.

-15

free

United States

25

ellamind GmbH

ellamind GmbH develops AI evaluation and agent software for regulated industries including health insurance, banking, and public administration. Its main product, elluminate, is an agentic evaluation platform offering offline criteria-based scoring, experiment comparison, version tracking, cost analytics, and quality gates before production, with EU AI Act compliance reporting. The company also builds AI agents and offers an Accessibility Agent for BITV and BFSG accessibility testing, with on-premise or private cloud deployment for data sovereignty.

N/A

unknown

Germany