Your AI Agents Are Running Queries You Can't See — The Database Observability Gap

Codenotary just launched AgentMon, Microsoft published an AI observability framework, and Oracle baked agent memory into its database engine. The industry is waking up to a hard truth: when AI agents autonomously query your databases, traditional monitoring is blind. Here's why your API layer is the missing observability plane.

Half of all enterprises are piloting agentic AI right now. Only 24% have it in production. And the number one reason pilots stall isn’t model quality, cost, or even hallucinations — it’s that nobody can see what the agents are actually doing.

This week, three things happened that make the database observability gap impossible to ignore:

  1. Codenotary launched AgentMon (March 31) — the first enterprise-grade monitoring platform built specifically for AI agent networks, tracking tool calls, data access patterns, and policy violations in real time.
  2. Microsoft published an AI observability framework (March 18) focused on proactive risk detection in AI systems, emphasizing that traditional APM tools miss the autonomous decision-making layer entirely.
  3. Oracle announced its Unified Memory Core and AI Database Private Agent Factory (March 24) — embedding persistent agent memory and pre-built data agents directly inside the database engine.

Three different companies, three different approaches, one shared conclusion: when AI agents access your data, you need to see every query, every tool call, every decision — or you’re flying blind.

The Problem: Agents Don’t Follow Scripts

Here’s the fundamental shift that breaks traditional monitoring. A human developer writes a query, ships it in a PR, and that query runs the same way every time. You can trace it, profile it, alert on it. Your Datadog dashboard knows exactly what’s happening.

An AI agent doesn’t work that way. Give an agent access to your database through MCP or a REST API, and it will:

  • Decide which tables to query based on the user’s natural language request
  • Choose how to filter, join, and aggregate based on its own reasoning
  • Chain multiple queries together, using the output of one to inform the next
  • Retry with different parameters if the first attempt doesn’t return useful results

Every interaction is different. There’s no static query to trace. There’s no predefined execution path to monitor. The agent is making autonomous decisions about your data access patterns in real time.

This is why traditional Application Performance Monitoring (APM) tools have a blind spot. They’re built to monitor known code paths. They can tell you that your /api/users endpoint is slow. They can’t tell you that an AI agent just ran a full table scan on your transactions table because it misunderstood a prompt about “recent activity” and decided to pull every row from the last five years.

What AgentMon Gets Right

Codenotary’s AgentMon represents the first serious attempt to solve this at the infrastructure level. Rather than monitoring individual API endpoints or database queries, AgentMon monitors the agent itself — tracking its behavior as a first-class entity.

The platform provides:

  • Communication path mapping between agents and services — which agents are talking to which databases, through which tools
  • Token usage and model selection tracking — what LLM is driving each agent’s decisions, and how much each interaction costs
  • File access and secrets handling monitoring — detecting when an agent accesses data it shouldn’t, or handles credentials in ways that violate policy
  • Data access pattern analysis — identifying anomalous query patterns that might indicate prompt injection, data exfiltration, or simple misconfiguration

That last point is the one that matters most for database access. When an agent is autonomously generating queries against your production data, you need something watching the pattern of access, not just the individual queries. Is this agent suddenly reading from tables it’s never touched before? Did its query volume spike 10x in the last hour? Is it accessing PII columns that its role shouldn’t permit?

BCG projects the AI agents market will grow at a 45% CAGR over the next five years. As adoption scales, the number of autonomous database interactions will grow exponentially. Without observability at the agent layer, every one of those interactions is a black box.

Microsoft’s Framework: Observability as a Security Requirement

Microsoft’s AI observability blog post takes a different angle — treating observability not as an operations concern but as a security imperative. Their framework centers on two capabilities that traditional monitoring doesn’t provide:

Evaluation: Measuring whether agent outputs are grounded in source material and whether tools are used correctly. For database access, this means validating that the data an agent returns actually matches the query it generated — catching hallucinated results before they propagate.

Governance: The ability to measure, verify, and enforce acceptable system behavior using observable evidence. This goes beyond access control. It’s about proving, after the fact, that an agent’s database interactions complied with policy. Audit logs for AI.

The governance angle is critical for regulated industries. If your AI agent pulls customer financial data to answer a support question, your compliance team needs to see exactly what was accessed, why, and whether the agent had authorization. A 200 OK from your API isn’t enough. You need the full chain: prompt → reasoning → tool selection → query generation → data access → response.

Oracle’s Bet: Put the Agent Inside the Database

Oracle’s approach is the most radical of the three. Instead of monitoring agents from the outside, Oracle is embedding agent capabilities directly into the database engine with its AI Database 26ai release.

The Unified Memory Core gives AI agents persistent, stateful memory inside the database — so agents can maintain context across sessions without external state management. The AI Database Private Agent Factory is a no-code platform for deploying data-centric agents as portable containers, with pre-built agents for database knowledge, structured data analysis, and deep data research.

Oracle’s thesis is straightforward: if the agent lives inside the database, you get observability for free. Every agent action is a database operation. Every database operation is logged, traced, and auditable through Oracle’s existing enterprise tooling.

It’s an elegant solution if you’re already on Oracle. But it also represents a form of vendor lock-in that should make any architect pause. Your agent observability shouldn’t depend on your database vendor. And for the majority of teams running PostgreSQL, MySQL, SQL Server, or multiple databases simultaneously, Oracle’s approach isn’t an option.

The API Layer: Your Natural Observability Plane

Here’s what all three approaches miss — or at least undervalue: the API layer between your agents and your databases is the single best place to implement observability.

Think about it. Every tool call an MCP-connected agent makes to access your database passes through an API. That API is the chokepoint where you can:

  1. Log every request with full context — which agent, which user, which tool, what parameters
  2. Enforce access control before the query hits the database — not after
  3. Rate limit aggressive agents that generate too many queries
  4. Validate query patterns against expected behavior
  5. Capture the full audit trail that compliance teams need

This isn’t theoretical. If you expose your database through a structured REST API with proper authentication and logging, you already have 80% of what AgentMon, Microsoft, and Oracle are trying to build — because every agent interaction is an HTTP request with a traceable identity, timestamp, and payload.

The problem is that most teams don’t have a structured API layer in front of their databases. They connect agents directly to databases through raw SQL tools, or they build one-off API endpoints that don’t have consistent logging, auth, or rate limiting. That’s how you end up with the observability gap.

Building Observable Agent-Database Access with Faucet

Faucet generates a complete REST API from your database schema in seconds, with built-in features that directly address the agent observability gap:

Structured, Predictable Endpoints

Every table gets a consistent set of CRUD endpoints with typed parameters:

# Install Faucet
brew install faucetdb/tap/faucet

# Serve APIs from your database
faucet db add --name myapp --driver postgres --dsn "postgres://user:pass@localhost:5432/myapp"
faucet serve

Now every agent interaction follows a predictable pattern:

GET  /api/v1/myapp/_table/customers?filter=age > 30 AND state = 'CA'   # Filtered read
GET  /api/v1/myapp/_table/orders?filter=customer_id = 123               # Related records
POST /api/v1/myapp/_table/support_tickets                               # Create operation

No ad-hoc SQL generation. No full table scans because an agent got creative. Every request maps to a documented endpoint with validated parameters.

Structured Request Logs

Every API call Faucet handles produces a structured log line with:

  • Timestamp and request ID
  • Method and path (which service and which table)
  • Status code (a 403 means an agent tried to step outside its role)
  • Duration, response size, and remote address

Because every agent gets its own API key and role, the API layer is where agent activity becomes legible. Feed these logs into your existing log aggregation (ELK, Datadog, Grafana) to start building a picture of what your agents are doing with your data. (Faucet does not ship a dedicated audit log; for compliance-grade trails, pair it with your gateway or database auditing.)

RBAC That Agents Can’t Bypass

Faucet’s role-based access control operates at the API layer, before any query reaches your database:

# Create a read-only role for AI agents: one view, GET only
faucet role create --name ai_reader
faucet role grant --role ai_reader --service myapp --component "_table/customer_profiles" --verbs GET
faucet key create --role ai_reader --label support-agent

# Sensitive fields stay out of the view the role is granted
# SSN, payment info, internal notes — invisible to the agent

This is fundamentally different from sharing a database connection. A connection string grants whatever that database user can do, to whoever holds it. API-level RBAC gives each agent its own key, bound to a role that names exactly which tables and which operations it can use, and denies everything else. It’s the per-agent granularity that agent observability requires.

MCP Server with Tool Annotations

Faucet’s built-in MCP server exposes your database tools with the safety annotations that agents and monitoring systems need:

# MCP is served at http://localhost:8080/mcp, authenticated with the agent's API key
faucet serve

Every one of the eight faucet_* tools carries an MCP tool annotation that tells the agent (and any monitoring layer like AgentMon) whether it reads or writes before it executes. faucet_query is marked readOnlyHint: true; faucet_delete is not, so under the spec’s defaults it is treated as destructive. A monitoring system can flag any agent that calls a write tool without explicit user approval.

OpenAPI 3.1 Spec for Agent Discovery

Faucet auto-generates an OpenAPI 3.1 specification for every database it serves:

curl http://localhost:8080/openapi.json

This spec becomes the contract that observability tools can validate against. If an agent tries to access an endpoint that isn’t in the spec, something is wrong. If an agent’s query patterns deviate from what the spec defines as valid parameters, that’s an anomaly worth investigating.

The Observability Stack for Agent-Database Access

Here’s what a production-ready observability stack looks like when AI agents are accessing your databases:

LayerWhat It MonitorsExample Tools
Agent LayerAgent behavior, reasoning chains, tool selectionAgentMon, LangSmith, Braintrust
API LayerRequest patterns, auth, rate limits, audit trailFaucet logs, API gateway metrics
Database LayerQuery performance, connection pools, lockspg_stat_statements, Datadog DBM
InfrastructureCPU, memory, network, container healthPrometheus, Grafana, CloudWatch

Most teams have layers 3 and 4 covered. Layer 1 is what Codenotary, Microsoft, and others are building now. But layer 2 — the API layer — is the one most teams are missing entirely, and it’s the one that ties everything together.

Without a structured API layer, your agent monitoring tool sees tool calls but can’t map them to specific database operations. Your database monitoring sees queries but can’t attribute them to specific agents. The API layer is the join key between agent identity and database activity.

The 40% Failure Rate

Here’s the stat that should concern every team building with AI agents: over 40% of enterprise AI agent projects fail. The top reasons cited are governance gaps, unclear ROI, and inadequate integration infrastructure.

Observability addresses all three:

  • Governance gaps close when every agent-database interaction is logged, attributed, and auditable
  • ROI becomes measurable when you can track exactly how many queries agents run, how much time they save, and what their error rate is
  • Integration infrastructure solidifies when agents access databases through a standardized API layer rather than bespoke connections

The companies that will successfully scale AI agents in production — the ones that join Pinterest in that 24% club — are the ones investing in observability now, before they need it.

What Comes Next

The convergence of AgentMon, Microsoft’s observability framework, and Oracle’s database-embedded agents signals a market that’s maturing fast. Within the next 12 months, expect:

  • Agent observability to become a procurement requirement — CISOs will demand it before approving production agent deployments
  • MCP tool annotations to expand — the current readOnlyHint/destructiveHint annotations are just the beginning; expect cost hints, latency hints, and data classification hints
  • API layers to become the standard for agent-database access — raw database connections will be treated like SSH access: technically possible, operationally unacceptable

The teams that build observable, auditable, access-controlled database APIs today won’t have to retrofit them tomorrow.

Getting Started

Deploy a fully observable database API in under a minute:

# Install Faucet
brew install faucetdb/tap/faucet

# Serve your database with request logging and MCP support
faucet db add --name mydb --driver postgres --dsn "postgres://user:pass@localhost:5432/mydb"
faucet serve

# Your agents now have:
# - REST API with request logging at :8080/api/v1/mydb/_table/{table}
# - MCP server with tool annotations at :8080/mcp
# - OpenAPI 3.1 spec at :8080/openapi.json
# - Roles and API keys managed with faucet role / faucet key or the admin UI at :8080/

Every query your AI agents run against your database is now a logged, authenticated, role-scoped API call. That’s not just observability — that’s the foundation for running agents in production.