Enterprise AI platform · 2025 –
Eval Engine, CI Quality Gates & Database Reliability
Senior AI Engineer, Platform
Challenge
Agent changes were hard to ship confidently: no consistent trace-backed evaluation, slow production-readiness cycles, and shared Postgres saturation hitting identity and agent services. This was platform-wide pain, not one bad query.
Approach
- Shipped an evaluation engine, SDK decorators, and CI quality gates so teams catch regressions before promote.
- Led database connection-pooling rollout (sidecar proxies, connection budgets, ADRs) to cap saturation on shared Postgres used by identity and agent services.
Tech stack
Evaluation engine + SDK · CI integration · Postgres · Connection-pooling sidecars · Prisma · ADRs · GitOps-promoted config
Outcomes
- ~50% faster production-readiness cycle with eval gates in the release path
- Reduced shared-Postgres saturation impacting core platform services
- Documented pooling decisions for ongoing SLO-driven operations
Verifiable patterns (redacted)
Eval gate flow (redacted)
CI pipeline runs trace-backed eval suite on every agent PR. Promote job blocks if weighted score drops below threshold on any critical workflow path. Score breakdown (correctness, latency, safety) attached as build artifact for reviewer sign-off.
Connection pool ADR excerpt
Sidecar proxy pattern for shared Postgres: each service gets a dedicated PgBouncer sidecar with connection budget caps. ADR documents saturation root cause, pool sizing math, and rollback path. Identity and agent services no longer compete for raw connection slots.
Eval scorecard template (redacted)
Per-workflow eval matrix: tool-call accuracy, retrieval precision@k, multi-step completion rate, and regression delta vs. baseline. SDK decorators auto-capture traces; CI compares against last-green baseline and flags drift with failure taxonomy tags.
Context
Shared platform data plane that benefits all enterprise tenant workloads.