Skip to main content

How an AI Agent Cut Production Change Reviews From 24 Hours to 12 Seconds

Kamalnath Devarakonda
Nov 20 - 6 min read
How an AI Agent Cut Production Change Reviews From 24 Hours to 12 Seconds featured image

In our Engineering Energizers Q&A series, we highlight the engineering minds driving innovation across Salesforce. Today, we spotlight Kamal Devarakonda, Senior Director of Software Engineering at Salesforce, whose team faced a dangerous question: what happens when an AI approves a production change because it assumes QA and an architect signed off, but never verifies either approval? Making that agent trustworthy meant proving every required signoff while cutting day-long reviews to seconds across 18 Salesforce products.

Explore how the team reconstructed release evidence scattered across incompatible systems and engineered deterministic approval gates that cut median review turnaround from approximately 24 hours to 12 seconds.

What is your team’s mission, and why could not production change reviews be automated with a simple checklist?

The team had to interpret inconsistent, unstructured approval evidence while making every release gate independently verifiable. A strict checklist would reject valid changes; a loose one would let risky changes through. With signoffs buried in nested comments and teams following different conventions, the mission was to eliminate manual coordination without losing the engineering judgment behind production reviews.

The team reviewed 15 to 20 production patches each week across 18 Salesforce products, including Revenue Cloud. Moving between Slack, change records, QA evidence, and release calendars made the team a bottleneck. Requesters chased updates, reviewers asked for missing links, and cloud leaders lacked visibility into blocked changes. To remove that burden, the team mapped the complete Change Advisory Board (CAB) process and committed its flow diagrams to GitHub.

The team then built ROAM, the Release Operations Agentic Manager, an Agentforce-pattern agent applied to release operations, to gather and interpret context while verified records determined whether each gate was satisfied. ROAM automated the CAB review; at the time of the interview, a human still performed the final approval click.

What made Slack and release-system integration dangerous when a schema mismatch could silently erase an approval?

An integration could return successfully while losing evidence required for a correct review. Early in production, a client read the wrong field name in nested replies, silently dropping a subset of approvals. The model received an incomplete account of what had happened.

That failure exposed the difficulty of combining three different schemas. Slack supplies unstructured conversations, GUS, the internal system of record, supplies change fields and nested activity feeds, and the release calendar supplies deployment windows and cloud-owner information. Those inputs needed a consistent representation before ROAM could evaluate them. Salesforce’s internal Matrix Agent Platform provided event ingestion, routing, durable workflow state on Postgres, and shared clients. Building on that foundation, the team added domain adapters and a normalization layer to assemble a consistent change-context object, reconstruct approval histories, and return structured results in the originating Slack thread.

To protect the pipeline against schema drift, the team introduced API contract testing with recorded live payloads checked on every pull request. A mismatch fails the build. Agentforce reliability, and AI agent reliability in general, depends on integration correctness: a better prompt cannot recover an approval that never reached the model.

ROAM architecture automates approvals with secure, auditable AI-driven workflows.

How did deterministic approval gates prevent LLM hallucinations from controlling production change reviews?

The dangerous behavior was treating an unverified signoff as fact. The team needed a boundary that prevented generated language from determining or overturning ROAM’s review verdict. Prompt refinement alone could not provide that control. The principle, which the team now applies to every Agentforce-style agent it builds, became: the gate decides, and the LLM narrates. Deterministic code reads the change fields, linked test run, pull request, and approval trail. The model explains the results but cannot produce or overturn the verdict. When the two disagree, the model loses.

The team applies the same grounding discipline to determining why a patch is being released. Structured GUS fields and linked incidents or bugs identify security fixes, customer-reported issues, and internal changes. Freeform justification serves as a fallback; conflicts with structured evidence go to a human reviewer. After generation, output validation re-reads the change record and fails closed if the explanation disagrees with the evidence.

The team also traced repeated sentences to concatenated framework outputs and corrected them at the output boundary. Those findings reinforced the focus on integration correctness and validation rather than fine-tuning or heavier prompt adjustments.

Why did fail-closed approval verification sometimes require ROAM to hand the review to a human?

Finding “approved” was insufficient. The team had to verify the approver’s identity and role, along with evidence for the target release. Missing links, expired tokens, or outdated rosters could make a required gate impossible to confirm.

ROAM, following the same guardrail discipline the team applies to every Agentforce agent, verifies each requirement independently. QA qualification requires a passing linked test run on the target release branch. Dual-developer review requires two approving developers with repository commit rights, excluding the pull-request author. Architect signoff must come from the architect referenced by the change record, while cloud-leader approval must come from a current member of the versioned leader roster. When a gate cannot be verified, ROAM routes the request to a human. Approximately 8% of requests receive no automated decision.

To prevent retries from introducing new errors, thread-ownership guards and idempotency keys block duplicate Slack posts, while change-record identifiers and decision hashes prevent decisions from changing on unchanged state. Every outcome enters a durable audit table so the review can be reconstructed later.

What made AI-assisted development risky when generated code and passing tests could both create false confidence?

AI accelerated implementation but shifted work into verification. Generated pull requests could contain hundreds of lines, and test runners could silently skip suites while reporting green. Neither generated code nor an agent’s claim that its fix passed constituted proof of correctness. The team therefore reviewed every AI-assisted change line by line, challenging implementations and rewriting them when necessary. Approximately 70% of ROAM’s codebase was drafted or refined with Agentforce-assisted coding workflows. Engineers authored the remaining 30%, including core orchestration, model-classifier plumbing, and activity-feed parsers.

Code review addressed implementation quality, but test results needed independent verification too. A verifier agent re-executes affected suites and checks exit codes before trusting a fix agent’s self-report. Unit and functional test coverage reached the nineties against an 80% target. These checks helped ROAM reach production in approximately four weeks. AI accelerated development while engineers retained responsibility for both the implementation and the evidence that it worked.

What production results made automated CAB reviews trustworthy enough to become mandatory?

Faster responses alone could not earn the release team’s trust. ROAM had to demonstrate that it could verify complete requests and decline ambiguous ones. The meaningful production test was whether that behavior was dependable enough for the team to require it.

Across roughly 550 CAB reviews over 10 weeks, the team recorded zero incorrect approvals in ROAM’s review decisions. Median turnaround from Slack post to reply was approximately 12 seconds, with the 95th percentile below 45 seconds. Upstream API round trips accounted for most of the latency. Release reporting improved alongside individual reviews. A manual database query for pending CABs and blockers became a slash command returning in under a second. Engineers can also request cloud-specific CAB summaries in Slack, for Revenue Cloud or Health Cloud, for example.

After observing three weeks of production behavior, release engineering made ROAM, the team’s first Agentforce-aligned CAB reviewer, mandatory for every production patch across the 18 supported clouds. The team trusted its speed because it also trusted its limits: the team optimized for decision accuracy over automation percentage, handing the review to a human whenever the evidence could not support a verdict.

Learn More

Related Articles

View all