Rivi work/purgo.ai × Rivi

building a Databricks Reflection Agent: 55% → 90% execution success.

Rivi built a self-correcting agent designed to make generated Databricks code significantly more reliable while keeping execution safely read-only.

agentic aiDatabricksreflectionevaluation
execution success · benchmark suite
generation only55%
with the Reflection Agent90%
~40 examples · 3 runs each

generating code isn’t enough if it doesn’t run.

generated code was failing unpredictably in Databricks, with limited visibility into why changes worked or regressed.

a failure returned an exception and nothing else - no diagnosis, no correction, no second attempt.

before · generation only
runspark.sql(generated_query)
errorAnalysisException: column cannot be resolved
resultfailed · no diagnosis · no retry
illustrative - representative of the failure mode, not a captured customer run

execute. observe. fix. retry.

the Reflection Agent runs candidate code, understands execution errors and makes focused corrections to the failing part before safely trying again.

the goal is closer to careful debugging than rewriting everything from scratch.

executeobserve the errorfocused fixsafe retry
loops until the code executes, or the attempt budget ends
after · reflection loop
attempt 1execute
errorAnalysisException: column cannot be resolved
reflectschema mismatch located · one clause at fault
patchfocused edit · rest of the query untouched
attempt 2execute
okexecuted · result returned
illustrative - the shape of a real recovery, with customer data removed

read-only by construction.

a self-correcting agent that retries code needs a hard boundary around what it is allowed to run. execution is constrained before a statement ever reaches the warehouse.

✓allowed
SELECTSHOWDESCRIBEEXPLAIN
reads and inspection - everything the agent needs to diagnose its own failure.
✕blocked
DROPDELETEUPDATEINSERTTRUNCATEALTER
destructive statements are refused before execution, on every attempt and every retry.

measurement first.

Rivi also built the evaluation foundation around the agent - repeatable benchmarking and end-to-end traces so every iteration could be measured rather than judged anecdotally.

benchmark suite
a fixed example set, re-run three times per change so a result can be trusted as a trend rather than a lucky pass.
end-to-end traces
every attempt, error, reflection and patch recorded - so a regression can be read back to the step that caused it.
per-attempt scoring
success measured per attempt, not per run, which is what made the recovery rate visible at all.

55% → 90%.

55%→
90%
execution success
78%
recovery of failed attempts
100%
destructive operations blocked
~40 examples · 3 runs each.

agentic ai that does real engineering work.

talk to us