Moshe Sambol, VP of Customer Solutions at Lightrun – brings more than two decades of experience spanning software engineering, architecture, cloud infrastructure, and customer-facing technical leadership. Before joining Lightrun in 2022, he spent nearly a decade at Google, where he held several leadership positions, including Cloud Customer Engineering Manager, helping organizations adopt and scale Google Cloud technologies. Earlier in his career, Sambol held engineering and development leadership roles at Oracle, Sun Microsystems, BMC Software, and JPMorgan Chase. At Lightrun, he initially led global Solution Engineering before becoming VP of Customer Solutions, where he focuses on helping customers adopt the company’s Runtime Insights technology and translate its capabilities into measurable business and developer productivity gains.
Lightrun is an AI-native engineering reliability platform designed to give developers and AI agents direct visibility into how software behaves while it is running. Its technology can dynamically capture logs, snapshots, metrics, traces, variable values, and execution context from live applications without requiring code changes or redeployments. The company is increasingly extending this runtime intelligence to AI-assisted software development through Lightrun MCP, which uses the Model Context Protocol to provide coding assistants and agentic tools with live application context rather than relying solely on static source code. This enables AI systems to investigate production issues, validate hypotheses against actual execution behavior, and support root-cause analysis while incorporating enterprise controls such as role-based access and sensitive-data redaction.
Your career has spanned hands-on software development and architecture, cloud customer engineering at Google, global solution engineering, and now customer solutions at Lightrun. How has this combination of building software and working directly with enterprise customers shaped your understanding of what separates an impressive AI agent demonstration from a system that can be trusted in production?
There is a big difference between showing what an AI agent can do and proving it can be trusted in an enterprise environment. This is because agents are only one part of a production-ready system. The framework around it is just as important. It must enforce least-privilege access, monitor activity, preserve an audit trail, prevent unacceptably risky actions, and bring in a human when necessary.
Agentic systems are inherently different from traditional software, because developers do not prescribe exactly how the system will work. We set a goal, provide tools and guidance, and the model determines how to proceed. That flexibility is powerful, but it also makes the system’s behavior harder to predict.
For enterprises, especially those in regulated industries, production workflows that usually work or take an unpredictable amount of time to complete are non-starters. Production environments contain sensitive data, source code, and intellectual property, so organizations need to be able to prevent agents from exposing that information, or taking creative but unacceptable routes to achieve their goals. This is becoming ever more important, as each week brings a new example of an AI system that, in its drive to reach a goal, ends up vulnerable to or causes a security exploit.
Most leaders I talk to still evaluate agents the way they would a new employee: on capabilities, judgment, and output. The real question isn’t whether the agent is smart enough. It’s whether the system around it can catch and contain the moments it isn’t.
Many enterprises initially believed that building an AI agent was largely a matter of writing an effective prompt. What did organizations misunderstand about the engineering, architecture, and operational requirements behind production-ready agents?
I think the biggest misunderstanding was an almost naive belief in the power of AI to solve any challenge, once given a well written prompt, relevant context, and suitable tools. Teams connected their LLM to code, documentation, tickets, and historical telemetry, and then expected it to reason its way accurately to the right decision.
What they did not build was a verification model for each step of the AI’s reasoning. One of AI’s great strengths is that it uses probabilistic reasoning, finding and taking one of many possible routes to reach a destination. In complex, interconnected production environments, that same strength introduces serious risk: a single decision can trigger downstream regressions, silent failures, or other unexpected behavior that threatens the operational resilience of a running system.
This is where deterministic steering becomes essential. The agent’s reasoning can stay probabilistic, but the checkpoints around its actions cannot. For an agent participating in an engineering workflow, this requires a verification step that checks its hypothesized next action against production reality, a deterministic gate rather than another probabilistic guess. It needs to see what the consequence of that decision will be, and approve it only once it has determined that the action is safe.
Looking across the first wave of internally developed enterprise agents, what are the most common architectural mistakes you are seeing, and which problems can be corrected incrementally rather than requiring a complete rebuild?
The core concern I keep coming back to is validation. Agents can become a black box: they collect information from a range of sources and then make decisions that look reasonable in principle but may not be appropriate for the realities of a messy, complex production environment.
That points to a more fundamental shift, and it is something we talk about constantly at Lightrun as we help customers build agentic automations for their engineering organizations. Teams need to rebuild the agentic flow itself and put gates on the agent’s actions, to ensure that its use of tools is subject to supervision, audit and review. Providing the agent itself with a strong feedback loop – including live runtime observability – focuses its context on what’s really happening right now. That access is what lets the agent validate its own design decisions, root cause analysis, and error mitigation recommendations against production reality instead of against assumptions based on static analysis of code or old telemetry.
A dramatic rebuild isn’t the only option. What can be done incrementally, and this is not groundbreaking but it is essential, is investing in the skills that guide the agent’s behavior. Carefully crafted and evaluated skills nudge the agent in the direction of a deterministic workflow. Teams do not need to rearchitect the whole system to get that benefit. They need to treat skill design with the same rigor they would give any other piece of production logic.
Why do some agents perform well during controlled testing but begin producing inconsistent, incomplete, or misleading results when exposed to real users, changing data, external tools, and complex production environments?
Controlled tests remove most of the variability that will define the production reality the AI has to contend with. The data is curated, tool behavior is predictable, permissions are known, and we cover a path we anticipated. When you release an agent to interact with real users and their effects in live systems, you are not comparing like with like.
Users introduce ambiguous requests and run concurrent actions, the system state is in constant flux, the agent often has to work from partial data, and external tools bring their own latency and failure modes on top of that. Because the model is probabilistic, each new variable creates another place where the workflow can diverge or compound an earlier mistake.
The dangerous part is that the agent can continue to appear to be working correctly while producing incorrect but plausible responses, built on partial data or on assumptions rooted in stale information. That is why production agents need continuous evaluation that keeps running after launch, explicit handling of missing data and tool failures, and live verification of a decision before it completes a high-impact action.
Lightrun places significant emphasis on giving AI systems access to runtime context. What information does runtime context provide that conventional logs, metrics, and traces may miss, and why is this information particularly important for diagnosing agent failures?
Conventional observability shows the external symptoms of system behavior, often aggregated, sampled, or filtered through dashboards and alerts that trigger on thresholds. They are usually dependent on decisions made by developers at the time the code was written: what information will be of interest in the future? What’s worth logging or measuring? Runtime context decouples visibility from this need to know in advance what might be of interest, and it provides granular data showing what’s going on under the hood, and how we got there.
The real gap is static versus dynamic data. Conventional logs, metrics, and traces are static, and produce a historic account of what happened. Lightrun’s runtime context is dynamic. It gives an agent the ability to place new instrumentation into running code, on demand, and observe the exact variable values, function arguments, object state, call stack, or branch conditions as they occur.
This distinction is particularly important for diagnosing failures in agent-generated code, because these are frequently silent. An agent can choose the wrong tool, pass the wrong argument, or act on a stale assumption, and still complete its task without triggering any error. A failure like that will not show up in static telemetry, because nobody knew in advance to instrument for it. Unexpected behavior requires a dynamic investigation directly on the running system, placing new instrumentation exactly where the agent’s model of the world diverged from reality, rather than relying on whatever was already being recorded.
That is what makes dynamic runtime context the natural verification layer for AI-generated decisions in engineering.
How can the Model Context Protocol (MCP) and similar integration layers allow coding agents to learn from real execution behavior without giving them excessive or unsafe access to production systems?
MCP and other controlled access to external tools (e.g. CLI wrappers) let an agent call a specific, scoped capability rather than being handed broad access to a system and trusted to behave. An agent connected through an MCP server for runtime context can request read-only evidence, a variable’s value, a call path, whether a threshold was exceeded, without touching write access, without the ability to redeploy anything, and without needing standing credentials to the underlying environment.
When redesigning a first-generation agent, how should enterprises approach tool permissions, memory, data retrieval, evaluation, human oversight, and fallback procedures as parts of one cohesive architecture rather than separate features?
You cannot bolt these pieces on independently because each one changes the others. The best places to start are the framework, the harness which controls the agent loop, and the overall workflow orchestration which ties together multiple agents and other actors. For a root-cause analysis workflow, for example, teams should decide what evidence is required, which systems the agent may inspect, whether it can publish a conclusion or only draft one, when a human must approve the next step, and what happens if runtime evidence is unavailable.
Once that contract is clear, the harness and framework provide the mechanisms with which to enforce those guidelines. MCP gateways can be leveraged to limit the agent’s access to specific capabilities relevant to its purpose. Tools can be granted with least privilege. Memory can be supervised, with sensitive data redacted deterministically. Retrieval can be designed around the evidence the workflow needs.
Evaluation, oversight, and fallback then close the loop. The system should measure whether conclusions are correct and supported, bring in a human when risk or uncertainty crosses a defined threshold, and stop or fall back to a read-only recommendation when it cannot gather enough evidence. A shared audit record should connect the trigger, permissions, evidence, tool calls, approvals, action, and outcome. That is what makes these components one production architecture rather than six separate features.
What safeguards should surround agents that can inspect live applications or participate in site reliability engineering workflows, particularly in regulated environments where access controls, privacy, auditability, and operational stability are critical?
This was one of the central design questions when we built Lightrun AI SRE. An AI SRE operates close to some of the most sensitive systems in an organization, so we designed it as a privileged operational actor, not a chat assistant. One important decision was to separate the inspection plane from the action plane. The AI SRE gathers evidence through read-only integrations and Lightrun’s sandboxed runtime instrumentation, with access restricted by identity, tenant, service, and environment. It can inspect live execution and generate missing evidence, but the runtime-inspection layer cannot modify application state.
In a regulated environment, that boundary has to be supported by RBAC, SSO, tenant isolation, PII redaction, retention controls, and an audit trail showing which tools and evidence supported each conclusion. We also need operational limits around how much data can be collected, how frequently the runtime can be queried, and which actions require approval. If evidence is missing or a conclusion cannot be verified, the AI SRE should say so and hand the decision to a human rather than act as if it knows more than it does. The goal is controlled autonomy: useful enough to accelerate an investigation, but constrained enough to remain safe for the live system.
As enterprises move beyond experimental agents, what measurements should determine whether an agent is genuinely production-ready, and how do you expect the relationship between AI agents and human engineers to evolve over the next several years?
I would judge production readiness by how often an AI agent’s actions produce the desired results, its conclusions hold up against what was actually true in production, unsupported conclusions are caught before action, and whether it fails visibly and safely when the evidence is not there. For engineering agents, verified outcome accuracy, evidence coverage, time to confirm root cause, successful fallback rate, and post-action results are the core metrics we should be focusing on.
Over the next several years, I expect agents to take on more of the evidence collection and first-pass investigation, as well as supervision of agentic workflows, and ongoing learning from experience and feedback, while engineers set policy, resolve ambiguity, approve high-risk actions, and steer the self-improving agentic systems. Trust will expand workflow by workflow. Agents that can trace their conclusions back to live evidence and clearly disclose what they could not verify will earn greater autonomy. Those that cannot will remain limited to narrow, low-stakes tasks, regardless of how fluent they sound.
Thank you for the great interview, readers who wish to learn more should visit Lightrun.
