
Free Daily Podcast Summary
by VentureBeat
AI gets real here. On “Beyond the Pilot,” top business execs share what actually happens after the AI proof of concept — from infrastructure and org design to wins, failures, and ROI. Not theory, but deep dives into how they scaled AI that works.
The most recent episodes — sign up to get AI-powered summaries of each one.
Instacart's CTO stopped his engineers from reading code. 97% of their codebase is now AI-generated — and they've cut the cost of serving Clementine, their AI shopping agent, to one-eighth of what it was eight months ago. Anirban Kundu (AK), CTO of Instacart, breaks down the internal architecture behind that number. The core insight: the harness matters more than the model. Instacart built a proprietary system called Maverick that routes planning tasks to frontier models and code generation to faster, cheaper open-source models like Luna and DeepSeek — cutting development AI costs by 75% while hitting their highest-ever weekly code merge count. Engineers are now migrating a 13-year-old Ruby codebase to Go and Rust without knowing either language. On the product side, AK unpacks Clementine's technical stack: a multi-layer query understanding system that replaced a frontier model with a fine-tuned SLM, semantic IDs built from full catalog embeddings to surface substitute products in real time, and an intent-based architecture that — by the third session — shifts users from ingredient-level searches to natural-language cart building. The result: measurable basket size growth in a grocery e-commerce category he calls "massively under-penetrated." AK also covers Instacart's MCP strategy for navigating the personal agent ecosystem (Meta Muse, Claude Duet, OpenAI Dots), why they've actively steered partners away from browser-based integrations, and what comes next: synthetic users, per-user experimentation at scale, and fully segregated enterprise code instances generated on demand. 🎙️ GUEST: Anirban Kundu (AK) | CTO, Instacart 🎙️ HOSTS: Sam Witteveen | VentureBeat --- 00:00 Intro — Why the harness beats the model 00:40 Episode intro & guest background 01:20 97% of Instacart's code is no longer read by humans 02:20 The journey to 100%: legacy pruning and engineer buy-in 03:00 Migrating Ruby to Go and Rust — without knowing the language 03:40 From code generation to full codebase regeneration 04:00 Prompt-and-loop architecture: the "why" behind the code 05:00 Multi-model planning: frontier for strategy, open source for execution 06:20 Building the eval and verification layer 07:00 Instacart's internal harness: "Maverick" explained 08:00 Why the harness is more valuable IP than the model itself 09:20 59% of companies struggle to swap models — Instacart's approach 10:00 75% cost reduction and record weekly merge count 10:40 Luna and DeepSeek vs. frontier: speed and cost trade-offs 11:40 AI acceleration on the ML side, not just code generation 12:00 AutoML with Bayesian optimization: shipping models overnight 12:40 Combining GBM/neural nets with SLMs for long-tail model coverage 13:40 Semantic IDs: embedding-based product catalog relationships 14:20 Clementine: intent-based cart building explained 15:40 How Clementine uses purchase history and real-time fulfillment data 16:40 Multi-layer semantic search: inverted index + LLM-cached results 17:20 How the search architecture was rebuilt over the past year 19:20 Business results: search conversion growth and basket size lift 20:20 How users shift behavior by their third Clementine session 21:40 Navigating the personal agent ecosystem: Muse, Claude Duet, OpenAI Dots 22:20 MCP over browser-based integrations: why and how 24:40 Clementine's frenemy strategy: connectors as acquisition funnels 25:20 Personalization moat: purchase history as a competitive advantage 26:00 Blending organic and ad results inside Clementine 27:20 Instacart's enterprise offering: Storefront Pro and Cart Assistant 28:40 Per-brand customization: persona control and product highlighting 30:20 Code regeneration enabling segregated enterprise instances 31:40 Open vs. proprietary model routing inside Clementine 32:40 Fine-tuning SLMs for search long-tail and Clementine query understanding 33:20 How Instacart reduced reliance on frontier model providers 34:00 Automatic model escalation inside the developer harness 35:20 What's next: synthetic users, hyper-personalization, and beyond AB testing 37:00 The verification problem: managing non-deterministic output at scale 39:00 Closing thoughts and what AK will return to discuss --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #LLMInfrastructure #AIDeployment #MLOps Learn more about your ad choices. Visit megaphone.fm/adchoices
AI agents can now be weaponized to run elite-level cyberattacks at scale — and your perimeter won't stop them. Box CTO Ben Kus lays out exactly how enterprises defend against agentic threats without locking down the tools their teams actually need. Ben Kus spent years at IBM's BigFix patching vulnerabilities at enterprise scale. Now at Box — where 95% of the codebase is permissions, security, and compliance — he's dealing with a new threat model: cheap, scalable AI agents that give any attacker access to elite hacking capabilities previously limited to nation-state actors. Only 14% of companies in VentureBeat's September survey said they could shut down a misbehaving AI agent within minutes. Ben explains why that gap is dangerous and what the architecture of a real defense looks like. The conversation gets technical fast. Ben walks through the "lethal trifecta" (untrusted input + sensitive data access + outside communication = critical risk), how Box uses action guardrails to sandbox agents even if they're compromised, why 44% of enterprise agents running on shared human credentials is a ticking clock, and how newer Anthropic models have dropped prompt injection success rates from 60% to roughly 2%. The second half shifts to retrieval. Ben breaks down why standard RAG fails at enterprise scale — permissions gaps, vector space degradation, poor lexical matching — and what Box's context retrieval system (hybrid embeddings + reciprocal rank fusion + agentic search) actually delivers: 90% token cost reduction and 4–5x speed improvement over naive model-provider retrieval. He also addresses the harness question directly: 59% of companies that swapped models in production saw performance drop, and the reason is usually the harness, not the model. 🎙️ GUEST: Ben Kus | CTO, Box 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat --- **CHAPTERS** 00:00 Intro — AI agents as elite-scale attack tools 02:00 Episode overview: Box CTO Ben Kus on agentic security and retrieval 03:00 What's actually changed in enterprise security: speed, severity, AI-assisted vulnerability discovery 04:20 How AI finds multi-domain vulnerabilities humans miss (PHP, font exploits) 05:00 When agents become attackers: democratizing elite hacking 06:20 Why perimeter defense is dead and what layered security actually looks like 08:00 VentureBeat survey: only 14% of companies can kill a misbehaving agent in minutes 09:20 Agent identity and credential sharing: when it's acceptable and when it's not 10:40 Misaligned agents: the agent-with-your-identity threat model 12:20 The lethal trifecta: untrusted input + sensitive data + outside communication 13:00 How Anthropic models dropped prompt injection rates from 60% to 2% 13:20 Box's approach: action guardrails, sandboxing, MCP server restrictions 14:00 Prompt injection via documents: how Box cleanses untrusted content 15:20 Action Guardrails feature: containing damage when agents are compromised 16:20 The SaaSpocalypse question: will Box become just a connector? 17:00 Why systems of record survive the agentic shift (headless platforms, MCP) 18:40 Agent-to-agent delegation: when Claude routes work to the Box agent 21:20 Why enterprise search has always been broken — and what's changing 22:40 Vector embeddings, semantic search, and the limits of pure RAG 24:00 Reciprocal rank fusion: combining lexical and vector search at scale 25:00 Box's "librarian" model: returning pages, not books 26:20 Context retrieval in Box Works: 90% token reduction, 4–5x speed improvement 29:40 Why throwing away shipped work every 3–6 months is now a core skill 31:00 59% of companies saw performance drop when swapping models — why the harness matters more than the model 33:00 Custom vs. generic harnesses: how the answer has changed three times in one year 33:40 Closing thoughts --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #CyberSecurity #RAG #AIInfrastructure Learn more about your ad choices. Visit megaphone.fm/adchoices
Home Depot cut delivery failure rates from 5.3% to 2.8% in Chicago using a purpose-built agentic system — and it's protecting 1.5 million at-risk deliveries a year across weather, geospatial, and customer preference signals. Home Depot CIO Angie Brown breaks down exactly how the sixth-largest U.S. e-commerce operation ($25B+ in online sales, 2,300+ stores) built its production AI stack from the ground up. The architecture is a layered agent hierarchy: a thin client planner routes to domain agents (selling, post-purchase, delivery), which hand off to fine-grained sub-agents by category — paint, power tools, configurables. Underneath sits a knowledge foundation built on Google BigQuery, hybrid search, unstructured vector databases, and a data mesh, all wrapped with agent identity management, security guardrails, and safety evals. On the model side, Home Depot runs Google ADK as its primary orchestration layer with Azure OpenAI held as an active counterweight to evaluate output quality — and Claude for code generation with OpenAI models used for code review. The Magic Forge internal platform handles their most compute-intensive workloads. Multi-model tension is a deliberate architectural choice, not an accident. Two production results stand out beyond delivery: a conversational IVR (Google NGA) now live in ~1,000 stores is handling 3x more customer questions without human transfer, and an AI-powered blueprint takeoff workflow for contractors turned quote turnaround from one-plus weeks down to 24–48 hours. Angie also covers where the architecture still has rough edges — deterministic escalation rules being layered back onto the conversational IVR — and what's next: per-SKU assortment optimization and agentic planogram generation across 30–40K SKUs per store. 🎙️ GUEST: Angie Brown | CIO, The Home Depot 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat --- 00:00 Intro & delivery failure rate results 00:57 Home Depot's technology footprint: 2,300 stores + e-commerce scale 03:20 Magic Apron walkthrough: AI-guided product discovery online 06:00 Fulfillment AI: how estimated delivery dates are computed in real time 08:20 Order Intelligence: weather, geospatial, and customer preference signals 10:40 Agent architecture deep dive: client planner, domain agents, sub-agents 13:20 Meta-agent question: how context inference replaces a top-level orchestrator 15:00 Full AI stack: Google ADK, MCP, LLM eval, prompt management, vector DBs 16:20 Multi-model strategy: Google vs. Azure OpenAI vs. Claude — how they decide 19:00 Magic Apron consumer rollout: what worked and what didn't 19:20 B2B pivot: List Builder AI and blueprint takeoff for contractors (24–48hr quotes) 22:20 How the agent framework has evolved: less summarization, more model intelligence 24:20 IVR in 1,000 stores: conversational AI replacing tree-based phone menus 28:00 Associate augmentation: Magic Apron in-store mode and plumbing aisle story 30:20 Customer adoption of AI IVR: what the call recordings actually reveal 32:20 Deterministic escalation controls layered onto conversational AI 32:40 LLM-referred traffic and the future of agentic commerce 37:00 What's next: agentic assortment optimization and planogram generation at scale 39:00 Outro --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #RetailAI #LLMInfrastructure #AgenticAI Learn more about your ad choices. Visit megaphone.fm/adchoices
A $600 day of AI coding spend sounds alarming — until you divide it by the PRs shipped. At ~$30/PR, it reframes the entire ROI conversation: the spend isn't the problem, spend with no return is. This episode comes from a live panel Sam moderated at VentureBeat Transform, featuring Emilie Schario (Kilo Code, acquired by Anaconda the morning of the panel), Amol Jain (Replit), and Jared Go (Symbiotic). The conversation cuts straight to what agentic coding actually looks like in production — not in demos. On architecture: Kilo routes across 500+ models and processes 10 trillion tokens/month, using expensive frontier models for planning and cheaper open-weight models for execution. Replit built an internal fleet of agents running in isolated cloud VMs — connected to GCP, Datadog, and Slack — that autonomously handled a gnarly multi-day bug and produced a merged PR with zero human intervention. Symbiotic runs tiered monthly token budgets ($150 / $450 / exception) managed directly by engineering managers, not finance. On org design: code review — not code generation — is now the bottleneck. Replit risk-scores every PR; low-risk ones auto-merge. The panel is aligned that "what does code review mean in 2025" is the defining question for the next 6–12 months. On the 6-month outlook: open-weight models take majority share of coding tasks, frontier models reserved for architecture decisions; 24–48 hour unattended agent work loops become standard; and model sovereignty (geographic data routing, on-prem isolated environments) becomes a hard enterprise requirement. --- 🎙️ GUESTS: Emilie Schario | Head of Engineering, Kilo Code (acquired by Anaconda) | Amol Jain | VP Engineering, Replit | Jared Go | Co-founder, Symbiotic 🎙️ HOST: Sam Witteveen | VentureBeat --- 00:00 Intro & Cost Per PR framing 00:40 Panel intro: Kilo Code, Symbiotic, Replit 02:20 Do engineers still read code? (Instacart CTO's 97% stat) 03:00 How Replit uses risk-scored PRs to auto-merge 04:20 Physical AI and safety gates for code review (Symbiotic) 05:00 Kilo Code: 500+ models, 10T tokens/month, Anaconda acquisition 07:20 Multi-model routing: frontier for planning, open-weight for execution 09:40 Symbiotic's stack: Cursor + homegrown SymCloud agentic harness 11:20 Replit's internal self-driving software engineering tool 12:20 MCP servers, skills repositories, and the 6-hour autonomous bug fix 13:20 Build vs. buy: where the real cost is maintenance, not construction 15:40 Replit replaces a seven-figure SaaS tool built entirely in-house 17:40 Token maxing: how teams burned their annual AI budget by July 18:20 Cost per PR as a ROI metric for AI coding spend 21:00 Symbiotic's tiered monthly token budget system for 900 engineers 23:20 Replit's 3x engineering productivity — and the GPT-5.5 Pro Max bill that forced a reckoning 24:20 Audience Q&A: model routing as a product layer for ISVs 26:20 Audience Q&A: PR review bottleneck, code deletion, and what review looks like next 29:40 6-month predictions: open-weight dominance, 24–48hr agent work loops, model sovereignty --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AgenticCoding #AIAgents #LLMInfrastructure #AIDeployment --- Learn more about your ad choices. Visit megaphone.fm/adchoices
66% of enterprise AI teams are actively engineering toward zero human oversight — but only 4.5% fully trust their automated evals. Half have already shipped an agent that passed evals and still failed a real customer. VentureBeat's latest research exposes a dangerous gap between deployment ambition and evaluation maturity. The most-used eval platform is OpenAI's native tooling. The second most common answer: no dedicated eval platform at all. Meanwhile, 51% of enterprises are relying on a hyperscaler's built-in guardrails as their primary AI security layer — not a deliberate security architecture. Fable's 19-day market absence changed the conversation. Enterprises that had built production workflows on frontier models got a wake-up call about single-vendor dependency. The result: GLM 5.2 is gaining serious developer traction as an open alternative competitive with Opus 4.8, and Tencent's new Hai 3 — less than half the size — is drawing attention for agentic workloads that don't require cutting-edge coding performance. Sam and Matt break down why companies like Shopify, LinkedIn, and Pinterest were already on open models, and why others are now catching up fast. On infrastructure: neo-clouds are currently used by just 2% of respondents, but 45% name them as their top planned investment area over the next 12 months. The market hasn't calcified. Matt and Sam preview what they expect to hear at VentureBeat Transform from engineering leaders at Intuit, Brex, Instacart, and Mastercard — including how Intuit bootstrapped and pivoted its agent architecture mid-build, and why agent conversations are generating higher-quality behavioral data than traditional user research ever did. 🎙️ HOSTS: Matt Marshall · Sam Witteveen | VentureBeat __ If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z — Chapters: 00:00 Research data drop: eval gaps, zero-HITL targets, and hyperscaler security dependency 01:00 Chinese models breaking into enterprise — developer fad or structural shift? 02:00 GLM 5.2 and Tencent Hai 3: what they do well and where they fall short 03:20 How Shopify, LinkedIn, and Pinterest are already running open models in production 05:40 Fable's 19-day outage: why 19 days changed enterprise AI vendor strategy 07:00 Intuit's agent build: bootstrapping, mid-build pivots, and real-time behavioral data 08:40 Neo-cloud infrastructure: 2% current usage, 45% planned investment next 12 months 09:20 Is the enterprise AI market still wide open? Anthropic vs. OpenAI for enterprise lead 11:20 The routing question: when do you send workloads to smarter vs. cheaper models? 12:20 Transform preview: what to expect from Amazon, Databricks, Brex, Instacart, Mastercard --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #LLM #MLOps #AgenticAI — Learn more about your ad choices. Visit megaphone.fm/adchoices
Shopify's distillation pipeline cuts production AI costs by up to 30x — and in some cases, the smaller model outperforms the frontier model on the narrow task. That's not a trade-off. That's a win on accuracy, latency, and cost simultaneously. Farhan Thawar, VP & Head of Engineering at Shopify, runs AI across one of the largest commerce platforms on earth. In this episode, he breaks down the exact infrastructure decisions Shopify made to avoid being locked into any single model provider — and why 29% of enterprise AI projects die from token costs, not model failure. Shopify built an internal LLM proxy that routes tokens across every major provider, enabling automatic failover when any one goes down. On top of that, their Universal Distillation Platform (UDP) lets any R&D team distill a frontier model (Opus 4, GPT-5+) down to a fine-tuned open source model (Qwen and others) for a specific subtask — in roughly a day, with evals baked in. Results range from 2x to 30x cheaper, faster, and more accurate than calling the frontier API for everything. Shopify currently runs roughly half a dozen of these distilled models in production, with more being added. Farhan also details River, Shopify's internal agentic substrate — a public-only Slack agent that queries their data warehouse, reads their PM system, and improves its own answers when engineers jump in to correct it. Plus: how Shopify governs AI-generated code at scale, why they killed their token leaderboard, how UCP is positioning Shopify's catalog for agentic commerce, and what a two-to-three year horizon looks like when agents start holding spending budgets and buying autonomously. 🎙️ GUEST: Farhan Thawar | VP & Head of Engineering, Shopify 🎙️ HOST: Sam Witteveen | VentureBeat __ If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z — **CHAPTERS** 00:00 Intro — Infrastructure-first philosophy at Shopify 02:00 Episode overview: token economy, distillation, and the cost crisis killing AI projects 03:00 Toby's "AI reflexivity" mandate and what it actually means for engineers 05:20 Ecosystem strategy: when to build vs. leave room for third-party developers 06:20 Developer tooling stack: GitHub Copilot (2021), Claude Code, Cursor, Codex, and Shopify's own River 08:00 LLM proxy architecture: bulk token purchasing, multi-provider failover, and usage reporting 09:00 Model agnosticism: why Shopify lets engineers choose their own harness 10:00 AI adoption beyond R&D — finance, HR, sales, and the Qwik internal deployment platform 11:20 Code governance: who owns AI-generated code going to production 13:00 Token maxing, leaderboards, and the shift from AI reflexivity to AI leverage 14:20 Circuit breakers: how Shopify catches runaway token spend without hard limits 15:20 LLM proxy deep dive: uptime, insights, and cross-team learning from usage data 16:40 River: Shopify's agentic substrate, public-channel-only design, and emergent HITL behavior 18:40 Model distillation explained: teacher/student models, narrow tasks, and the trade-offs 21:00 Universal Distillation Platform (UDP): how any team submits a distillation job in ~one day 22:20 Tangle: open source pipeline visualization for distillation workflows 22:40 Who uses UDP today, and Farhan's vision for auto-selecting the distillation target model 24:00 Evals in practice: golden datasets, Toloka for data generation, threshold-based deployment 26:20 Sim Gym: simulating A/B tests for small merchants without enough traffic 27:40 Pulse: async AI insights on store performance and conversion 28:20 GPU infrastructure trade-offs: when running your own inference makes sense at scale 29:20 Frontier vs. distilled model split in production — and why dev tokens stay frontier 30:40 Should Shopify train its own coding model? Why Farhan says not yet 31:40 UCP protocol, agentic commerce, and how Shopify's catalog surfaces in every LLM 35:00 Early signals: agentic commerce growth rate and the shift away from SEO 36:00 What developers and entrepreneurs should build for now given multi-channel uncertainty 37:20 River as a hive-mind agent — and what "truly agentic" actually means 38:40 Two-to-three year forecast: agents with spending budgets, autonomous purchasing, proactive outreach 40:00 Project Glasswing, model access loss (Fable), and the case for multi-provider architecture 42:00 Why every company should have a backup plan — and how Shopify built theirs years ago 43:20 Wrap-up --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website
If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z _ He negotiated seat-based (unlimited) licenses before token costs exploded — and projects only a 20–30% spend increase when that changes. MassMutual's CIO rebuilt a COBOL mainframe app into a working web prototype in 7 days — work that used to take a 15-person SI team 90 days. That's not a pilot. That's a new build-vs-buy equation. Sears Merritt, CIO at MassMutual, runs AI inside one of America's most regulated legacy environments. In this conversation, he breaks down the architecture decisions, cost structures, and security posture behind real, production deployments — not roadmaps. On infrastructure: MassMutual routes all agentic tool calls through centralized API gateways with identity and access controls, using Amazon Bedrock as a proxy layer. That multi-harness design preserves model optionality while enforcing FinOps discipline. On model selection: a trust score rubric drives every model decision, balancing cost against user experience. In their IT contact center, that rubric led them to choose the more expensive model after users said the quality gap was worth two extra seconds of inference time. Productivity results are concrete: 30% boost in developer output across the SDLC, call resolution time dropping from 10 minutes to under 1 minute for specific call types, cost per interaction from dollars to cents. On the security side, MassMutual is embedding AI into its SDLC for vulnerability scanning and compressing cyber response cycles from days to hours — building agentic tier-one and tier-two capabilities to match the accelerated threat landscape that frontier models like Mythos have exposed. 🎙️ GUEST: Sears Merritt | Head of Enterprise Technology & Experience, MassMutual 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat — If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z — 00:00 Intro & COBOL Modernization Preview 00:01:15 Guest Introduction: Sears Merritt, MassMutual CIO 00:02:00 Multi-Vendor Strategy & Avoiding Lock-In 00:02:30 How MassMutual Evaluates AI Tools (Cost vs. Experience Rubric) 00:03:15 12-Month Contracts and Switching Optionality 00:03:30 AI Standards Cycle: MCP, A2A, and the Early Internet Analogy 00:05:15 Measuring Developer Productivity: 30% SDLC Boost 00:06:15 Contact Center Results: 10 Minutes to 1 Minute, Dollars to Cents 00:07:00 Managing Token Cost Explosion 00:07:30 Seat-Based vs. Consumption Licensing Decision 00:08:45 Token Maxing While the All-You-Can-Eat Window Is Open 00:09:45 Building FinOps Infrastructure for Model Routing and Optimization 00:11:15 Outcome-First Model Selection: When to Pay for Opus vs. a Cheaper LLM 00:13:15 Trust Score Framework: How MassMutual Picks the Right Model 00:15:00 Sponsor: OutShift by Cisco 00:15:30 Claude, OpenAI Codex, and Multi-Harness Agentic Architecture 00:16:30 API Gateway Design: Identity, Access, and FinOps Controls 00:17:30 What the Usage Analytics Revealed (And What Merritt Was Afraid to Find) 00:18:15 Projected Token Cost Increase: 20–30% Off Unlimited Plan 00:19:45 Power Law Usage: Top 10% Consuming 80% of Tokens 00:20:45 COBOL Mainframe Modernization: The 7-Day Prototype Workflow 00:22:30 The Full AI-Assisted COBOL Migration Playbook 00:24:00 Implications for IBM and Mainframe-as-a-Service Providers 00:25:15 Open Source Models, DeepSeek, and the Cost Efficiency Question 00:27:45 Chinese Models in a Regulated Environment: Evaluation Criteria 00:29:30 Agentic Security: Identity Management and the Evolving Threat Landscape 00:30:00 How Frontier Models Changed the Threat Velocity (Not the Threat Types) 00:31:15 Fighting AI With AI: Agentic Tier-1 and Tier-2 Cyber Capabilities 00:31:30 Project Glasswing and the CISO Community Response 00:32:30 Embedding AI Into the SDLC for Security Scanning 00:34:00 Closing: When Will Agentic Standards Consolidate? Advice for Builders --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #LLMInfrastructure #AIDeployment #MLOps Learn more about your ad
Pinterest's open-source AI stack costs 90% less than frontier models — and their custom-trained recommender outperforms off-the-shelf alternatives by 30% in accuracy. Pinterest CTO Matt Madrigal breaks down exactly how they did it, and what enterprise AI teams can actually replicate. Madrigal walks through the full architecture behind Navigator 1, Pinterest's conversational shopping assistant built on Qwen 3 VL — and the specific decision to rip out its native vision encoder and replace it with PinCLIP, Pinterest's proprietary multimodal embedding layer. That swap alone closes a 20x inference latency gap and makes the economics work at 620 million monthly active users. This is the clearest public explanation yet of how a scaled platform operationalizes the "core vs. context" principle for model selection: open-source and custom-built where it touches the user, frontier models where speed-to-prototype matters more than cost. The conversation also covers the Taste Graph — Pinterest's knowledge graph across hundreds of billions of pins and 15 billion boards — and how post-training on that proprietary data lets a smaller, fit-for-purpose model beat a larger frontier model on production metrics. Madrigal details their eval framework: gold set benchmarks, product-level evals tied to engagement and merchant click outcomes, and a structured A/B test pipeline that runs from engineer PRs through to live user signal. On the organizational side: how Pinterest manages a "default yes" multi-IDE policy (Cursor, Windsurf, Claude Code, Codex) without collapsing security posture, how they segment sandbox environments between ML engineers with Taste Graph access and general application developers, and why Madrigal measures AI coding ROI in token usage and experimentation velocity — not lines of code. 🎙️ GUEST: Matt Madrigal | CTO, Pinterest 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat 00:00 Show Intro and Guest 01:17 Open Source Cost Breakdown 02:20 Pinterest Multimodal Roots 02:37 PinClip and Embeddings 05:46 Core vs Context Models 07:43 Navigator 1 Assistant Stack 11:52 Benchmarking and Evals 13:29 Accuracy from Proprietary Data 17:16 Taste Graph Explained 18:29 Taste Graph in Training 22:22 Fighting AI Slop 25:16 Developer Tools and Velocity 27:57 Tool Choice and Governance 28:56 Security Sandboxes and CICD 30:57 Wrap Up If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #OpenSourceAI #AIInfrastructure #LLM #MachineLearning Learn more about your ad choices. Visit megaphone.fm/adchoices
Free AI-powered daily recaps. Key takeaways, quotes, and mentions — in a 5-minute read.
Get Free Summaries →Free forever for up to 3 podcasts. No credit card required.
Listeners also like.

Me, Myself, and AI
Leaders from top companies share real-world strategies for turning artificial intelligence into measurable business results.

The AI in Business Podcast
Interviews with AI executives reveal practical applications and strategic insights for business leaders adopting AI.

Learn AI
Learn AI

AI For Humans: Weekly AI News, Tools & Trends
A weekly breakdown of major AI news, tools, and breakthroughs for both newcomers and seasoned enthusiasts.

NVIDIA AI Podcast
Explores how artificial intelligence and emerging technologies are driving innovation across science, sustainability, and industry.

This Week in AI
Four AI experts analyze the latest news and trends in artificial intelligence each week.

What Comes Next with Arun
Explores how data and AI strategy create competitive advantage in business, based on real-world enterprise experience.

Pioneers of AI
AI experts discuss how artificial intelligence is changing society and shaping the future.

No Priors: Artificial Intelligence | Technology | Startups
Discusses AI advancements and implications with researchers, engineers, and founders shaping the future of artificial intelligence.

How I AI
A practical guide to using AI tools in work and life, featuring guests who share specific, actionable techniques and workflows.

Sources with Alex Heath
Candid conversations with tech leaders and founders about AI, innovation, and the future of Silicon Valley.

AI + a16z
Discussions with AI engineers, founders, and experts on the technological and industry shifts in artificial intelligence.
AI gets real here. On “Beyond the Pilot,” top business execs share what actually happens after the AI proof of concept — from infrastructure and org design to wins, failures, and ROI. Not theory, but deep dives into how they scaled AI that works.
AI-powered recaps with compact key takeaways, quotes, and insights.
Get key takeaways from Beyond The Pilot: Enterprise AI in Action in a 5-minute read.
Stay current on your favorite podcasts without falling behind.
It's a free AI-powered email that summarizes new episodes of Beyond The Pilot: Enterprise AI in Action as soon as they're published. You get the key takeaways, notable quotes, and links & mentions — all in a quick read.
When a new episode drops, our AI transcribes and analyzes it, then generates a personalized summary tailored to your interests and profession. It's delivered to your inbox every morning.
No. Podzilla is an independent service that summarizes publicly available podcast content. We're not affiliated with or endorsed by VentureBeat.
Absolutely! The free plan covers up to 3 podcasts, any in the catalog. Basic raises that to 15, Pro to 40 and Max to 100. Browse our full catalog at /podcasts.
Beyond The Pilot: Enterprise AI in Action publishes biweekly. Our AI generates a summary within hours of each new episode.
Beyond The Pilot: Enterprise AI in Action covers topics including News, Technology. Our AI identifies the specific themes in each episode and highlights what matters most to you.
Free forever for up to 3 podcasts. No credit card required.
Free forever for up to 3 podcasts. No credit card required.