OpenAI’s Data Agent vs Talk2Astra, Our AI conversational Agent for Private Equity
By Sai Mali A., Ph.D., Chief AI Officer & Shailesh Sridhar, AI/ML Engineer
Recently, #OpenAI released Data, a first-party plugin for #ChatGPT that connects to your warehouse so you can ask a question in plain English. Data picks the tables, writes the queries, runs them, and returns an answer. The pitch is that you can ask why a metric moved, where a team should focus, or how big an opportunity is, and get something back that you can share. It also claims that the plugin checks its sources where possible.We refer to this as the “Data Agent” throughout the article.
Our internal AI team tested the Data Agent against our conversational agent, #Talk2Astra, using data from a PE-backed industrials client (all data was anonymized). To run this test, we used real questions that were asked of Talk2Astra.
Rules of the Test
An answer counted only if it had the right number andthe disclosure a finance team requires before using it: what basis the figure was calculated on, and what caveat applies. The standard we used was whether a CFO could put the response into a board pack without requiring deep review. No partial credit was allowed.
Why Talk2Astra Won
Talk2Astra came into the test with three things that the Data Agent didn’t have:
1. Modeled data
Tables cleaned up and joined, so the agent doesn’t need to rebuild every definition from the raw source systems. On one question, rebuilding those definitions from raw data pushed the Data Agent from 17 steps to 46.
2. The business’s actual definitions
These are the conventions that usually live inside people’s heads, things like what counts as a product platform, which cost lines run negative, and how to match a product’s parts list to the right plant and the right month.
These definitions are encoded inside our platform, Astradis.
3. Orchestration
Talk2Astra already knows where to find the answer to a question, so it skips the discovery and goes directly to compiling the right data. Before it shows a number, it checks that the number is realistic and stops any answer that’s way off.
To be clear, OpenAI claims this: reusable context is a feature of the product, so our subsequent runs used the Data Agent how it was intended to be used. Most gaps were found in the data and definitions, yet even when both were shared, Talk2Astra retained the advantage overall.
We also tested an early version of our semantic layer (a guide that tells an AI agent exactly where every business term lives in the data) which again lowered the Data Agent’s score. The “deep dive” at the end of this article explains why that happened.
Why an LLM can check sources and still confidently return a wrong answer
OpenAI says the Data Agent checks its sources wherever it can and keeps the evidence visible. Yet, it still produced answers that were confident and wrong.
Several incorrect answers confusingly arrived with a named or explicit table, making them appear right. In this test, we paid close attention to accuracy. Imagine you are moving fast, and you have a clear table and chart that’s explainable and seemingly derived from your business. This gives you false confidence to proceed, unknowingly resharing incorrect information.
The cost to resolve an error of this magnitude is hard to put a number on, not to mention the diminished credibility of the person who shared it.
The failures are different, too.
For example, the Data Agent’s wrong answers are often wrong by large multiples, and the mechanism is consistent: it reaches a general table, finds it cannot express the bucket the question asks for, invents a filter, and commits.
One of Talk2Astra’s errors was in a roll-up cost for a product line. The company already publishes that figure, and Talk2Astra worked it out from the underlying data instead of reading it. Talk2Astra included the right items, but calculated them on a different basis, so the answer came in 12% low.
Impact of a wrong number
Ask an AI tool about their pass rate, and you’ll learn very little. For example, 63% and 85% sound somewhere in the same neighborhood.
The problem is that when accuracy matters, they are actually far apart.
Talk2Astra’s primary miss came in 12% below on a published cost. That’s the kind of gap a controller catches on a variance report. A number that’s 12% off is still directionally correct, so insights drawn from it are more likely to hold.
Across the configurations we tested, the Data Agent’s misses ranged from 18% to 671%.
To exacerbate the issue, some of the responses had clear explainability and supporting data, making them easy to fall for.
However, a number never stays static. It moves into the monthly board pack, a pricing review, or a sponsor update. Other teams pick up the incorrect number and use it. By the time someone traces the discrepancy, you’re unwinding a weeks’ worth of decisions, things like a quoted price or an approved spend; work that can’t always be reversed.
The other problem is what endures after the error is resolved. A board that catches one bad number checks the next one, and the one after that. Finance goes back to verifying everything manually, which is the exact bottleneck these AI tools were meant to remove.
What this means for a portfolio company
Most companies buy an AI agent first and plan data work for a second phase.
That’s the wrong order. Here’s why…
On raw data, the Data Agent got 22% of our questions right. Pointed at modeled tables, it got 63%. Once we added the definitions Talk2Astra uses, it got 78%. The agent and the questions never changed, yet the data and definitions account for most of the gap between 22% and Talk2Astra’s 85% accuracy.
By starting with an AI agent, your team is working with a tool that’s right ~1 in 5 times. And, the wrong answers come with a source attached, so they look convincingly accurate.
The lesson? Write your definitions down and give them to whichever agent you use.
On raw data, we asked the Data Agent for revenue across five product platforms. It made up its own meaning of “platform” and overstated the answer by 7x. When we handed it the definitions Talk2Astra uses, it went from 63% to 78% accuracy, with no other changes.
Building the foundation is central to the work we do with PE-backed companies, and critical to the ‘value’ derived from any third-party AI tool.
Interested in learning more about Talk2Astra and how we work with PE-backed companies to drive value? Reach out: info@teragonia.com
The test
The questions. Twenty-seven questions about cost and margin, written over several rounds of testing to match what the company’s own people actually ask. Roughly half are totals. The rest ask which thing drove a number, which is where the work is. Correct answers came from the company’s issue-report reporting, validated test runs, and direct confirmation from its finance team.
Naming. The Data Agent is OpenAI’s data-analysis product, distinct from the general ChatGPT model. Talk2Astra is Teragonia’s conversational agent inside Astradis, its AI operating platform for private equity. This benchmark compares those two capabilities and nothing else.
How answers were scored
Each question carries up to three checks. A value, which must land within tolerance, usually 3%, of an independently verified number. A forbidden value, meaning specific known-wrong figures from earlier defects that must not reappear. And wording: a required caveat, the basis a number was computed on, or specific items the question asks to be named.
Board-ready answersrequire all three. Miss the number or omit a required disclosure and the question fails. This is the column to read if what you want to know is whether an answer could go to a board unedited.
Right number, caveats ignoredre-scores the same stored answers with the wording checks off, which separates the arithmetic from the communication. It is the softer view. A right figure with no stated basis still causes trouble in a finance function, because the next person to use it has no idea what it includes. It is counted out of 21 rather than 27 because six questions have no numeric anchor. Both columns are shown because the ranking is identical under each, which is the check against grading to a result.
Caveats
Talk2Astra’s figure is what it got in two out of three runs. Every Data Agent figure is a single run. Three runs of identical Talk2Astra software scored 85%, 85% and 78%, with four questions flipping between them. The ranking is the finding. The decimals will move between runs.
Every row is the same 27 questions. One further question was withdrawn from the set because our own reference answer for it was wrong. No system is scored on it.
One client, one engagement. Consistent with what we see elsewhere, and not an industry study.
Anonymization. Client identity withheld. Figures rounded, and one example is shown as a ratio rather than an absolute. Detailed per-question results, timings and methodology are available to technical reviewers on request.
Every wrong number, by setup
Layer by layer
Each component below is the single difference between two Data Agent configurations, measured against raw source tables as the base. Because we changed one thing at a time, each effect is measured directly rather than inferred from the agent’s reasoning. Against that base, modeling the data is worth +41 points, our semantic-model prototype −11, our definitions handed to the Data Agent +15, and Talk2Astra as a whole +22.
About Teragonia
Teragonia turns private equity value creation playbooks into agents that drive the operating rhythm of mid-market businesses within commercial, operations and finance functions. Its team pairs operators who have spent their careers inside private equity, as investors, management teams, and M&A advisors, with AI and ML engineers who have built production systems at Morgan Stanley, GE Research, Barclays, Deloitte, and PwC.
An AI engineering leader and systems architect with a PhD in Aerospace Engineering from the University of Toronto and over a decade of experience across the full delivery lifecycle, Kwesi turns frontier model capability into governed systems that hold up under real operational load. Throughout his career, he has led enterprise technical teams and delivered production-grade, governed AI platforms across major financial institutions, pensions, and utilities. At Teragonia, he combines computational depth with enterprise delivery experience to help private equity firms and their portfolio companies capture value through applied AI across the full investment lifecycle.
Foundational Tech Infrastructure
Our core analytics and AI platform drives informed decision-making with enhanced clarity and focus, and rapidly unlocks enterprise value
Core Features:
Connect All Your Data Sources
Integrate data from multiple source systems effortlessly
Critical for breaking down silos and creating a unified view
One Trusted Data Source
A secure, centralized cloud hub for all your data insights
Foundational for reliable decision-making and enterprise-wide alignment
Interactive Dashboards
Visualize complex data through easy-to-understand dashboards
Empowers leaders with actionable insights
Enriched Data Flow, Fully Automated
Reverse ETL capabilities enriches your data and ensures your data flows exactly where it is needed for function teams to act on
Enables real-time, action-oriented data flow
Additional AI capabilities are actively in development
Scott Briggs
BS International Business | American University of Paris
BS Computer Science | American University of Paris
Seasoned DevOps and infrastructure engineer with expertise in AWS, Kubernetes, and Terraform; led cloud migrations and scalable infrastructure projects at Sfara, FanDuel, and Kickstarter.
With over 15 years of experience in small and medium-sized startups, Scott is a seasoned expert in designing, optimizing, and maintaining robust, scalable, and secure infrastructure. He specializes in automation and embedding security from the ground up, consistently delivering reliable systems tailored to meet dynamic business requirements.
Prior to joining Teragonia, Scott made a significant impact at Sfara, where he built the company’s entire infrastructure from scratch. He engineered systems capable of supporting hundreds of thousands of users with seamless scalability, implemented automated development pipelines, and introduced observability tools to monitor and manage resources effectively. Additionally, Scott led the infrastructure team in achieving ISO27001 security certification, ensuring security was integrated into every aspect of the system and transforming it into a critical asset for business-to-business operations.
Beyond his technical expertise, Scott has a proven track record of managing and mentoring high-performing teams. As a Senior DevOps Engineer at FanDuel, he gained invaluable experience in scaling infrastructure and optimizing resources to support millions of daily users, aligning technological capabilities with organizational goals.
Jack Amedio
Master’s in Human Resources | University of Illinois
Bachelor’s in Management | Loyola University
Former Financial and Operations Manager at Houlihan Lokey, Golin Harris, and MSL Group.
Jack is a highly driven, cross functional professional with extensive experience in operations and administration.
Prior to joining Teragonia, Jack held financial and facilities management roles for Houlihan Lokey, MSL Group/Publicis, and Golin Harris in which managed and created processes and trainings for multiple functional areas ensuring operational and administrative procedures were well planned, efficient, cost-effective, and aligned with business objectives while ensuring initiatives, internal events as well as client events propelled employee and client engagement.
Jack holds undergraduate degrees from University of Illinois and Loyola University Chicago and has completed graduate certificates in Business Administration, Strategic Human Resources, and Operations at Cornell, CUNY-Buffalo, and University of Illinois and is in the process of completing a Master’s in Human Resources at Loyola University Chicago’s Quinlan School of Business.
Mason Taylor
MS Analytics | Georgia Institute of Technology
BS Management Information Systems | Oklahoma State University
Former analytics engineer at Cyderes and ConocoPhillips with a Master’s in Analytics from Georgia Institute of Technology and a Bachelor’s in Management Information Systems from Oklahoma State University
Mason is an Analytics Engineer with deep experience in data analytics, business intelligence, machine learning, and cybersecurity. He brings a proven track record of leading analytics engagements spanning architecture, insights, visualizations, and delivery.
Before joining Teragonia, Mason was a Senior Analytics Engineer at Cybersecurity MSSP CYDERES where he built a scalable, standardized, and secure analytics architecture for over 300 clients across many industries and consulted with them to deliver insights through bespoke data driven solutions. In addition, he managed the data delivery of the insight platform leveraged by the Security Operations Center to respond to incidents in a timely and effective manner.
Prior to joining CYDERES, Mason worked in ConocoPhillips’ Analytics and Innovation Center of Excellence holding varied roles within the Data Analytics organization from Data Engineering, to Business Intelligence, and Data Science. He delivered robust data solutions in all operating units for various functions including Engineering and Production, Finance, IT, and more. Including projects to standardize cost and production data across operating units.
Mason started his career at The Williams Companies in cybersecurity and transitioned to cybersecurity at ConocoPhillips where he found his passion for Data Analytics through SIEM management, detection engineering, and threat intelligence.
Grace Sun
Bachelor’s in Finance & Accounting | Georgetown University
Former analytics engineer at Houlihan Lokey and financial analytics at JP Morgan Chase with a Bachelor’s in Finance & Accounting at Georgetown University
Grace is a seasoned analytics engineer with specialized expertise in crafting and implementing analytics solutions that drive agile, informed executive decisions in M&A and value creation for private equity-backed companies.
Before joining Teragonia, Grace was a part of the data science and business analytics team at Houlihan Lokey. She has excelled in harmonizing, enriching, and analyzing data from diverse sources, providing key insights that enabled private equity investors and portfolio company executives to make rapid, data-driven decisions across the investment lifecycle. She has developed novel analytics solutions, including deal sourcing and evaluation tools for platform investments that employ a buy-and-build or de novo growth strategy, as well as post-close value creation and KPI reporting tools for operators and management teams.
Grace has also worked at JPMorgan Chase & Co. in the Global Finance and Business Management rotational program, where she built analytics solutions to evaluate banker attrition and KPI reporting within the Global Private Bank.