0:00All right. Hey, everybody. Welcome to the show. We are super pumped to have you here, and I am super pumped to have Peter here from Virgil Data with me. Quickly, before I intro Peter, for those of you who don't know, I'm Jacob. I work in DevRel at MotherDuck. MotherDuck is building a scalable cloud data warehouse built on top of DuckDB that does all the annoying things
0:23that you don't want to worry about when you're using DuckDB. With that being said, I am going to hand it off to Peter to introduce himself and then get us rolling. Thanks, Jacob. I'm Peter. I'm the founder of Virgil Data, and we're going to be doing a deep
0:39dive today into context for data agents. I wanted to start with the promise of data agents, which looks like this. Here I am in Claude co-work, and I hooked it up to my data warehouse, and I asked it a question and got an answer back. This is pretty amazing. I think this is what the
1:03future of BI looks like. I didn't have to write any SQL. I didn't have to craft a semantic layer. Claude did all the work for me. Lurking beneath this extremely cool use case is an unfortunate truth, which is that if I ask the same question 10 times, I get a bunch of different answers.
1:23Sometimes, Claude tells me that our revenue was 12. 7 euros. Sometimes, it tells me it was 10 million. Sometimes, I get an answer of 11 million. That's the problem we're going to be solving today. How do we get consistent, accurate answers from the unpredictable machines that are
1:43LLMs? The answer you probably won't be surprised to hear is context. Today, we're going to talk a lot about what good context looks like, and we're going to give you some best practices for writing and managing context in your organization that you can start implementing today. Why do I care so much about context? As I mentioned, I'm Peter. I'm the founder of
2:08Virgil Data. Virgil Data is a platform that writes, serves, and tests context for data agents. Because our job is to write context, we care a lot about knowing what good context looks like, and today, I'm going to share some of that research with you. I want to start with what happens when there is no context. What I've done here is I've plugged
2:32Claude into a sample database called DABSTEP. If you're following Jacob, you might have heard of DABSTEP. It's a sample data set with a bunch of credit card information. For this base case, I'm not giving Claude any context at all. I'm going to run a bunch of tests against it and see how it
2:50does. As you might expect, it doesn't do so well. Here's what we're looking at. These are tests. Tests are simple prompts that we run 10 times, and we compare the answers they generate to the correct answer. You can see here for some of these simpler questions, Claude's able to get it right
3:13every single time, but for questions that involve understanding this business or this data set, Claude consistently gets it wrong. Our goal is to get to greens across the board, and we're going to do that by writing good context. What is good context? Well, we define
3:36good context across three dimensions. The most important aspect of your context is that it's accurate, which is to say it lets data agents correctly answer the questions that you ask them. The second dimension we care about is speed. Your context can have a significant impact
3:55on how quickly data agents answer questions, and that also affects your costs. The last thing we care about is maintainability. Is your context easy to read and write? Does it break easily when your schema changes? We're going to take these dimensions, and we're going to apply them to
4:15three common formats of context that we see out in the wild. We're going to look at an English-only context file. We're going to look at using a semantic model as a context file, and we're going to look at a context file that's composed purely of SQL. Jacob, how's our polling functionality? Our polling's looking great. Should we throw the poll up? Let's throw the poll up.
4:40All right, we have a poll. Let's go. Let's go. Which is best? There is a prize. Oh, love it. Love it. Okay, so we're going to look at these three different formats
5:00and assess each of them for accuracy, speed, and maintainability. I'll give two more seconds for poll answers to come in. We got a lot of semantic model votes. Cool. Okay, no one votes for English. We're going to start with English. Here's what an English context file looks like.
5:19It's a bunch of definitions and a little bit of schema explanation, and many of you will be surprised to find out that English context does really well. We are able to get to complete accuracy just writing English. While this may seem surprising,
5:42we know that LLMs have been trained on vast reams of English. English is actually a great language to feed them. They are fluent in English. They know how to read and write it. English is also really easy to maintain. You don't need any technical acumen to write English context. It doesn't break when your schema changes. Jacob, what are some things
6:04our audience should know about English context from a database perspective? Yeah, this is such a good question. One thing that I think the natural instinct is when you start adding context to whatever engine you're using is you're like, okay, well,
6:23I should just expand the column description. If your column description is company name, or your column name in SQL is company name, you'll be like, okay, let's add some text that says this is the name of the company and something like that. Don't do that. All it does is add extra
6:42tokens. What you want to do is add information that is not obvious to the LLM. If you're using words to mean other concepts than what they normally mean, or you're getting more specific, those are really great ways to add context. The column name by itself is going to be the first thing the LLM looks at, so those should be good too. I think the other thing is that
7:07negative prompting doesn't seem to work super well on current generation LLMs. What I mean by that is when you say, hey, we want to use median, not average here, that seems to, depending on the model you're using, again, you're probably not going to run the highest end models all the time when you're running this stuff, but especially on lower,
7:29less sophisticated models, negative prompting causes it to actually use the function you don't want it to use more often instead, or worse, use it in something totally unrelated because it read that context and said, oh, there's a note about not using average, and so it uses something else that it shouldn't use when it doesn't need to aggregate or something. Figuring out how to land
7:50positive context and then use your language to either add to the world that the LLM knows about or get more specific are the ways I think about using English to interact with it. Awesome. Our next contender is the semantic model. There's a bunch of different formats of this out
8:14there. Cube, LookML, Omni, but they all share similar properties, which is they're these highly structured languages designed to take higher level concepts and programmatically compile them into SQL. I saw we got a lot of votes for semantic modeling as the best possible
8:35context format. Jacob, if you were to channel someone who voted for this, why might this be an appealing context format? Yeah, it's such a good question. I think, look, we want to get the same answer every time. We want it to be right 100% of the time. We really care about the result being
8:56consistent because we want to empower everyone who's using it downstream to be speaking the same language. I think we really care about the composition in that case. That's my use case. Also, generally sometimes there's things that are very hard to model in English or SQL. What comes
9:15to mind is the symmetric aggregate stuff from LookML and now Malloy. Those things are hard to model in other places. Maybe you want to use a semantic layer. Yeah. Bad news for all of you who voted too. The semantic model format is the worst format for context of the three that we're
9:39looking at today. It does okay on some questions. Yeah. But for some of these harder questions, it failed completely. Which model, Peter? What was this? Which model are you using? This cube. No, no, sorry. Which language model? Oh, Claude. Okay, great. So, Opus 4. 6 or 4. 7 on this.
10:01Yeah, yeah. Thank you. Amazing. Hey, this is just one set of results for one sample data set. So, I encourage you to run your own tests on your data with your context. But I want to point out that this result is not surprising because while we know that LLMs have been trained
10:20on lots and lots of English and lots and lots of SQL, LLMs as a rule have not been trained on these semantic modeling languages. So, we shouldn't expect them to be great at interpreting them. We also know that these semantic modeling languages were built in a pre-LLM era. So, they weren't designed to serve as context for LLMs. So, if you take one thing from this
10:45presentation, let it be this. Do not feed your raw semantic model into an LLM as context. Take the step of getting an LLM to first translate it into English and you'll immediately get better results and a context layer that's easier to maintain. Okay. Now, we're going to contender
11:05number three, SQL. Here, our SQL context file is just a bunch of question and answer pairs. Jacob, how do you think SQL performs? It's probably pretty good. I mean, this is like a home game for me. I was like the lone SQL voter. So,
11:26I would say probably better than semantic models and slightly worse than SQL would be my guess. Or sorry, slightly worse than English, but we'll see. Oh, no, no, no, you're fine. You're fine. It's a weird artifact. You're actually right, Jacob. SQL does really well. We're not seeing your screen anymore. We just lost it, by the way. I don't know. Go back a slide, maybe.
11:49I don't know. It seems like it went away. Let me. . . It's the tragedy of a live demo. Yeah, yeah. It popped in. The eval result popped in and then
12:03popped out. Oh, oh, something is happening. Okay. We'll just wait. Here we go. Here's those tens. And I want to point out, there's one important dimension in which SQL outperforms English.
12:28What do you think, Chad? How does SQL do better than English? Oh, I'm going to put a comment in here so I don't pollute it with my own thoughts. I'll pull up English and then we'll start English loading. We'll start English loading. So here's English. Here's our English results. And here are our SQL results. Who can spot the difference?
12:57Determinism. Fewer steps. All points to Jack Welty. That's right. SQL can perform substantially faster than English as context. And I want to dive into why that is. So let's look at this kind of complicated question. Our SQL-based context consistently does it in two to three steps.
13:21And the way it's able to do that is it looks at the context, it finds a sample query that is very similar, and it changes one parameter. So it doesn't have to do any exploration at all. It says, oh, I know exactly the SQL to write. I just have to swap out the vendor ID. Now let's look at what happens when we ask the same question, but give it English context.
13:46Here, our agent is pulling up relevant documentation. It understands the steps it needs to take, but it has to do a bunch of exploratory analysis to get to the right answer. So if there are big thorny queries that you want your LLMs to generate all the time,
14:07and you care about performance, consider adding some snippets of SQL to really speed up those questions. SQL is not without its drawbacks. It is significantly more brittle in English. Because sample SQL that you write is directly tied to your schema, when you update your schema, your SQL-based context will break.
14:33Jacob, you've done a lot of research on how to leverage SQL as context effectively. What do you want to add here? Yeah. Views are great, which Peter nicely put in the slide. And I'll call it out for two reasons. One of them is you can give them names that are very specific. You can tell the system what the view name is, so it's easy for it to find it in semantic search
14:58that the LLMs are really good at. So that's the first part. The second part is the LLM can also introspect the view, right? Because it's in the information schema of the database. And because of that, it also means you can say, hmm, I see that I have a view here that has this name. Maybe it says like, you know, payment ACI calculation or something. That's like related to the DAB set problem set. So, oh, hmm, well, I have something that's related to that,
15:21but maybe not exactly the same, or I have an idea of how I want to solve that. Let me go look and see how this view solves it by inspecting the information schema and seeing how similar that is to the way I'm thinking about it. And so the LLM can use that as additional context, in addition to actually using the view, it can also introspect it. And I think those are super, super helpful.
15:43So all those SQL objects, if you can find a way to make them available to your agent, are very, very powerful in terms of adding context and, you know, making less terms on answering questions. Awesome. All right. So now we're going to talk about, oh, let's summarize. So we got three contenders, English, SemanticModel, and SQL.
16:05And our take is let your context file mostly be English. It's accurate, it's easy to maintain, it's relatively fast. And if you care a lot about performance, you've got some long running queries, feel free to add some sample SQL. If you've got a semantic model that's mostly not SQL, consider translating it to some combination of English and SQL
16:28instead of using your raw semantic model as context. Okay. So now we know some best practices for writing context. I want to shift to talk about how we actually manage context files as an organization. Context files can be pretty long. So you might be wondering, well, shoot, like, do I have to write 20 pages of context before I can ask an LLM a question on my data? What do you think, Jacob?
16:53Do I need to write a lot of context before I can start playing with LLMs? Yeah, I mean, this is a great question. I think, like, you know, I am definitely not patient. I just start firing. And thankfully, you know, with the right tooling, I think we can just start doing it, right? So let's see where this takes us.
17:15Well, I'd say, yeah, you can, Jacob, in particular, you can. The extent to which you can get away with writing no context really depends on the makeup of your organization. So when we at Virgil talk to customers about starting to use LLMs for data analysis, we say, look, you've got two kinds of users.
17:36Your explorers are your data native analysts. They are domain experts. They feel comfortable reading SQL. They expect LLMs to make mistakes, and they're eager to use them to move faster because they feel comfortable auditing their results. Your pedestrians are the folks that don't want to audit results.
18:00They just want to ask a question, get an answer, and be able to trust that answer. So the amount of work you need to do to make LLMs safe really depends on your explorer and pedestrian makeup. You're probably in one of three buckets. Some of you in this call might be a solo explorer. So you're just starting to play with LLMs for data analysis,
18:25and you're the only person at your company that's doing that. This is a really fun place to be. If this is you, you don't need a lot of context management. It's never too early to start thinking about your schema naming conventions, as Jacob mentioned. And I would encourage you to start with a blank context file and begin gradually adding to it when you see things go wrong. Now, some of you might be in the second bucket,
18:50which is a team of explorers. And if this is the category you're in, it's time to start thinking about how to share context with other people in your organization. There's a lot of ways to do this. If you're using Claude, you can upgrade to a plan that allows you to leverage shared skills. We see teams using GitHub to sync local context files across computers.
19:15You could also use a tool like MotherDuck or Virgil that supports shared context. If you have shared context, it's probably time to have a lightweight conversation about who's responsible for updating that context file. Is it just you? Is it everyone on the team? Do you need some sort of PR or review process? And while you don't need tests at this stage,
19:38it's a good time to start thinking about and playing with tests so you get a sense of whether or not your context file is doing what you want it to do. Jacob, I feel like MotherDuck is basically a large team of explorers. What have you learned about managing this stage? Yeah, it works. I agree with you.
20:00at. But I think what we have observed recently, you know, in the last, let's see, we launched our remote MCP, which expanded the internal TAM of users who could use our data warehouse. We launched that in December, middle of December. And that turns you, for better or for worse,
20:21from a team of explorers to a bunch of, I can't remember what the other side of the metaphor is, but now you have a bunch of people who, like, yeah, pedestrian, like, you have pedestrians who, like, don't know, it's like the movie Elf, you know, where he's, like, running through New York City. He's like, the yellow ones don't stop, you know. That's what's happening with your other users who are not, who have not explored before. They're just like, oh, like, you know, I just,
20:44like, I just, like, quadrupled revenue. And it turns out there's something called a join that, like, was wrong. It's like, yeah, don't, don't do that, you know. So I think, like, you know, mostly explorers, but it has definitely shifted massively to more people, like, trying to use it with much less context about what the database is and, frankly, about what the data is, but a deeper
21:04context on, like, what the domain is and the problem they're trying to solve. So, yes, all of this, all this stuff lines up to what I've been, you know, what we've been learning too. But we've been, AI, for better or worse, accelerates us into, like, oh, crap, this is, like, a first level problem we need to solve. Okay. So let's talk about that use case, which is you're responsible
21:27for supporting both explorers and pedestrians. This is both an exciting and a terrifying space to be in, right? We want to avoid this CFO getting the wrong revenue number from Claude and putting that in order for it. Yes. So at this point, you have a couple more responsibilities. The first is
21:47tests become mandatory. You need some empirical way of validating that your data agents are producing correct answers all the time. You also need granular access control for that context file. You want explorers to be able to update it, and you need pedestrians to be able to
22:06reference it, but you probably don't want them updating it. The last invitation I'll leave here is that context files at this stage can quickly become really onerous to manage. So consider investing in a system that automatically curates context. And if this is the bucket you're in, I would love to talk to you. Okay. That's it. So we talked about some best practices for what
22:31context should look like. We talked about how to manage context as your organization grows. If you're interested in leveraging test cases for your context or automated context curation, please hit me up at this link. If you've got questions for me, here's my email. If you've got questions for Jacob, here's his email. We're now going to do a little Q&A. All right. We've
22:56got some great questions in the chat. Peter, I'm going to pull them up and throw these kind of. . . Let's see. We'll start in order here. We've got a great question here from Jack. You'll see it on the screen as well. Why do you all think SQL performs better than semantic models if semantic models are basically just SQL wrapped in some structured data? Okay. Well, semantic models have
23:20a bunch of different formats. So if your semantic model looks a lot like SQL, it's probably going to compare. . . It's probably going to perform similar to the way SQL does. I would say the best practice here is always going to be test things. It could be that your dataset and your semantic model work really well together and you don't need to make any changes to your context.
23:43So just run some emails and see if that's true for you. Yes, I would agree with that. Let's pull this one off. Let's see what else we got here. All right. Great question here. Let's see. So does it mean that it might be better to have a few
24:03shot examples plus English together? Yes. Short answer, yes. I'll say that there's a couple of caveats here. One, you need to be careful that anytime you are duplicating information, you just need to have a way of making sure that any update applied to your English also gets
24:25applied to any place that information is. And the second caveat is that the extent to which SQL is useful really depends on how close the SQL is to the questions that people are asking. So if you have a lot of questions that tend to be very similar, sample queries can give you a lot
24:46of value. If you've got a really disparate set of questions that people are asking your agents, you're going to get minimal value from SQL-based context. Yeah, that's right. That's totally right. I think there's also. . . Well, there's some good questions here, but there's one that really ties this in from Stephanie here. And I think, what have you found useful in ensuring the feedback
25:10loop is close with pedestrian AI usage? This is a really, really good question. And I think, Peter, if you don't mind, I would love to just get a quick thought on how this works. And you might. . . I mean, you kind of buried this a little bit through the slides, but you were showing the Virgil UI too, which I think has a perspective on this.
25:31What have I found useful in ensuring that the feedback loop is closed with pedestrian AI usage? So I think you've got three tools to help you here. The first is you want to unleash explorers on your dataset before inviting pedestrians to it. Because what those explorers
25:52are doing is they're writing context and they're creating test cases, and they're ensuring that they're sort of building the infrastructure that lets pedestrians be safe. The second thing that you want to leverage is you should absolutely be consistently looking at what questions
26:11people are asking your data. Ideally, the questions that people are asking are covered by your test cases, but you also want some mechanism to alert you when people are asking questions that are outside the bounds of testing so that you can add test coverage or update your
26:30context. All right. Good answer. Yes, you do need to think about the loop. All right. Here's a good one. This is a good question. I love this one. Is this condition on the quality of your SQL code base? Well-defined columns, comments, structured modeling. Do we think this will hold
26:48up in a messy SQL code base as well? Love this question. I feel like a Jacob question. Jacob, do you want to take this one? Yeah, sure. I can take this question. I mean, here's I think the thing that I've noticed as I've been working on this stuff is the answer is no, but also
27:10what I found is everything works badly in this case. It's not like, oh, we have this one golden semantic model across this really messy database, then it'll solve all of our problems. I've definitely seen as someone who in the mid-2010s had a slide that said if we could
27:33just have one source of truth and then repeatedly funded teams for millions of dollars to build and solve that problem, ultimately, if it's hard for LLMs, it's hard for humans. And the other side, if it's hard for humans, it's hard for LLMs. I will say the big challenge with applying something like, let's just say a semantic layer or it could be views or whatever, just some sort of
27:54programmatic interface on top of a bad foundation is you end up doing insane contortions to maintain it. And it becomes really, really hard and painful to keep running. And part of the challenge, I think, with this work in general is how do you keep it up to date? How do you keep it live?
28:16And yes, you can solve with a very complex semantic model, like what was revenue in Q1 of 2025? And then the rules change for revenue and there's new tables added and you have to create a new metric for the next quarter of revenue. I've been in companies where that happens and it's just like you can't, it's impossible to scale. It's super painful.
28:38Everyone's waiting around and you got to figure out a better way to loop on this stuff. And so that's kind of my answer. Yes, you will probably get short-term better results with something much more structured, but I would never advocate for fixing something that way.
29:01Okay, next question here. Let's see, going down. Why not both? Here we go, from Dennis. I love this question. Why not both? Markdown with English text and SQL examples that work as knowledge base for human and agent both. Yeah, do that. Sorry if that wasn't clear. I recommend doing that. That's what our context files look like. The context files we build contain a mixture
29:23of English and SQL. Amazing, yes. You should definitely do that. And I think actually you can kind of mix all three of the things that were talked about together because sometimes there's just stuff that are so complex that expressing it in English or SQL is not possible or too verbose.
29:48And I think, unfortunately, it's not like, just only do this one thing. Context is super important in how you solve these problems too. Okay, next question from Andreas. In what way would MotherDuck help manage context? Is any saved notebook in MotherDuck available as context? Can we let the explorers change the notebooks?
30:11Really good question. We are actually working on kind of a context object. It's not quite live yet. I was hoping it'd be live for this webinar. We also support comment on tables and columns, and so we've seen that actually move the needle pretty massively on internal eval sets. We will have something more concrete here that is available via the MCP,
30:35where it will work very, very similar to what Peter has cooked up at Virgil actually. But really just kind of like as you learn about your data, it writes what we call context fragments, and then you can kind of promote those up the hierarchy or share them. By default,
30:56they are personal just for you and kind of low quality from like a flagging perspective. And so there's a methodology in place that we're working on to allow you to promote that. Of course, why this takes a long time to build, and I think Peter can speak to this too, which is like as you build this product, you have to also run evals. That is fairly slow process because it is non-trivial
31:18to do it on like what real data looks like, not just benchmark tasks. So I think like that would be, that's how we're thinking about it right now. We don't have it available completely yet. I would say that like, you know, the best context, and I don't think I mentioned this earlier, the best context is actually table names and column names. Sorry, everyone. Start there. And that
31:43means you maybe want to have longer names than you've had in the past, right? Because your LLMs are writing SQL instead of humans. So you can do that already. A lot of this stuff you can do actually on any database, not just on MotherDuck. Yeah. All right. Next question here from Steve. It seems like database specific skills with reference material of SQL plus English could be the sweet spot. What do you think, Peter? Yeah, I think we just, we just covered this.
32:08Okay. Amazing. I think so too. All right, here we go. From YouTube. As someone working in a non-English company, I have mixed feelings about the argument that plain English is better than the semantic models. I would. Yeah. And you know, we haven't, I haven't done any testing on
32:26like natural language that's not English. So yeah. Do like a find and replace through this presentation and where I said English, consider me having said natural language. I would say try natural language as your first context and see how that goes. Yeah, yeah, yeah, yeah.
32:48I think I, I think I agree. I mean, obviously like I think, you know, because most of the, the bigger labs that are doing this training are training and mostly, you know, are like starting in English. I'm sure they're, they have a lot of other contexts from other languages too. It's probably unfairly tips of scale that way, but you know, I would suspect that
33:07many other languages work too. What about code as context? Sometimes data pipelines iterate quickly and data shape changes. AI teams were going to unstructured data sets. What do you think? Yeah, sure. Like go ahead and throw your dbt files into context. I, I do want to caution you against this approach of like throwing everything into context and see if it works. I really
33:32recommend starting with no context at all, seeing what breaks and adding context incrementally in order to get tests to pass. If you just start by throwing everything into the pot, you get a big old mess and it can be really hard to determine what's actually being referenced and what's just noise.
33:54Yeah, that's a good, good, good question. Yeah, I agree. I will also add like, I think I've seen teams do what I've seen teams do that doesn't work, that, that seems like it would work is to be like, okay, have all this context. I'm just going to give it to Claude and be like, structure this as context and put it in, you know, whatever my context layer system
34:18is, right. Whether it's in a database or whatever, without kind of like actually, you know, kind of like trying to do like a one-shot prompt on that, your results will be bad. Like, I think like really a lot of this stuff is like, you need to develop intuition on this or, you know, you use someone
34:37else's kind of blessed workflow. It is really hard to get one shot, really good context. A lot of it requires you to like, kind of carefully consider what the LLM knows and doesn't know about your business, which also determines like, depending on the industry, it may know more or less. If you're in a niche industry, you're going to need to provide more context. If you're in a,
34:57you know, a well understood industry, less context. And it's about mapping, you know, concepts together. Yeah, I would say it does help for sure, but like figuring out where the human, like a human, what I found is basically like human in the loop improvement of context is like five or 10x better than LLM improvement of context without a human in the loop.
35:22Even with like the top frontier models, that's just because they don't know what they don't know. All right, great question here from YouTube. What set of architecture lightweight would you recommend for organizations who do not want to expose their data to LLMs? I'll start with this question. Yeah, you probably need to, this approach probably doesn't work
35:43for that. I don't know if I have a recommendation because I am LLM-filled and I don't know, Peter, if you have anything, any thoughts on this? I'm also completely cloud-filled, sorry. Yeah, okay. Sorry, no good answers, but I'll take it. Okay, let's see here. Great question from Dennis. Imagine the context is 100 million articles. That will never fit in the context of an LLM. Need a solution with real-time search inside this external context to feed to LLM plus
36:06DuckDB. Yes. Wait, I want to, can I, I need you to find, so I want to read it what I said earlier, which is please start from a blank context file. I think Dennis is rebuilding Google from first principles, which is totally fine. And we can practically do it for a very, very expensive way
36:29in a very, very expensive way with LLMs today. But like, yeah, I like, yeah, that's your problem. I'm actually curious to learn more, but anyways, go ahead. Go ahead, Peter. Sorry. Yeah, I want to know where Dennis works. Dennis, it's unlikely that an LLM needs access to 100 million articles in order to correctly query your data warehouse.
36:52Yeah. I mean, I guess like maybe this is like a scope matching problem. Like I could definitely see this being a problem space broadly in data, but like probably not a scenario where you use a data warehouse. I could be wrong though. You know, obviously feel free to ping me, Dennis. I'd love to chat more. All right. From Jacques. Regarding ensuring test coverage, do you track all question answers and evaluation metrics like consistency, accuracy, steps, et cetera?
37:18Do you want to start with that one, Peter? Do you want me to? Yeah, please. Yeah. So the tests are pretty expensive, right? Like tests involve running often complicated queries many, many times, and you don't want to paralyze every single question that everyone asks your data agent because you'll quickly bankrupt yourself. So our approach at Virgil- OpenAI. Yeah, exactly.
37:43But you can if you're a lab. Everyone else- You can if you're, yeah. Our approach at Virgil is we make it really easy for you to create tests. So, you know, when you just did like a little question answer pair, we'll say like, hey, click this button to make this a test. But we avoid treating everything like a test case
38:06because we would bankrupt ourselves. Yeah. I would also say that like what I've seen is that generally questions kind of can be categorized. And what is important is your eval set should represent categories,
38:29not specific questions. And so if you can do that, then you can build a generalizable eval. And I found that works very effectively. In fact, if you- I have some research I need to publish around this that I just haven't gotten to yet around using DABSTEP in this way where I'm training on a set of, I think, 30 questions and answering a set of 450 from that train set of 30 with,
38:51you know, 90 percent plus accuracy. So you can generalize them. You don't need a complete set. The other thing is like one thing I hear about evals is like, well, the eval environment never matches the actual question asker environment. And I would just say like in that case, like build your application so that it can be a better- like better represented by evals, right?
39:15Like maybe that's a little bit of a chicken and egg problem for some folks. But like, you know, we can build some pretty cool stuff that lets us think about how evals can make each step of our process better. Okay. Let's see. Next one here. What is- okay. I'm gonna have to start
39:36picking some here. Let's see. All right. There's some ontology questions which I'm not going to take. Not because I don't think they're interesting, because I don't think I have a lot to say. Let's see here. All right. This is a great question from Joran. What would be a good starting point to start learning to set this up correctly? Are there any setup sample data sets?
40:00yeah. The unfortunate truth about sample datasets is they're designed to be user-friendly, which often means that LLMs can answer questions about them really accurately without needing any additional context. So if you were to download the, what's the, like, Waffle Shop dataset, Jacob? Jaffle Shop. Jaffle Shop.
40:24Yeah, you know, like, like, the Claude can answer any questions you throw at it about Jaffle Shops without you needing to do any context work at all. Yeah, I would say, like, to get value out of this, you should really be starting by looking at your own production dataset, but you could start very simply. You could start with a Claude skill. You could start with a markdown file on your computer,
40:49or you could use a tool like Virgil or MotherDuck. Both are pretty easy to set up. Yeah, that's right. Yeah, yeah. I'll just plus one that. All right. I'm going to do two more questions here, and we'll be out of time. So here's the first one. Great question here from Phil. We're not going to have 100% agent accuracy in the near term. Agree. Have you seen workable,
41:12efficient, elegant feedback loops where productions can get help double-checking results? Peter, why don't you start? Yeah. Your pedestrians, the people who double-check results, do not need to be SQL fluent. They need to be able to derive, they need to be able to create canonical answers to questions. So maybe they have access to another BI tool that they trust, or they have access to
41:33a source of truth, and they're able to say, hey, that's not right. Revenue was $12 million last year. And I encourage you to invite people like this to give feedback to data agents. If you do that in the context of Virgil, or in the context of what MotherDuck is building, you can leverage these people to create context without asking them to audit SQL.
41:59I would agree. I do think that also what we found on the MotherDuck side is we produce a set of dashboards, right, like any normal company. And giving users the ability to, or what we found is
42:20that pedestrian-type users can have the dashboard on one tab and the LLM on the other tab, and they can do a little bit of work themselves, too, to basically be like, oh, yeah, good. I sliced and diced this revenue number by X dimension that's not in my BI tool. Let me make sure that the let me make sure that the number matches, like the high-level number matches, stuff like that.
42:42Obviously, there's still risk, right? But let's be clear, there was always risk with humans. I used to work in accounting. I can't tell you the number of times that I personally discovered multimillion-dollar issues in financial statements, okay? And this was already reviewed many, many
42:58times by a human. I would say that maybe I'm a little bit dulled to the problem, but I guess these problems already happen, right? And not because of malice, right? Just because these
43:17problems are hard. And so let's find a way to walk the line where we can provide high-quality answers, right? And use LLM to the best of our abilities, but also balance on like, hey, sometimes there's going to be mistakes, and how do you allow for that type of environment,
43:38right? It's very tricky. All right, last question here. It's kind of a longer one. Let me read it real quick from Anshuk, which I think is a great question. Many times it's been seen, it becomes very tricky what content you put as much as natural language SQL use case increases, regression increases, and you keep changing the context. You're fine-tuning some models per domain related to SQL. Is it more useful? So basically, I think the question here, I'm going to zoom out slightly,
44:00is like, when should you fine-tune a model for your domain? Which I think is more of an AI question and less of a context question. I think to me, this becomes an ROI question. Yeah. I really think. . . So let me tell you a story, and then we'll wrap this up, which is
44:24at MotherDuck, we trained a natural language SQL model right when I started, so almost two years ago. And it performed the best on our benchmark for like a three-month period. And then a GPT for 3. 5 or something came out, and immediately it was the better model. And then the next model came out, and that was the better model. And then the next model came out, and it was the better model.
44:44So what we have found is that, at least for natural language SQL at this moment, each model supersedes the capabilities of fine-tuning previous generation models. And so, yes, maybe if you're big enough scale, you can do that where you're training models at the same cadence as the big labs. But I think at this very moment, we have not hit that peak.
45:06I think there's some use cases where maybe you could make things small enough, but I think really, if you're making them small enough, I would almost say, can you make it just straight-up deterministic and not use an LLM at all? So anyways, that's my perspective. I don't know if you want to add anything to that, Peter. Don't do it. Don't train your own SQL predictive model.
45:28Yeah. Our results have been, hey, it's possible, but there's better ways to probably spend your tokens at this moment, in my opinion. All right. Thank you, everyone. Peter, thank you so much for being an awesome guest and leading us through the really cool research you're doing and the product you're building over at Virgil. We'll be sending out everything as a follow-up,
45:51which Jerel just noted. Thank you, Jerel, for throwing that out. We really appreciate it. Feel free to ping either us, either me or Peter on LinkedIn, Twitter, wherever you're socialing. Send us emails. We're happy to chat. Thank you, everybody. Thanks, all. All right. Let's wrap it up. I'm hitting the button.
