0:01Hello, everybody, and welcome to another show and another time to talk about data and DAX, of course, as always. And today, I have the pleasure to have Josh with us, which we're gonna talk all about his use case, how he actually moved from, you know, a classic AWS stack.
0:25If you're familiar with AWS, we're gonna go over the old way and the new way. And just in general, his honest experience, you know, the good and the bad. We are so happy to hear the bads. There is always room for improvement. Nobody's perfect, but just in general, why this switch in terms of infrastructure
0:47and what's the benefits of it around to, towards MotherDuck specifically here. Josh, welcome. Thank you, Medea. So can you quickly introduce yourself and where are you joining from? We have people from New Orleans. We have one of our dev rel in New Orleans at the moment. Awesome, yeah. So I'm Colin from Houston.
1:12I'm Josh, I'm the co-founder of PriceMedic. And it's a company we started five years ago in big data. And essentially, we have a data set that can tell you the price of healthcare across the entire United States. And you know, it's funny, in the US market,
1:32you would expect a lot of procedures and doctor's visits and MRI scans to cost the same, kind of in the same area, but it really isn't the case. And the US makes it a lot more of a difficult system to traverse as a patient or even as an employer, things like that. And so we have data that answers all of these questions.
1:57Okay, so like what's your major, is it really customer, typically, just to understand? Yeah, so our kind of avatar of a customer typically is either a healthcare provider group that is pricing their procedures in the market, or an employer who has employees that use healthcare and they want to make sure that their employees
2:22are getting a good deal on healthcare, right? Okay, I see. And what's your background specifically? And like, what's your day-to-day job looks like? Yeah, yeah. I mean, that's a difficult question because our day-to-day job, everybody's changing these days with AI, but yeah. Oh yeah, absolutely. No, I started with big data out of Johns Hopkins University.
2:47So this is actually a startup that my best friend and I founded out of school. And in school, we were studying healthcare and getting ready for med school. And this is, I was also a computer science major, and this was something that fell kind of naturally in our laps, healthcare data set
3:10that can help patients price shop healthcare. And after I heard that, we hit the ground running. So. Okay. And you talk big data. Can you a bit, like, can we come back to just the definition in these days? Big data, yeah. What is big data? Because it's really interesting.
3:34There is, you know, a world premise, if you haven't heard of, you know, the big data is dead from Jordan Degani. Disclaimer, it's my boss, but you can agree to disagree. I'll put the link right there, which basically foreclaims that not everybody has big data
4:01and that the hardware of today has, you know, much more power in terms of hardware than we used to have. So what we call big data, you know, 10 years ago with on-premise cluster and so on is actually can run, you know, in your phone now with like 128 gigabytes of RAM. That's what we used to have on server side.
4:23And so now it's like, okay, what is actually big data? Gigabytes, 100 gigabytes, it's not big for, you know, a laptop can crunch that quickly. Yeah. But you do mention big data. What's your definition? Yeah. I think on that just because the parent is a series, of course, the definition of, it's not just about the volume, it's the 3V in big data.
4:47There's a velocity, right? Oh yeah, exactly. Volume and variety, but yeah. A lot of that question is how you slice and dice the data, right? Yes, exactly. But for us, if you want to kind of envision the size of our data, there are tens of thousands of procedures, thousands of insurance companies that we have data for, and then millions of healthcare providers
5:12that are administering all of these procedures. So if you multiply those all together, we're getting more than trillions of rows. Our data lake size is a petabyte, over a petabyte, and it is constantly being filled every single month with new data. So it definitely is not something you can have
5:35on your laptop querying. And the most interesting part is, how can you slice it so that people can get a portion of it with very, very condensed value? Because nobody needs that full petabyte to answer their question, right? Yeah. Yeah, so there is actually a question
5:58I can ask the audience is, what's your total, I would say, size of data? And usually what I ask is, and basically we have 10 to 100 gigabytes, I'm writing right now, and then one terabytes,
6:22and then more, one petabyte. But one petabyte is, I'll put the poll in a couple of seconds, but you can also ask yourself in the audience is, what's my total footprint in terms of size? And then what's I am actually using it daily? So it's what you mentioned, like how do you slice it, right?
6:44Because you never access all the data at once every time, right? And so, but one petabyte is really much above the average. Now, the question interesting is, how many people do you need to manage a petabyte data stack? Yeah, so we're a pretty lean company, right?
7:08We're four people, three engineers, and in order to fill a petabyte, let me tell you, it takes a lot of automation. And so it also takes some pretty heavy data sources. So each of our data sources output tens of thousands of terabyte sized files every single month that we have to ingest, right?
7:32But we do it through automation. And luckily, we've got the advent of AI that can greatly assist in not only automated data pipelines but self healing automated data pipelines. And that really allows us to fill a lake that is a petabyte. And then we can go from there and maintaining the files underneath it,
7:55the archiving strategies and all of that fun stuff. Yeah. But we're getting to do it with four people, which is crazy. Do you think it's even much easier right now or it makes sense to have such a lean team? Or I'm just curious with the recent tooling, team have been starting to be more lean,
8:20but I'm just curious, what's your thoughts on scaling data team, link it to data size? Yeah. Yeah. And really it comes down to, I mean, do we need a bigger team, right? To provide value to our customers. And at this point in time, our data sources
8:43and the amount of data that we can ingest and utilize is fairly constant. And so if we're able to get that pretty much full utilization from a lean team, it's going to, there's no reason to grow the team, right? And you could keep it automated and that is the best.
9:05And it's really, are your team members spending all day working on the data stack or can they spend some time working on product as well? And the actual utilization of the data, getting the data and having the link there is legitimately the first step. Yeah. All the downstream applications of it and extracting the value from the data is where you want your team to be spending most of their time.
9:28And so it's just that basic question of can your team allocate their time adequately to getting the insights, right? Yeah, and this is actually a good point that I think people just in general, data teams are going to move towards that
9:51in a sense that infrastructure is going to be solved for them like in the past you would have, depending on the size of your cluster and the size of your data, you will scale your data team mostly accordingly. I mean, at a point a certain time it's okay, but like you need just different kind of tech stack when you have such a large data set.
10:14And so I think it's changing now where I think those people, instead of like just getting the data there and clean and with the pipeline, we'll focus more on the products and the insights. I don't know, what's your take on that? No, exactly that, right? And really the value of data is really constituted by what you can extract from it.
10:39It's not the data itself, right? And so that is where most of the value lies. And if you can spend a lot less time maintaining that stack, it's going to be a lot more valuable to your customers at the end of the day, right? Because they're not seeing all of that infrastructure and that overhead on their value side.
11:01And so that is something you really want to optimize and make sure that you can minimize resource costs there. Yeah, exactly. So we do have a couple of, so most of the people are say, if I look at the chat, because everybody, some people commented, so it's not in the pool results, but you have 10 hundred gigabytes,
11:2310 hundred gigabytes, 110 terabytes, 20, yeah, hundred. So mostly hundred to 10 terabytes is what's like, even if I go into in-person meetups and so on, that's the maximum on their footprints. Maybe they have some archive for security, but that doesn't really count.
11:47It's more like for recovery. And 10 terabytes today, it's not that much. Like if you think that like new laptops, Apple launched their M5 and the baseline series two terabytes, right? Yup. And so, and that size I think was, far from there like 15 years ago.
12:10So that changed a bit the equation on what you can do with your compute. So let's talk about the nitty gritty and get inside actually your stack and the before. Can you work a bit through like the old way, what's your data stack look like? So the biggest thing is we're natively,
12:33we started out as an AWS stack and AWS has a plethora of tools that made it pretty easy to integrate with. But at a base level, we filled our lake in S3. And so we had a parquet data lake that was mostly not transactions, but analytics. So we're ingesting these sources and for the most part,
12:58the data does not change too frequently. So parquet data lake, looking mostly to just query and do read access. And so in order to do that, we had a couple of AWS tools that we were using. The big one for kind of big queries across a petabyte was Athena, and that can traverse the entire lake. And it made it really easy
13:21for if somebody was looking for a niche portion of the data, we can go there kind of ad hoc and use a bigger query tool to query that subset of the data lake. And Athena did that. Kind of big downsides there was, you know, Athena's fairly slow. It's not something you want to query in real time
13:43and a customer is going to be kind of working with the data as it's loaded. It is something that should be- Can you just like say a bit like what's your, you know, I think giving some time expectation on the response query, because I think in analytics it can be, you know, it can really vary. It can be, if you have a large job,
14:06it can be minutes or it can be seconds. So like, what's your- Yeah, there are kind of two avenues to that question. The first one is background jobs, right? Things that the user doesn't have to interact with. Those can, usually a good performance for those for us with sub 10 minutes. We want them to get access to their data slice fairly quickly in a reasonable amount of time
14:29where they don't necessarily have to leave the session. Now, if they're actively working with the data and kind of pivoting it and trying to get some insight from it, then you want your latency to be below seven seconds for us. And that really just keeps their attention span and they're not upset with, you know, manipulating the data, right? I'm just curious, like those seven seconds,
14:52is that something you measured or based on previous research? I think it sounds about right for me. Yeah. But I mean, if you have every time seven seconds, that might be, you know, a bit frustrating. Like, I don't know, what's your thoughts on this, like kind of like interaction time? Because I'm challenging this because, you know, a lot of people have been asking for real time dashboard, right?
15:15And it's always a question, how many decisions are you going to be able to make? Exactly, right? Yeah, so on- And that seven seconds really is when you go freshly and freshly enter the app and you get to a data hydrated screen. Okay. That is your initial population. Initial load, like the cold start.
15:39Yep, the cold start. Under seven seconds from there, you really hope to be sub-second in every query. You want them to be able to filter and sort and, you know, have near real time response. Yeah. Right? And anything, yeah, anything above that's going to make your app feel pretty laggy, right? Yeah, yeah. But the- Oh, Redshift, yeah.
16:02So initially Redshift was to try to answer those needs, so sub-seconds app. And you mentioned here in the design, we have 10 plus minutes scan. Yeah, exactly. And can you tell a bit about like usage, like front-facing tool that was consuming this data from there? Yeah, so we had a couple of usage patterns.
16:24We have a DuckDB WASM kind of data explorer that users previously would have to go through our Redshift variant of it to explore just a raw table and kind of export it and look at the underlying data. But our biggest use case was embedded BI. And so AWS QuickSight was a natural shoe
16:49into this legacy stack that we were using that users would get a slice of data provision, the slice that had the most value to them. And we'd load it into Redshift and QuickSight had this thing called AWS Spice, which was pretty much an in-memory version or in-memory database that we can use for low latency.
17:11But embedded BI and kind of a data explorer tool was really how they were interacting with it. I see. Okay. So yeah, so that's like pretty standard stack. I think one more thing I just want to tackle with you is that were you using any table formats? Are you, why not?
17:33And you know, already in this stack before, because I think it's, you mentioned Parquet, is it vanilla Parquet or are you using any table format? Yeah, and the biggest thing is we didn't have any transactions, right? And so we could use, we didn't need the ACID compliance of a table format. And so Parquet, when you sort Parquet correctly,
17:58you can really get some great performance out of it. And so with Athena, everything was in Parquet. Redshift, we did have some native table formats there. And Spice itself was enabled native table format in where you use your various queries to hydrate those dashboards or those datasets.
18:20And all of our dashboards lived off of those Spice datasets. And that's how we were able to achieve near real-time query latencies. But the biggest problem there is you've got to fill those Spice table formats, you've got to fill those Redshift tables, and that all takes time, right? You have to pull it from the Parquet data. Okay, but this is also interesting
18:43that you haven't got the need for a table format. I think it's interesting to push back that everybody's all the fuzz around Iceberg, Delta Lake. And I'm curious in the audience, if you have already an implementation of Iceberg or Delta Lake, are you happy with it or do you regret it? And where are you standing right now?
19:09Are you investigating? We have also, there is also DuckLake, which is a specification from DuckDB Labs, which is not tied to only DuckDB. But it's also interesting to see that for your use case, you mentioned that your data doesn't change that much, right? Yeah, and it's bespoke to our use case because all we have to do is we're not changing row by row
19:32at any point in time. We're just loading and unloading partitions. And that is very easy to do with Parquet. And you don't have to worry about any of that ACID transactions. And when you look at all the table formats, that's the big plus you're getting from them. And we didn't need that, right? Yeah, yeah, no, super great. I think it's a great use case to show that you can go simple.
19:57And one thing I learned over the years as a data engineer that the biggest challenge as you grow your infrastructure and just your use case is just keeping the things simple. So the latest tech that provides the features is not always there and it doesn't prevent you to move at some point if you need it to a table format. Exactly.
20:19But let's look. . . let's talk about hypertension. Like do you want to get a few words on what is it and what challenge you had around this and yeah. Yeah, absolutely. And kind of to segue from our legacy structure into our MotherDuck structure and kind of the breaking points
20:39and bottlenecks we were hitting. Well, the biggest thing is Athena and Redshift both need dedicated clusters that we could not scale reasonably for us. It either required a lot of. . . you know, there's serverless Redshift, but that had a lot of cold start. And then there's the provisioned Redshift, but that was very, very expensive. And we had these
21:01compute clusters that, you know, you'd have to scale accordingly and ended up being very, very, you know, not friendly on the AWS bill. And then Athena, you can either use their serverless architecture, which has a single compute cluster that doesn't scale too well. It essentially only
21:19allowed 20 concurrent queries for us until we provisioned a new cluster of our own. And that is also very expensive and it didn't allow us to account for users that needed more workloads, you know, peak hours, things like that. Very, very difficult for us to scale. And so as we started getting more and more customers and downstream users of our data,
21:43we ended up looking for a solution that allowed us to be a lot more malleable and responsive to the workloads and the usage patterns of our customer base that didn't really incur these massive downtime or uptime charges that we would have had with Redshift and Athena and
22:05provisioning our own cluster. And so MotherDuck's hyper-tenancy, you know, fell right into place with this. Every user gets their own provision set of resources and we can manage each user based on how they query our data. And the best part, it's serverless, right? So we're not paying
22:24for that extra uptime and those hours. One of the most interesting things that you'll see when you go into MotherDuck is, you know, every database system, every data system is going to make some some choices in how they structure the, you know, the query patterns, the read patterns,
22:45the write patterns. And so, you know, with that, when we transitioned to MotherDuck, we ended up figuring out that one server or one Duckling is not actually sufficient for the uses of our users. And the reason was, you know, you can read from a data lake all day with
23:08a Duckling, but we also had instances where users were doing concurrent writes and some of those writes would take a very long time and some were very, very quick. Some were low overhead copies, things like that. And it ended up making sense for us to actually scale a single tenant or user
23:29to multiple service accounts to properly balance those writing workloads. There's also some security pluses we gained from this. And then additionally, when dives came out, we expanded this into a third service account. And really what we were looking to enable is how can we give a single user
23:52the ability to perform those sub-second queries and sub-second writes in some cases, but also be able to create big slices in the background and big writes at the same time and provision those data slices without impacting their. . . Impacting the others. I think that's really
24:12the big thing at the end is that some users that have bigger needs are okay to wait, but they shouldn't impact. And usually it's like a small portion of your user base. Yes. And so, because again, not everybody has big data. Like here it's like the provider, but like as individual, right?
24:36And so it is, you don't want the big ducks to impact all the small ducklings. Yes. But yeah, so we, I think you had. . . That's a bit the model that you have. Did you just explain?
24:56Yeah. So now let's tie it back to kind of what does our customer look like and give some context to this slide as well. So most of our customers are large organizations with tens to hundreds of users that are sharing these data slices and need access across the organization to things that are provisioned. Yeah. That makes sense.
25:21Yeah. One of the biggest things that we ran into is there's a lot of sharing overhead and a lot of intertwined data links between user databases if we are just using a native MotherDuck share pattern. And what that looks like is, let's say user A provisions a 10 terabyte slice of data
25:43that they would like shared with the rest of the organization. Well, we would have to go to user B through Z and create that share link to each of those users. And it ended up being a lot of overhead. It was kind of a trickle up for us. And that ended up being a pattern that we saw across
26:04a lot of our users and one of the most frequent operations from our end. So we had this interesting notion of what if we had an organizational account kind of overhead that linked all of the users and had a shared database that every single user had access to. Right. And one of the interesting
26:27things that you have to do there is a user that does not own a share cannot write to a share. Yeah, indeed. And so if we had user A who provisioned this 10 terabyte slice of data, how can we give it to this organizational service account that already has a database shared with all the users to redistribute that 10 terabyte slice of data? And so we came up with this
26:51promotion architecture and where you can actually do a low overhead copy of all of this data and promote that 10 terabyte slice that maybe took an hour to provision from the user service account. We can do a three second copy over in the organizational account and then put it in the
27:11database that's shared with all of the downstream consumers, the other tenants of that organization. And that allowed us to, one, not only isolate the compute of those large data slices that our users were provisioning, you can have all of the users provision their own large data slices
27:30concurrently at the same time. But it allowed us to ignore that kind of spider web of sharing that we would have had to do if that data slice originated and stayed in a user A service account. Now we have kind of a single trickle down architecture where an organization can own a
27:51slice of data, but it doesn't necessarily have to compute or do the heavy lifting in the curation of that slice of data. And it really allowed us to offload where the computation lived and how users were able to really play with our petabyte data lake. And so what is, I'm just curious because here, so the pattern enables really
28:17to, you know, have isolate compute and provide, you know, those copy, you know, as fast as possible, even if it's a large thing. What is now the thing missing if you had a wishlist, you know, the first thing you would like to have that we don't have yet? I cannot promise anything.
28:42I said this from the start of kind of our relationship with MotherDuck, but multi-rider support would be awesome. And I know there's some talks about it being in DuckLake. And it's very, very hard when you design a database to go back on one of the kind of founding decisions. We're only going to have single rider support. But in DuckLake,
29:04multi-rider support might be something cool to have. Right. But also what's interesting is that you still manage to get your use case out there with kind of like multi-rider in a sense that you have multi-user writing, right? Yeah. Yeah. But it's always like coming down to exactly what you said, how you slice and dice. And I think
29:26there is a bit misunderstood on, you know, I run DuckDB also in general in MotherDuck. It's like, oh, it's single note. There is limit in writing. It's like, yeah, but it depends on your work. And I've heard tons of customer, you know, having different ways of doing it. And it's how you,
29:47you know, distribute the thing. It's bytes, bytes, not byte-byte dense, not the TikTok, DeepSeek. DeepSeek that build their own DuckDB distributed framework and they slice it at distribution, I think at partition level basically. So each compute can only touch
30:11one partition at a time. So there you have much, much less conflict. And again, it's depending on how you people, your user are using the data. But as soon as you know what are the query pattern, there is always a solution. Sometimes it might not sound like the most, you know,
30:30I would say cleanest, but there is definitely a solution. And I'll put on that on the, again, on the multi-rider if we can make the experience even better. But yeah. I've been asking for day one, but it'll be. Yeah. You talked about dives. We haven't mentioned, I haven't mentioned about dives yet.
30:54Also, if you want to know more about hypertenancy specifically and, you know, what MotherDuck provides at that level, because here, Josh is using that a lot. We have all the information on our website and in the docs. And can you explain a bit
31:18how you've been using dive and what was like, what was you, what were you using before? Because it feels like you've been doing a big bang in your stack. But I'm curious if it's actually was this transition and when dive came into play. Yeah. And dives, you can see are kind of carved
31:37out as their own separate service account for all of our users. And, you know, this is part of the novelty of dives and, you know, they just came out and right now the interplay between existing service accounts and dives don't work too well because of naming conventions. And so we've
31:55actually provisioned separate dive service accounts to a lot of our users. But let me, I think it'll be great to set up why dives are even in this equation, right? Yeah. And so maybe just a quick recap on my side. Dive are basically data visualization that you can just vibe code it
32:19with your favorite AI agent and that are hosted on MotherDuck. So leave next to your, you know, your data. And you can also embed that because at the end of the day, they are just JavaScript codes. So they are really easy to customize. And I will also call out,
32:41maybe let me just share for a second my screen presents. I'm not sure if I'm going to be able to share because. . . So we have also a dive gallery where you can get inspired about
33:06what actually you can build. So there is even puzzles, but there is multiple example here. You have an interactive earth graph for night sky atlas with all the data and all the data are lived in MotherDuck. And you can be, you will do even some slide now at MotherDuck as a slide deck
33:29as dive. Again, they are just JavaScript codes and you can embed them. So you see on the website, they are embed them and you can embed them into any application. So that's a quick context for what dive is. So you want to tell us like where these come? Yeah. Where do dives fall in for us?
33:51Right. And so, you know, going back to the start, we are a data company, right? And, you know, the value in data comes from the value and the insights you can pull and extract from it. Right. And, you know, one of the most beautiful things about data is there are so many ways, different ways to tell the story. And when we sell our data product, when we work with our
34:16customers, the ways that we see the data, our data want to be analyzed and how users want to analyze our data are kind of limitless. Right. We have this long tail. We have some common use cases, common questions, but then there is kind of an infinitely long tail of other metrics and
34:40answers that people want to explore and dive into for their own states and get value on from our data set. And so as a data product company, we've really figured out that, you know, one of the biggest things that we need to show is our responsibility to our customers, not only in the
35:05sales process, but in the data set experience process is how quickly can they get to those insights. And for our customers previously, if they had a question or had some kind of use case
35:23that they wanted to use our data to fill, it would end up taking our team several hours to go into our BI tool, compute some metrics and really get a proof of concept on the table using our data to answer their questions. And that ended up being quite a big undertaking and definitely not
35:46something that we could have used in our sales process. When you're trying to communicate the value of the data, you want to show proof of concepts and metrics and things like that of using your own data and spending hours on prospects, developing dashboards to answer their questions was just not something that we could do with a lean and mean team, but also
36:11just a big time sink for us to do. And so Dives really helped bridge the gap there, one on the sales process and where we can connect our data lake or big petabyte data lake to MotherDuck's MCP and then Dives to posit questions that prospective customers would have
36:33had. And they can actually see those questions answered with our data pretty much in real time. And the Dives only take a couple of minutes to create. And it actually became one of our most powerful sales tools is showing how our data can answer the questions that they have
36:53and using Dives to kind of act as the glue in that scenario. And then some of those answers to the questions were so good that Dives ended up being, for some of our customers, the primary interface in which they explored our data set. And that's one of the best parts
37:13about Dives is they're natively kind of a JSX app and they can have as much depth as a JSX or React app inside of your application, and you can really, really flesh them out. And so when you have a really bespoke use case for one of your customers, for your data product, it's something that you can actually flesh out and dive into without too much manual overhead.
37:37So that really kind of ties in how Dives exists in our product, both as a customer experience and both as a very powerful sales tool. Nice. I think we need some pragmatic hands-on to kind of like just showcase what you just explained. Can you walk us through that? Yeah, absolutely.
38:00I think it would be a good time for a quick demo right now. Yes. All right. Get my screen up. Yeah. Awesome. So this is my beautiful Claude Max subscription.
38:20But one of the biggest tools we've been using for our sales process is MotherDuck, MCP, and Dives. We connect it to our data lake and then we have customers that have these questions that we want to show our data can answer. So in this case, as I mentioned, we're a
38:41healthcare company that can tell you the price of healthcare. So we've got this employer here that for all of their employees, they use the United Healthcare Choice Network. It's a very large insurance company in the United States. And we want this employer to be able to give
39:01their employees an app that allows them to price shop healthcare. The employer wants to save money on where their patients are getting care. And because the price of healthcare varies so much in the US, it's a very valid use case and can save a lot of money, millions of dollars
39:21for an employer. So we want to see if we can use our dive and our skill that we wrote to answer this question. So I've just put this question in. We've got the output from Claude. Thinking through all of this, we made a couple of skills. It asked me some questions back. And at the end
39:43of the day, with a couple of prompts and less than 10 minutes, we get a nice app that we can use to demonstrate what kind of stories our data can tell. And so this is. . . they've called it, Claude has called it Price Compass for this mother ducking cool company.
40:07And they're really trying to enable their patients to price shop healthcare. And so we can see that using our data set, if they are based in the Houston metro area, it has queried and shown that,
40:22all right, and ultrasound can cost $119 to $567 in the same area. And then down here, it shows where exactly you can get the the ultrasound and how much it costs. And really demonstrate kind of as a proof of concept to this employer that yes, our data set
40:46can absolutely be used to help your patients price shop where they're getting care using your insurance network. And really, if you put this in front of a prospect in a sales meeting that you're trying to sell your data set to this, not only it's not a hypothetical anymore,
41:08it's actually showing that your data has been used to answer this question and very much fill their need. And so this is this is one of the most common use cases that we see is customer has a question, our data set can answer it. How can we show them it can answer it without spending 10 hours, you know, developing a dashboard that, you know, for something like
41:32this could have taken a long time. So just to wrap up, if you can go back on the history of clouds also before. Yeah, yeah, it's true. So we have some queries that's being done. So that's using the model that can CP, right? Yep. And that's basically accessing your data, generating
41:53the query exactly there. And, and then after we do have creation of a dive. And so dive here is render as a preview. So it's with some with some live data. And, and this leave and it's
42:13hosted. So it's as a Claude app. But if you are on another client, you can get link and it's hosted on MotherDuck. It's just that we have support for for Claude apps. So it's directly integrated in the UI. And, and you have in your additional analytics there. But I'm curious, how do you
42:37manage because that's for sales flow? Yep. How do you make governance around this? So I'm pushing back like just the old way where you had hundreds of dashboards, right? And what is what is the source of truth? Or you're not feeling it because it's at the user level, and they mostly create
42:56one or two dive and they're happy with this? Yeah. And, and really, there's a couple of things that we get to do here. And it depends on how pivotal the dive is to the user's experience. If they are creating their own dives, our service accounts and the service account architecture
43:16actually allows them to securely create their own dives, keep them isolated, and work within the constraints that are secure. And that allows them to kind of play with their own data that they own, and use dives at their own discretion, kind of through Claude Code or Claude, however they see fit.
43:39But if a dive actually gets fully embedded into an app, it's kind of like a, let's say, oh, wow, this answers my question so well, that I would like this to be kind of a pivotal experience, something that is refined over time, but actually more of a not as interactive app,
43:58or not as editable app, but something that I come back to day after day, and it's like a KPI dashboard, for example. Yeah, those we actually use CI flow to maintain. Okay. And CI flow allows
44:16you to expose the underlying code of this dive, and kind of see it there. Yeah. As I said, it's just JavaScript, and you have access to it, to the SQL interface. Yep. And then you can use CI CD to maintain those, and have those be kind of governed portions of your product. So when you
44:38have a bespoke dive, you can embed it, and maintain it through CI. Yeah. Yeah. Okay. No, but that's a great use case. We've also used like for sales flow at MotherDuck also to just showcase. I think that works well. I think it's a new, the blocking point has been for a long time
45:04how you shape your data, but that's your products, that's what you're selling, right? And then what it's done is like, how do you make decision based on this data? And usually the answer is through data visualization. You can put a basic table with some KPI, but there's just a bunch of stuff that
45:24you need to explore. And I think a dive has been really nice to see how you can put custom things and custom highlights depending on your usage. How do you see the usage?
45:44We are getting towards the end. I'm just curious. How do you see the usage going forwards in terms of just data visualization and dive? And what's the next thing that you would look forward to for your user to even have faster answer? And when I mean faster answer, it's not just about
46:06the infrastructure. It's just like the whole experience end to end. Yeah. We call it time to insight. Yes, exactly. Time to insight is the right word. Yeah. And really right now, most of our dives flow is governed by us. We don't expose them raw access to our MotherDuck
46:27databases or anything like that. And that we would want in the future, and this is kind of our Q2, Q3 launch, is a governed MCP for Price Medic, where you can create dives that are natively embedded into our UI and customer experience. And that will allow us to really
46:51reduce that time to insight and giving a secure approach for our customers to interact with their data through dives and really reduce the time to insight. Right. Yeah. So it's better integration, basically, towards your specific stack and specific tooling so that there is much less
47:12friction. And I think it's also worth to mention that those framework are still relatively new, right? And MCP is standard, right? But it does behave differently depending on the client sometimes, because the model behind is different, right? Even if the tooling of the MCP is
47:31deterministic. And even for MCP apps, for example, as far as I know as today, it's only within Claude because Anthropic is kind of like leading the standards. Yeah. Just all that to say that the distribution, the standardization and what you call out to time to insights,
47:56it's still like an environment which is evolving and maturing, I think. So I'm looking forward to have much more standards so that we can build those things and not changing every four or five months because there is a new standard framework there. We have a couple of questions that we can
48:17tackle. How do you think about PII when using MotherDuck and the MCP? Yeah. The biggest thing is, you know, for our sales team, that is easy to manage because we control all of the data inherently internally. And so it's not too big of a worry for us there as long as we're using
48:39secure practices. It's really when you give your customers access to the MCP and dives. And that is kind of at the tail end that we were just talking about. How are we going to securely give our customers access to that? And really our initial. . . They currently don't have full access to dives,
48:59and we'd love them to, but we really have to think about the security practices there and tying back into our current service account architecture. It's how can we isolate their data and then ensure that when they interact with MotherDuck MCP, they only have data that they
49:19should have access to, as well as ensuring that the whole stack is kind of governed in the connection to MotherDuck as well as to dives. And so really that is something when we give access to our customers is going to be one of the biggest questions we have to answer. And since embedded
49:42dives are so new, it's an ongoing question. Yeah. Two more questions and then we close it. If you can answer, give your thoughts quickly. Josh, do you see the data finding interacting with skills to take action? I actually don't really understand this one.
50:02I have some interpretation of it. Most of our sales flow, we actually create a set of skills that allows dives to know how to interact. Sorry. Yeah. The skills, agent skills.
50:21Yep. Agent skills. And that has by far sped up the time from the initial prompt to the creation of the dive because the agent doesn't have to re-figure out all of the things that previous agents have had to do. So skills definitely speed up the dive procurement. Yeah.
50:43The other question I had is sometimes when you have too much skills, it tends to actually slow down the biggest. I think it's always a balance on what you repackage versus just providing as a markdown skills. Final question. How do you make the decision to say
51:05MotherDuck will be sufficient for data volume in the next X years? That's a great question. One of the nice things that we do is we are always going to have an S3 data lake that we will use as our kind of sole source. And the compute engine that lives on top
51:23of that can be variable. Now, that has allowed us to swap from Athena and Redshift to MotherDuck. And for now, MotherDuck is beyond our dreams in performance. But it really is that kind of segregation of principles of where does our data live and how do we query that data?
51:44Yeah. And I think that's why the whole premise of combining a lake plus a warehouse is always working. There is no fight about one or the others. You can use both. Yeah, exactly. All right. That would be it for today. Thanks again for joining us on the show.
52:05Recording will be available. You will receive it towards emails and YouTube. And I'll see you in the next ones. And thanks again, Josh, for sharing your story. Awesome. Thank you, guys.
