0:00All right. Hey, everybody. We're super excited to have you here. I'm Jacob here at MotherDuck. I've got my partner-in-crime here, Colin, as well. Colin, say hi to our friendly audience. Hi, folks. Amazing. All right. So what we're talking about today is what it looks like to actually do cross-platform analytics
0:24using Iceberg and MotherDuck together. Lots of fun nuance and things like that kind of packed in here. Of course, we will be orchestrating a lot of this work using dbt. And that's why I'm super, super excited to have Colin here with us, who comes to MotherDuck from dbt, among other things. But he is a bit of a wizard as it relates to that stuff.
0:48So we're super pumped to take it into here. Feel free to ask questions in the comments as we go through here. We do have space reserved at the end for Q&A. But if there's good if there's good questions that are topical to what we're working at, we will fold those right in as we as we keep going here.
1:07And I think with that, I will hand it over to Colin to kick us off here, do a few slides and then jump right into a demo. Yeah. Hi, I'm Colin. Yeah. Like Jacob said, I come to MotherDuck from dbt labs. This is actually I think now we're in month two here at MotherDuck.
1:29One of the newer MotherDuckers in the thing, my kind of elevator pitch is actually that I never really meant to be software engineer to become a software engineer. I just sort of hung around the wrong people. I originally I was like a market researcher and analyst and just sort of slow walked my way over to this side of things. So I've kind of done all the jobs.
1:53I've been an analyst, I've been a data engineer and now I've been a software engineer for a while now. So today I'm going to talk about my racing to the top cross-platform text analytics on F1 radio data. So kind of jumping in. I'm going to talk a little bit about like, what is this data that I'm going to be working with?
2:14The architecture of the kind of the demo and how we're connecting these different platforms, the development lifecycle. And then we'll jump into like walking through the code, looking at how it works and then looking at some of the results. We'll do a quick look at the results and like what this kind of what you kind of end up with.
2:35And then maybe we'll do a little live debugging and we can take a look at how you can work with this in practice and make updates. So, yeah, F1 radio. So for those not familiar with Formula One racing, drivers go around the track. And while they're doing so, they talk to their crew over radio.
2:57The Formula One governing body makes these available. And the fine folks at Open F1, shout out to them, this would not be possible without it. We love free open data. They actually provide the they've cataloged the URLs to actually go and download this data.
3:19So what we have. So these look like what's the picture in front? These breaks. These are just comments that they're making back. Sometimes they're talking about strategy or tires or the weather. Lots of things that go into this. So what do we. So if we want to understand what is it, what are people actually talking about?
3:40And so we have audio data, these MP3 URLs. So we need to actually go download those, transcribe them, clean them up, process them.
3:53This is a really good fit for a distributed runtime where we can actually parallelize lots of CPU intensive tasks. But we want this to end up back in Mother Duck, right? Like we already have a bunch of our Formula One data.
4:15I'd already done this. I already have my the rest of my data is in Mother Duck. All of my reporting is in Mother Duck. It's really great to use. It's really snappy and performant. So I want to end up back in there. So what we can do is use Iceberg to connect a Spark job. Right. Which is pretty well suited for this task. Right. I want to run some ML models and do some inference. Right.
4:40I'm going to take this audio. I'm going to use OpenAI's Faster Whisper. If you've ever used that for your open source model. And then I'm going to pass that to BERT topic to do theme extraction with topic modeling. There's kind of some I guess now old school text sort of natural language processing.
5:02And then I'm going to take that data, persist it in S3 and register it in a Glue catalog. And then Glue catalog exposes an Iceberg REST endpoint, which we can attach is a new database.
5:20Is there an issue with people seeing the presentation? I think it looks good over here. Let's. Yeah, it looks good. I think it might be an issue potentially with on the StreamYard side, but not something we can solve at the moment. Why don't you why don't you just just for fun?
5:41Why don't you just let's I'm going to remove it and then why don't you reshare it? OK. Is that OK? Well, we'll see. We'll see if this solved our problem. OK. All right. Cool. Reloading the Web page fixed it. OK. Good, good, good tech support from the audience.
6:00Thank you. So what we're going to do here is we're actually going to use Mother Duck Flights, right? Our lightweight Python runtime to trigger and run our Spark job, which will write to Iceberg. Then we're going to have that orchestrating flight run DBT using the Iceberg data as its source.
6:20And then we'll use that to transform it and then we can visualize it in dives. So, yeah, like I was saying, this is a we have our our sort of flights pipeline orchestrator here. It's going to kick off the Spark job. Increment the process and write to Iceberg.
6:44DBT core can transform that in flight or serverless. Yes. Sorry. I kind of glossed over that. That is AWS is lightweight. High spark spark runtime. So you essentially provide a duck.
7:01You can provide a docker image with a spike pie spark in it and then spark and then that it will run that for you. You could but you could replace your serverless with your spark runner of choice. In this case, I went with it because it's very easy with its glue integration to then get to iceberg quickly,
7:22which is really what I wanted. But yeah, worth calling out. The EMR serverless bit is like there. Every cloud provider has some version of this. And then actually connecting iceberg to mother duck. I want to gloss over that. The is really a two step process. You need to, you know, mother duck.
7:46We need to know how to connect to AWS. So you can provide a secret. And then you call create database. This is really no different than a normal create database, except in this case, the database already exists. So what we're really saying here is, hey, there's a pointer. This is a pointer to an external database. And that database is then going to get loaded.
8:10And from then on, you can treat it like a native database, right? All the same commands will work. And you can actually even do like bidirectional rights. Actually, I set this to read only, but you can do like bidirectional rights between external catalogs.
8:30So this is another thing that kind of I wanted to highlight that kind of I didn't mean for this to become part of the talk or the presentation, but kind of fell out naturally as I wanted to have multiple agents working in parallel. I wanted to have kind of developed a SDLC.
8:53And sorry, I used some jargon there. Software development lifecycle. So I want to be able to have a staging version of all of my resources be created when I open a pull request to update my repo. So in this case, what I use is we have a tool called Blueprints,
9:13which will actually for all mother duck resources actually spin up staging versions of this tag by your branch. And I'll demo this when we kind of get to the code walkthrough. But the kind of key thing is I can actually do manual end to end testing here of my pipeline.
9:33And then oh, and then also when this gets merged to main, the deployment can happen automatically. That's right. Colin, I want to call it a quick question I think is relevant here. I'm just going to highlight it quickly. For someone new to Iceberg, can you explain what role it plays in this? Yes, I think Iceberg is a. OK, so first of all, Iceberg is two things.
9:55It is a file format and a catalog and a and a interface. So the interface here is the more important bit. So you hear this is like an iceberg catalog. And what that provides is a industry wide standard for how external data catalogs can be loaded by different compute engines.
10:19Right. So prior to many people, I think I've heard this expression, right. The separation of compute and storage. But when that happens, that was really because like that, most that storage was still a proprietary format and a proprietary interface to the actual compute engine.
10:38Now, with Iceberg REST catalogs, you can actually have the storage be completely independent from the compute engine. Right. So Spark can can interact with the same data catalog that Mother Duck can, that Databricks can, that pick your pick your modern warehouse of choice.
10:59The. So what it really is here is sort of the the handoff point between my two compute engines. Yeah. Right. So Spark is processing the data and it's going to put it in this location and declare it in a nice structure format. And then Mother Duck can discover it. Right. You notice I didn't have to call create table on the actual S3 data.
11:22And that's because Mother Duck will actually discover the tables once I call create database. Can I can I'll expand on it slightly. I think like the main thing here is that, you know, Iceberg came out of Netflix as a way to hold to hold like huge amounts of historical data.
11:43And so, like, generally, when you're using Iceberg, you're using it for really, really large data sets that are typically partitioned and you don't need to keep all of that hot. But you may want to process it, you know, in massively in parallel with something like like Spark.
12:06And so it fits in really nicely, kind of in that place in your stack. And then, you know, we get a clean interface like Colin was talking about to analyze it in Mother Duck. And then I do see I do see a follow up on this, which we'll we'll jump into quickly and then I'll let you keep going here, Colin.
12:26But I do think I was actually thinking about this. I was like, this actually Duck Lake would probably be a better fit. I think I think because we're not talking like like huge like to your point, like I think Iceberg really shines when you have like a huge data set that you need to update frequently. Yeah. Right. It's actually really good at like upsurge and stuff compared to three.
12:49And this actually probably would be a better fit for it in this case. Why didn't they use Duck Lake? I think it was just it's very easy to use Iceberg from Spark. Yeah, I think, you know, I would add to that, like, you know, I think I think part of our goal here is to get to show how easy it is to work with things that are not first party ecosystem controlled.
13:14Right. We love Duck Lake. Right. Like but also like the compatibility path between Duck Lake and Mother Duck is extremely strong and we do it all the time. So, you know, I couldn't I couldn't let Colin get off too easily and just do this in Duck Lake. Obviously, we love Duck Lake. So you could you could indeed you could indeed also do this in Duck Lake, too.
13:37I think the other thing I would say here is this is also a good example of like Iceberg because it's so widely adopted. It means you could swap Spark for your compute engine of choice. Right. All of the major platforms at this point support writing to Iceberg.
13:56So if let's say you have like a legacy, you know, you have a, you know, you have your Databricks, you have your Snowflake, you have your whatever compute engine, you can pretty easily go from there to. Yeah. Yes. Ice demo mix. Yes. That's pretty much what we're talking about here. Yeah, exactly.
14:18And honestly, one of the harder parts of this demo was finding a use case that wouldn't obviously be better served by running in Mother Duck. I say that, obviously, somewhat biased. OK. Do you want to talk about how all it could be Azure and Duck Lake? Yeah, this could be, you know, I think that is kind of the point.
14:42Spark here is an implementation detail. It is not actually like conceptually that important to this whole workflow. You could replace Spark with like a whole bunch of other things. Yep. So, OK, so we're going to pause here and jump to this.
15:01So this is the repo. It's a public repo. People can go look for it and find it. The. It's here to look at, so we've got our flights. Hey, Colin, can you zoom in a little bit? It's a little small. Oh, yeah, that's my bad. Perfect. That's great. So let's look at. Our radio topics pipeline.
15:26So already here you can see we've got our requirements on text. Our blueprint. We'll kind of showcase this later, but this is kind of important for actually how we do the like CICD here. And then our radio, our actual flight pipeline. And this is its primary job. It does really three things.
15:48And those three things are runs the Spark job. Runs DBT. Once the Spark job is finished and then checks that the data is loadable from dives. That's it. And that keeps this pretty simple to run.
16:07And then we are going to look at our. Spark job. But. Which and this is kind of this is why I think going with something like Spark is kind of fun, because you can actually upload just like a docker container,
16:30which makes it nice for doing things like tuning a whisper model, which I got a little too into for a while. Not super important for this demo, but like there are a whole bunch of things related to like F1 radio data that I learned here. I like using whisper and production that are pretty interesting, I thought.
16:54And, yeah, I mean, we could go through a lot of this stuff, but this is the thing, right? You have a lot of things like, for example, if you want to clean up the text data, you need a lot of like custom code to do this. And these are things that are like pretty well established in the like Spark ecosystem for doing this kind of data processing data.
17:16But, yeah, as you can see, a lot of like name anonymization and struct handling. And then we've got. Our DVT project, right, which is going to come after this, the some of the stuff here that I think we've done here is like, for example, the Pyspark code is going to create a bunch of topics.
17:40And so I have actually created a mapping file, which its goal is to map things in the keywords in the topics to like a display name. So this will get formatted nicely in our reporting. And, you know, we've got our very standard. I don't know, people might be pretty familiar with.
18:02DBT, but you can have your DBT project here. One of the call outs is actually that I don't run the full DBT project in my topic pipeline orchestrator. I'm actually using a selector to run just the subset of DBT models relevant to that pipeline.
18:20And then I have a different flight that runs the other ones on a weekly basis to get the latest race data. The. What's going to do? And then we have our dives data, our dives application, which I'll go look at.
18:44Right now, I could stop here. Any kind of questions on the repo structure? I don't see anything yet. Keep rolling. Cool. Oh, let me let me grab that. Can you drop the repo link just in the chat here? Thank you.
19:02That would have been smart. Sweet. People if you want to take a look at it, they can. So, OK, so what? Zoom in, zoom in, zoom in. Thank you. Thank you. Thank you. One more. There we go.
19:22Perfect. So what we get is a bunch of topics which kind of shows over time. And this is, for example, by a driver. But we can see this. What are the ones that I kind of thought was interesting? Other here is actually cutting off some data.
19:43Right. Why was 64 messages about another topic? And I want to go into that because. So why isn't that? I wanted to look at this version of it, which is a little bit more interesting because
20:08we can see weather showing up for Montreal. And when, yeah, rain and weather only does this. But a lot of the messages in Montreal were actually about weather. And if you look that up, that's actually about like that was super rainy race. And so a lot of the like thematic analysis is picking that up.
20:36Colin, do you want to talk a little bit about, you know, so for those that don't know, Dives is a way to serve React and JavaScript inside of MotherDoc. And I think one thing that I'll just quickly share that Colin did that's really interesting is using the blueprints, kind of in his CICD flow, is when he's making modifications to his dive, because it's all
21:00source controllable in code, you'll see that the name has like a weird suffix on it, right? And that suffix on the name of this dive, if you look on the left hand sidebar, is because that's his branch for a change to this dive, right? And so we can get like, we can take advantage of the fact that kind of all of this whole stack, right? So all the way from,
21:23you know, ingesting this data with Iceberg, you know, and defining our Docker image on ingestion for Spark, and then visualizing it and, you know, transforming with dbt and then visualizing with dives, this whole stack is actually one mono repo, right? And so like, that means that from a standpoint of like building the thing, right, like, if he has an issue anywhere inside this,
21:48it is the, you know, things like agents, right? Can use have all of the available, you know, context to solve a problem if there's if there's some problem, you know, kind of in this workflow, right? And I think that's like, pretty, a pretty neat thing that you can, you know, build out of this. And part of what we wanted to illustrate is, you know, all of these pieces,
22:12one mono repo, all deployed, one change, you know, goes through goes through, you know, full CI CD, kind of all the way on your entire kind of data value chain, which we think, which I think is really, really important. Yeah, and actually, it will go so far as to
22:30Oh, yeah, don't fail me now. Oh, no. That's gonna need to reauthenticate. Hold on. Here, I'll hide your screen for a second. There you go. I mean, it's fine. It's just gonna tell it that's just an Octa page change.
22:49Yeah, all right, we're back. We're back. So yeah, the what I just showed you is actually me iterating with my my cloud agent here. And blueprints will actually provide a deployment plan and actually give me the link to my drive in the PR.
23:10Oh, that's pretty cool. All right. So you actually have it. So it links in links in Yeah, so I think that's that is part of it. And like, if I was, I set it up. So like, if I was also modifying a flight, it would deploy a staging version of that flight and connect the dive to the flight. You want to explain what flights are just again, for the people at home, or flights is the very lightweight Python runtime.
23:34Where you this is what I was kind of showing with the like main. py. It's the orchestrator, that is kicking off my spark job and running my dbt and making sure the dives are up to date. It is really well suit. It's a single node runtime. So as opposed to like, say, pi spark, which is like a lot of overhead from needing to coordinate lots of nodes,
23:57flights is a single node. It's kind of if you've ever thought, why can't I, why can't somebody just run this Python code for me? That's flights. Yeah. So what, you know, is really helpful here is it gives us that, like, really lightweight glue that Claude's really good at right of working with.
24:18So, yeah, and the other thing is, because it's so lightweight, it's very easy to deploy a staging version of it. And just, you know, it's really easy to deploy a staging version of it. And just update it so that it writes things to there. So yeah, I think, and I think that's a, that's an important call out, right? Because the more that we can make this easy to do and like
24:40low risk, the more we can actually kind of like line our agents up to do most of the work, right? And move us into a place where we are reviewing, you know, reviewing agent fixes when things go wrong, instead of, instead of when things go wrong, instead of, you know, doing all the investigation ourselves,
25:01so we can do other high value stuff. And in this case, I think because it's monorepo, because you can stage it, it also means you can review the end end. So if I change this, well, I also added some like custom GitHub action stuff to do the same thing with my Spark application. So let's say I want to make a change to the whole pipeline. I can do that in one PR, test it in one, in a whole environment and ship it as one.
25:28And I think that's right. Like instead of needing to cascade, like, okay, I updated my Spark job. Now let's go update the transformation code, like update my dbt project and then go update my dive to use that. I can do that all in one change if I want to. Or if you want to get really fancy,
25:47you could use stack PRs that I think GitHub is adding support for to probably make that even more. Have you, have you, okay, I'm going to, I'm going to pause. Maybe I'm just a really bad engineer. I have not found stacks PRs work for me at all in GitHub. I don't know, have you used them yet, Colin? I used them in the past.
26:08All right. This is a skill issue on my part then. I do think it is like, no, it is a skill. I think use stack PRs is a different workflow. It's a different way of thinking. You need to think differently about like how you sequence changes. I do think it's like, I think it's also kind of like a personality of how you develop. Like, I don't think everyone's stack PRs are going to
26:31work for everybody. I do think they make it really easy for reviewers. So I think, you know, it kind of, it is one way, especially if you get clawed to use, you know, use it. It is one thing way to get around the problem of my agent generated a 2000 line change and I have no idea of knowing
26:51like sifting, reviewing this is now such a chore. So I, I think in the long run, it's probably how most people will develop. Interesting. All right. At least for major features, but yeah. That's a side opinion. Unrelated to this, OtherDoc has no position on stack PRs versus
27:11unstack PRs, obviously. Yeah. Any, like, so anything else we should, we want to cover here on like, yeah. Anything you want to cover here? No, but I think we have an interesting question that is
27:31something that you know a lot about. What would the alternative be for DBT? DBT will not work for petabytes of data. Do you agree with the premise? No. DBT doesn't care about the size of the data because DBT doesn't actually do the data transformation, right? Your warehouse is the
27:52one doing the data transformation. I think, I wonder if the, I wonder if the question from I think, I wonder if the, I wonder if the question from Joel here is like pointing towards how difficult it is to like maintain incremental, incremental updates or, you know, or build incremental models compared to like, you know, full refresh models, for example. I know, although I know there's like a lot of affordances coming out
28:16to improve that, right? Yeah. I mean, you've got micro-batch now, which will parallelize building multiple incremental partitions. The, so yeah, I guess I would need to know more before. Yeah, yeah. I think, I mean,
28:37I think before that, but like, what is the alternative to DBT in this scenario? If we just take that question. Yeah. Drop in other data to open source data to transformation frameworks, you know, SQL mesh. Sure. Lots of, you know, I think there are other tools out there. I'm not obviously as familiar with them and I have, you know, fair disclosure. I obviously have a vested
29:01interest in DBT core as a long time maintainer of that project. I think, you know, your other, and then also like, this isn't a ton of models. Like you could probably do your own custom SQL if you wanted, right? Like, yeah, yeah, totally. Just like one option is like, you can also just generate the SQL to do the incremental updates yourself. Yeah. I think, I think like the main
29:26advantage of DBT, I think for, for like complex projects is the automatic building of the DAG, right? Yes. So it's always going to execute in the right order. You can't, you can't introduce circular dependencies. You know, I think one thing that I know Colin,
29:44I saw you put a PR on DBT, DuckDB for running flights as Python model, like DBT Python models, for example. That's another way that, you know, people can do this, right? There's also like the, I think the classic way that people solve this kind of pre-DBT was like airflow and then using
30:00airflow to like link steps together. Right. So I think that is a, there's lots of ways to solve this problem. I mean, in the days before DBT, I used stored procedures, you know, like there's a, there's so many ways to think about this problem set, but ultimately the, I think the durable,
30:22the durable primitive that I see out of DBT that I would am reticent to kind of use other tools for is a hundred percent like linking together sources and references. That, that part is extremely annoying. And, you know, obviously there are other tools that do it, but having that part of it so that things always execute in the right order is great for avoiding just like classic,
30:46really stupid data quality issues. Yeah. I like, I think the, to add to that, like, I think there is also within like, in terms of materializing an object in a warehouse, DBT also just does a lot of the boilerplate stuff for you. Yeah, that's true. Right. Like when you want to go call create, when you, when you, okay. If you want to replace a table in a warehouse, you don't just call
31:10create table, right? There's actually like a sequence of steps that you need to do and you need to do them correctly. And for those of us who did DE in a pre-DBT world, you actually had to do, you had to do those steps yourself and you had to figure out how to not do it in the wrong way and not screw it up. And not accidentally drop to the table, your target table, such that there
31:33was no data, right? Right. Do it in the right transactional way, which database has a different approach. Sure. I totally agree. I think there is like a collectivization of the best way of doing that is a big thing that, you know, a framework like DBT provides. The, so yeah, I, yeah. It looks
31:58like it's Arc, it's Dia, which is from the same people as Arc browser, but it's their next one. It's more agent focused. I've been trying it out. If you like Arc, it's very similar, a little bit different, but yeah, it's called Dia. Amazing. We've got another good question in here that I'll just pull up. Do you touch on data modeling across the unstructured or
32:19structured and unstructured data? Yeah. So I think the, the thing here is that like, we're kind of taking the unstructured data, right. Which is the audio. And we're trying to like, impose a sort of structure on it by like extracting the thematic topics in the text.
32:40I would say, so there's not a lot of like data modeling coming out of that. I mean, I'm putting this in the chat, but like the high level is really, is really that we're going to make the unstructured data structured. And the way we do that is with the whisper model,
33:02right? Yeah. Yeah. So yeah, we are, yeah, we're, we're turning well, and it goes, we're turning it into then a string, right? So we've got our string, our transcribed string, hopefully. And then from there, we're deriving the topics in that string. Yep. And then we're really then joining that to the, the data that we have about each race,
33:27right? Who the drivers are, what team they're a part of, you know, we could eventually also try to do some sort of analysis, like comparing how well each driver did to the topics, like what topics are most associated with a driver winning or losing over a weekend, you know? So yeah, you're,
33:50yeah. Yeah, I think like, you know, ultimately, we want to add some structure to it. Obviously, MotherDuck is a relational database. So we want to make it, you know, ultimately fit into something we can analyze with a relational database. And so, you know, kind of that, that transformation pipeline inside of,
34:12inside of Spark is, is the way that that was done. Yeah. Cool. What else do you have to share here, Colin? Anything else you want to hit on? No, I, so we, one thing we can look at is some, we kind of did the like live debugging,
34:31looking at things. Yeah, I think, I think that's good. We can call it. Sweet. Well, we're definitely, we have time here, you know, we, we held back time here to, to take questions. So we'd love to get some questions in here while Colin and I just riff a little bit here while you get your questions into chat. You know, I think the thing that,
34:55that I didn't mention earlier, but kind of that I'll reach back to, you know, you know, part of the reason to kind of choose Iceberg as the integration point is really just to show how easy it is to tie these pieces together. And I think like really what we, what we're thinking about here is like, yes, it could be Iceberg. Yes, it could be Spark. Yes,
35:17it could be Duck Lake, but really like, it's just a stand in for like, whatever things are happening upstream of MotherDuck. It's usually pretty easy to kind of get them into your database and then use something like dbt to kind of govern the transformation to something that your end users can, can, can consume. And so, so like, as you think about what this looks like,
35:39you know, it could be any number of things that we, that are happening upstream of you, right? That maybe you have in control. Yeah. I mean, the question just came in the chat here. Sure. I'll just, I'll just throw this up. What's the main purpose of using Spark in the pipeline just for ingestion? Can we use other tools to replace Spark here? Colin, why don't you,
36:00why don't you take that? Yeah. You could probably use like Ray, Dask. You could roll your own if you wanted to. Actually. Right. Use it. Use it. Arbitrary sandbox. Yeah. Use arbitrary sandbox. You could, one of the thoughts we had, I actually had about this
36:19was that I could probably build a, a version of this on flights where I like implement where essentially I have like a controller flight that just kicks off at number of flights that do the downloads and like essentially mirrors what PySpark is doing. Right. The only actual challenge I had was like the embedding, the whisper model. I like some of that stuff. Yeah. MVP by flights. Like,
36:43yeah, like you could do that yourself. The thing that, you know, I think Spark provides is actually like the ecosystem of primitive of like is really well established. Yeah. Of just running this stuff, the node management, right. Like running lots of compute nodes of like highly customizable compute
37:04nodes is something that's very easy to get up and running in Spark. It's not necessarily the like most performant or your best option long-term, but like it's a good one. Yeah. I think, and I think like, you know, I think one thing that, that I would just add is, you know, for better or for worse, as we all become more AI-pilled,
37:24incumbents have a real advantage in how they're trained into the weights basically. And so what that means is that something like Spark that's been around for a long time, these jobs are fairly easy to just like point your claw at and be like, do the thing. That doesn't mean, again, to Colin's point, like, does that mean it is the most effective or like most, you know, dollar,
37:48you know, dollar weighted, most effective way to do this? Like maybe, maybe not. Honestly, probably not. I think there's a lot of inefficiency in Spark, but it's also, you know, an extremely well-tested primitive to Colin's point. And so that also means that AI is really good at writing it and that we can, you know, focus our, focus our brains and use our limited,
38:11you know, human capacity on things that are harder and need more attention from us. Now, again, if you have like, you know, petabytes or more that you're processing, like that may actually be worth your attention because there's so much money and time being spent there. Right. I don't, so I'm not going to say that everyone, you know, that's not the most important thing for everyone, but certainly like in general,
38:34what we're seeing, what we see is that like, you know, figuring out how to get value of the data kind of holistically is the best way for humans to add value to their pipelines. Right. And you know, doing that one, one amount of repos, really nice. A problem of like, this isn't even a big data problem, right? We're not talking actually a lot of rows. This is like 5,000 rows, 6,000 rows, right? Like it's not a ton of data. Even the audio
38:57data itself, it's not huge. But it is like a highly parallelizable problem. And I think that's like one of the things where it is just a lot easier to do that in a like multi, in a distributed compute world. And that's actually kind of why I hit upon that as like the problem here to identify
39:20is not that like you couldn't do this in a single node world. It's just a lot easier to do in the distributed world and it's going to be a little bit more performant. Yep. But to go back to the cloud thing and actually the like, why dbt, right? Like one of the advantages of using dbt with cloud is there's so much dbt out in the open in the world that cloud actually is pretty good at writing
39:44dbt models and doing modeling in dbt. And there's less for it to go wrong. Also things like the unit test feature and dbt core become valuable, right? Like the ability, like the built-in feature for a quad to be. to test and validate its work is pretty valuable. I agree.
40:07Um, let's see any more questions in here. I'm not seeing anything coming through. If you have any questions here, we'll hang out for a little for a couple more minutes and then if and take questions, but if not, we will we'll wrap this thing up. Yes, indeed, it is dinner time. Uh, and you're up right now. Enjoy your time out there, Oliver.
40:31Um, okay. It seems like we're pretty good here. Um, you know where to find us? Uh, Colin, what's the best place to find you on Twitter? LinkedIn? Where should we? Yeah, I'm not a very online person. Well, that's good because I'm extremely online. Um, which is fine. Um, if you all want to find me, I'll just
40:55drop my LinkedIn here. You can also find me on the Twitter slash X part of the internet. If you feel so inclined, username Mattson J. Uh, almost everywhere. Um, okay. Let's see here. Uh, we do have one question here. Um, from Juan about AWS glue. I'm going to just pull it up. I actually don't know a lot about this, so I'm going to defer this one to you, Colin, but ABS can
41:19be used to extract transform data. This case, you're only using it for moving question mark. AWS shoved like five products into like three or four products into glue. Uh, one of which is a PySpark runtime. Um, and you can totally do that if you want to use their very like managed version of Spark. I don't, I didn't really think about it, but you probably could do something like that.
41:43You could probably make it work here. Uh, I don't think it allows you to bake a custom Docker image. That would probably make it harder to do some of the like ML tweaking stuff I did here. Um, but yeah, so there is a compute engine in glue. Glue has its own thing. Um, but in this case, I'm using just the glue data catalog. Yeah, just the catalog is the thing that lets interoperate
42:07with mother duck. Yeah. Yeah. If you want to be clear, it's glue data catalog plus lake formation. Yeah. Which is the add on that puts the actual iceberg rest catalog on top of your glue catalog, which is then itself on top of S3. Um, and it just sounds like Russian nesting dolls. That's because it is all the way. Uh, yeah, I promise this is not
42:31sponsored by AWS is, um, compute team at all. Uh, but yeah, they're yeah. Services upon service over on that side. Yep, exactly. Here's the other thing. You don't have to use any of this. Theoretically, if you want to, you could go host your own iceberg rest catalog and you could just write the data to S3 and register it there. You could use snowflakes Polaris catalog. Yep.
42:55You could use, uh, there are a bunch of options here. It could be delta unity catalog would work the same way. Uh, these are all just like whatever flavor of as long as it's as long as it surfaces and iceberg rest endpoint, you can do this same thing with it. Exactly right. Exactly right. Um, okay. We got one more here that we'll just take and then we'll wrap this
43:19up. Thanks everyone for your questions. Um, Rick, thanks for giving us a home game type question. What is the recommendation? So there's a seven day demo available. Is that the way to go? Absolutely. I think we see people that succeed the most kind of in their trial come with like a use case in mind. You know, if you're just kind of like, oh, is this a database I can play around with, you know, um, you know, uh, maybe, maybe the time's not right, but as soon as you have kind of like
43:43a use case, especially something that like fits horizontally in your business, meaning you have like a, Hey, I need to ingest some data and then I want to model it. And then I want to visualize it. Um, we really, we really find that helps a mother duck as an awesome remote MCP that you can just drop into your cloud or to your open AI, uh, GPT subscription or you know, any other harness. Um, the MCP is an awesome,
44:07awesome way to get started. Um, it's in the marketplaces there. So even if you're in like a corporate account, um, you can get it added to your sub and just start fooling around with it. Um, it's the main way that I use mother duck. Now, uh, I used to be, you know, I used to love our UI duck to be as this awesome UI duck to be slot or duck to be dash UI. If you run that, you get this beautiful UI to use duck TV. And I've basically just stopped doing that.
44:31And now I am fully MCP build. When I wrote data questions, I ask it, um, there's a whole host of kind of features in there that make it all just like work really, really seamlessly and get started really quickly. Uh, on data problems that frankly, I, you know, I built my first data warehouse in 2015 and it took us six months to buy the hardware to run it on. Um, and now we, someone can say, Hey, I have a problem shaped like this. And it is,
44:55you know, an hour later I have, I have something, um, anything four hours, like getting my first spark job set up in like 2016. Yeah. Actually like getting everything built and running and like all like understanding all the bells and whistles. And now Claude does that in five minutes. Yeah, yeah, exactly. Um, the, I will say I, I like, I am pretty
45:19and I still like using the UI. Um, I like poking around data. I don't trust a warehouse that I don't query myself. I don't know that might be, but I might be more old school than Jacob about this. I think uh, you know, my domain, uh, you know, mostly marketing and, and marketing is usually the realm of export CSV, put it in a spreadsheet and do your own work. And so the fact that now I have
45:43the power of, um, now that I have the power of, of, of where a warehouse on that things that would take me hours, take, you know, 10 seconds, which means I get the time. I get the time back to actually look at the data and understand it's right. And you know, for me, that's the right trade off. Um, but I totally hear you, you know, especially when you're building like durable data pipelines, I don't think you can just defer all of that to,
46:07you know, the AI yet. Um, uh, all right. That's all we got. Yes, we are. Yes. Yes. Yes, we are HIPAA compliant and we signed BAAs. That's part of our enterprise deals. So now HIPAA compliant. Yeah, that, that, uh, correct. We just, we, we acquired our friends at tower. They're now part of mother duck. And as part of that,
46:31um, our flight pieces are HIPAA compliant too for doing data processing. So really cool. All right. Thank you everybody. Uh, awesome work. We will see you, uh, see you later. Thank you.
