--- title: "The dawn of the Data Intelligence Engineer" canonical: "https://motherduck.com/blog/the-data-intelligence-engineer/" related: - title: "How AI Is Reshaping the Modern Data Stack | MotherDuck | MotherDuck" url: "https://motherduck.com/videos/duckdb-ai-modern-data-stack/" - title: "A Chief of Staff's Guide to Context Layers | MotherDuck" url: "https://motherduck.com/blog/chief-of-staff-guide-ai-agents-metrics/" - title: "Data Engineering Is AI Engineering | MotherDuck" url: "https://motherduck.com/videos/data-engineering-ai-engineering/" --- # The dawn of the Data Intelligence Engineer I asked Nick Ursa, our lead data engineer (and MotherDuck co-founder), how AI had changed his job. His answer: "This is the first time in my career with no backlog." You'd think that would leave him with plenty of time on his hands, but he's busier than ever. While the backlog is gone, the answers still have to be right, and a human still has to decide how much quality control to do. Whereas once upon a time his job was to make sure the numbers he was computing were right, now his job is to make sure that the numbers that an LLM or agent comes up with are right. That's a different job from working a ticket queue. At Data Outpost on November 4, I proposed a name for this new role: the Data Intelligence Engineer. I think it's now the emergent role on an AI-enabled data team, and I’d like to share why. ## **What a data team is for?** A data team has one job: enable the business to make good decisions. The faster they can make those decisions, based on a more accurate picture of what is happening, the better those decisions can be. ![01-why-do-answers-take-so-long.png](https://motherduck-com-web-prod.s3.us-east-1.amazonaws.com/assets/img/01_why_do_answers_take_so_long_638340027d.png) The hard part has been the route a question takes to become an answer. It moves like a relay. A marketing leader comes into work on Monday and notices that there was a spike in churn over the last week. Why did that happen? They file a ticket with the data team to investigate. An analyst picks up the ticket, and as they dig in, they realize some key data is not exported from one of the data sources. They file a ticket to ask a data engineer to look into it. That needs a change in one of the data pipelines, so the data engineer makes the change, tests it, runs the backfill, tests the backfill, and resolves the ticket. Then the ball is back in the court of the analyst, who can build the dashboard that was needed. By the time it ships, the marketing person can no longer remember why they asked in the first place. The churn rate recovered a couple of weeks later on its own. This isn’t calling anyone out as not operating effectively or urgently; nobody in that relay did anything wrong. But every handoff means waiting, context is lost, and the half-life of the ability to effectively act on the answer to the question slowly decays. As an aside, we should note that there was a whole generation of AI enabled tools that didn’t really help; the co-pilot era made some of the turnarounds faster, but the problem was that there were too many handoffs to be able to act quickly enough. ## **Something is different this time** Before December 2025, the case against AI for analytics was easy to make. LLMs wrote bad SQL. Business logic was unknowable. The sensible conclusion was that this was never going to work. In fact, I used to talk about how data teams were more or less exempt from the disruption that was happening in other areas of software. Then, all of a sudden, the ground moved. New models got better at SQL. Business logic turned out to be something an agent can figure out with some helpful context. And agents went multi-turn, which means they don't have to get it right the first time. They can try a query, look at what came back, and try again, just like a human analyst would do. Put those together and it … just … works. Self-service BI has been right around the corner for decades, what makes this different? Well, as a bit of an existence proof, here’s an example; the presentation I gave at Data Outpost was not created as a powerpoint deck or Google Slides, it was an [app that I built using MotherDuck and Claude](https://motherduck.com/dive-gallery/dives/data-outpost-episode-iv-a-new-hope-for-data-public/). I used live datasets from four different data sources, built a number of interactive and animated visualizations. Anything I did could be done by virtually anyone; I didn’t look at a single line of code or SQL. It is important to stress that this isn’t just a way to automate building your dashboards. BI is typically pretty static, with maybe a couple of drill-downs. But with AI, you can build much richer visualizations. And what’s more, if you can build a dashboard with a prompt, you can go beyond that and also build a full-blown app. Allowing business stakeholders to answer questions with data on their own has an even broader benefit than just shortening the data team’s backlog; it actually allows them to make better decisions. By cutting down the amount of time it takes and the number of handoffs involved, they can ask more questions. They can ask follow up questions while they still have all of the context in their heads. This means they can make confident decisions faster. ![02-feedback-loop.png](https://motherduck-com-web-prod.s3.us-east-1.amazonaws.com/assets/img/02_feedback_loop_8c55eaeb9f.png) ## Which of these jobs can be a prompt? AI is dramatically changing the job information workers perform. Software engineers often find their time being spent babysitting Claude sessions and rubber stamping agent-written code. What about the traditional roles of the data team? Which of their jobs, to put it a bit provocatively, can be a prompt? ![03-which-can-be-a-prompt.png](https://motherduck-com-web-prod.s3.us-east-1.amazonaws.com/assets/img/03_which_can_be_a_prompt_4c7cdd5762.png) As uncomfortable as the answer might be, rather a lot can be done via a prompt. In the not too distant future, writing SQL queries, creating dashboards, and building dbt pipelines are not going to be things that require specialized knowledge to do. Agents are likely better than humans at these tasks already, and they’re only going to get better. Our Data Engineer, Nick, moved us from one version of Airflow to another in a weekend, with minimal supervision, which once would have taken a week. Writing code or SQL or building dashboards was never the whole job, though. The need for accurate data and the ability to understand it doesn’t go away. And someone has to be accountable, who can sign off and say “yes, this is right”. That takes a human in the loop. Quoth Nick, one more time, “Data eng, regardless of tooling or who does it, is an accountability function." ## What still needs a human Once agents are writing the code, what’s left in the data world? It turns out quite a lot. Humans need to make sure that other stakeholders without the world “Data” in their title are able to solve problems with data. That means they need to ensure a clean data model with appropriate context and annotations, a clear ontology to ensure that terms and concepts are being applied consistently throughout, and also they need to make sure that things continue to work. These aren’t things that a model can intuit from data or history. A model knows SQL. It doesn't know how your company counts revenue, or which of several similar-looking tables is the real one (it is `revenue_cpx_new_revised_v2_use_this_one` ) . A model doesn’t know that when the sales team talks about revenue they mean commissionable revenue, and the finance team uses something different. Context is a crutch that is difficult to maintain. What’s even better is just building a clean data model in the first place. There is a good analogy with software; I tend to consider it a user experience failure any time any time your user has to read the manual; a cleaner design would have made whatever they were trying to do obvious. Context is just a manual for agents; if you can make the data model simple and clear enough, it can save them a lot of time and energy (and tokens). ## Meet the Data Intelligence Engineer The jobs to be done for a data person in the age of AI don’t really match any of the typical responsibilities of other data team members. I think it is different enough to merit a new title, which I hereby dub the “Data Intelligence Engineer” The Data Intelligence Engineer automates the data engineer's work (ingestion) and the analytics engineer's work (transformation and modeling) by creating and conducting software agents. They encode their knowledge of the system into shared context that can be used by other agents to answer questions about the organization. These include things like which fields are joinable but also what someone in the company means when they say “ARR”. Evals are another big part of their responsibility; these are the system tests that validate that when you ask questions with a known answer, the answer is correct. A good eval suite can give confidence that the types of questions that users ask of the system are going to be answered correctly by the agent. Evals also are a way of knowing when something breaks. The goal of a Data Intelligence Engineer is to enable self-service. That is, anyone in the organization should be able to open Claude or ChatGPT or another agent and answer questions correctly about the business, use that to build apps and visualizations, and in the end, make high quality decisions with the benefit of data. . When the Data Intelligence Engineer does the job well, everyone who asks a question gets the experience of working with the smartest analyst in the company. ![04-data-intelligence-engineer.png](https://motherduck-com-web-prod.s3.us-east-1.amazonaws.com/assets/img/04_data_intelligence_engineer_7b1cb6d68e.png) Day to day, that means they are going to be building frameworks, building the data model, curating the context, preventing things from getting broken, fixing them when they break anyway, running evals, and being accountable for the surfaces the business depends on. We wrote the whole thing up as a [job description](https://motherduck-com-web-prod.s3.us-east-1.amazonaws.com/assets/img/Job_Description_Data_Intelligence_Engineer_Google_Docs_f935282ff7.pdf), because we believe this is the way things are going and it's time to give it a name. If you're a data engineer, an analytics engineer, or an analyst, you'll recognize parts of this. Those roles were already blending into each other, and morphing into something new. This is a role you play, whatever your title says. ## Won't this be chaos? It's a fair question. More people building means more chances to be wrong, with less oversight. There is the possibility of making stupendously silly mistakes by acting on the wrong data. I’ll leave it up to the reader whether they think that stupendously silly mistakes are something new or not. The scary thing is that now the data team can be held responsible for them. There are two things that help. The first is that the closer you are to the problem, the better you can spot errors. The person who lives with the churn number notices when it looks off. From experience, we’ve found that results actually improve because the business owner has a fingertip feel on their area; they know when something isn’t right, and can often spot problems even in the source data. The other thing that helps is that you don’t need to watch everything with the same level of vigilance. Another quotable quote from our friend Nick is that “Agents need supervision proportional to their blast radius." That is, if something is going to be used for important reporting or decisions, you should pay a lot of attention. If you’re trying to figure out how many sandwiches to order for lunch, it probably isn’t going to cause a riot if you order too many with habanero mustard because Claude imagines it is extra tasty. You can think of this in three circles of supervision. The outer is the Wild West. Anyone can build, they hopefully build on a reasonable foundation, but the data team keeps their hands off. The next tier is the Curated set. This is a set of dashboards, visualizations, apps, and metrics that is important to the business in some form or another. When something breaks in these dashboards, the data team should make a best effort to fix them. Finally there is the inner circle, the Core. This contains key metrics for the organization, things that are important to get right. The data team should make sure that these are high quality and predictable at all times. The board number lives in the Core. ![05-three-tiers.png](https://motherduck-com-web-prod.s3.us-east-1.amazonaws.com/assets/img/05_three_tiers_a5e45dc81b.png) ## **What does this all have to do with MotherDuck?** On MotherDuck, we build mechanisms to empower your AI enabled data team. Flights are vibe-codable pipelines that can pull data into your warehouse and translate it into a sane model. Context lives in Guides, which are markdown files describing your data model and your business. A Guide starts out as private, but an admin can promote it to apply to the whole organization so that every person and agent works from the same definitions. We call the data visualizations that can be created by agents “Dives”. Every person and agent gets its own Duckling, a dedicated DuckDB instance. All of these are reachable from SQL, so one person or agent can work a problem across ingestion, transformation, context, and apps without a handoff between tools. ## The only way out is through We're in a chaotic moment. Chaos breeds the biggest opportunities, and the tools hand you some real super-powers if you're willing to pick them up. We're all learning how to do this together. The data team isn't going anywhere. Someone still has to own the data model, curate the context, keep things working, and answer for the results. No-backlog-Nick has more to do than ever, and but the job is changing and not going back. So who's doing this at your company, and what do you call them? PS - if you want to see my slides from the talk, check out the [Dive Gallery](https://motherduck.com/dive-gallery/dives/data-outpost-episode-iv-a-new-hope-for-data-public/)