The Bad, the Good, and the Ugly: Navigating the New Waves of Machine Learning ft. Adi Polak

0:14Adi Nice to have you here. Adi has about a decade of work experience She started as a machine learning

0:22researcher, then Swift software engineer, then took a deep dive into the ocean of data over at Microsoft back again. And now she's in developer experience making sure developers have the best time ever, speaking up for them and making their experience a walk in the park. And on top of that, she's also a

0:42social media star. She's been dishing out knowledge on Twitter, LinkedIn, Medium. And if that's not enough, she's also an author. Her book, Scaling Machine Learning with Spark, is a must-read for the tech crowd, obviously, because it's published by O'Reilly. I mean, you know that it's legit, right? She's here with us and I'm grateful for her time. So, Adi, welcome.

1:06Thank you, Rely. So good to be here. So, transitioning from a software engineer to data. So, you were a machine learning researcher, then you went back to software, then you went back to data. Can you tell us a bit more about the challenge of such a transition? Yeah. Well, first of all, there's a lot of advantages. One is you bring a lot of

1:26skills to the table and you're also able to see and understand internal customers. So, if as a data engineer you build platforms for, you know, ML folks or data scientists or BI folks, then you have a better understanding of what do they need, how do they understand things, what is the terminology. The challenges are, you know, the other way around. So, coming from machine

1:50learning or software engineering, now you have to deep dive into data modeling, normalization of databases, the database management, how all the different aspects works in order to make sure it's high-performance, highly available, and really delivers on the promise that it needs for the

2:10organization and to deliver impacts. So, that was a good transition that taught me a lot about the field and was also enabled me to serve the internal customers better because I came from a customer's perspective. So, I knew what I was looking for. I knew what I needed. So, when you know what your customer needs

2:31are, it makes it easier to deliver. Let's put it that way. So, what would be the easiest transition, right, in your opinion? Machine learning or, like, data, I would say, infrastructure person to software engineer or the other way around? That's a really good question. So, if a data

2:52infrastructure. . . So, some data infrastructure engineers are actually distributed systems engineers because, yeah, if they're using heavy users of open source tools and they're doing the more side of the data lake side of the house, Hadoop, all these systems, so they probably have some solid

3:13software engineer skills on top of, you know, understanding data, understanding distributed systems, and so on. If they're coming from a pure data world, like the data warehousing, SQL, and so on, sometimes the transition could be a little bit harder because they'll need to improve their programming skills,

3:36really work on things like clean code, how project structure is built, testing, and so on, because it's slightly different projects altogether. So, both of them have challenges. I think it really depends on which side of infrastructure people are coming from. Yeah, but both of them are highly, highly required skills.

4:01Yeah, and I've seen people transitioning from one way or the other. I feel like, as you mentioned, it depends on your background, but if you're more in the data warehouse, you care more about the customer, we're talking about that, because you know how the thing is used, right? And it's a software engineer going into data, you know, for him it's just the code base, which is different in the world of data

4:25because it's inherent with so many different use cases. Can you share with us the inspiration and the reason behind writing your book? Yeah, so I guess it started with a lot of frustration. Always. Which is always, you know, all the mistakes. So, I did a couple of rants on Twitter and then figured out why it all turned into a book. Exactly,

4:47something like that. So, starting from, you know, more on the machine learning and data science side, I worked at the lab and I did a bunch of projects for Fortune 500 companies, and then I got exposed to big data. Now that was 2013, 2014, so almost a decade ago, and I realized that, hey, you know, I'm kind of like missing the skills

5:09on how to actually operate on top of this big data, so we had some access to Hadoop and Spark and so on, but it wasn't as great as it is today, and so I moved to the data space, and after some time I realized that people are still struggling with the same questions that I had, and I thought, you know what, how about I'll just capture all that in a book, like instead of meeting people over

5:33over and over again, how about I'll just capture all that in a book and serve more people that way, and so I did, and O'Reilly were, you know, super kind to publish the book on their behalf as part of the, you know, publication. So, starting with frustration, acknowledging it, understanding it, and understanding that more people are following that path, and also this week we did the book signing

5:57together with a company that helped sponsor it, and it was a huge line of people, and everyone repeated the same thing. It's like, oh, we've been doing data science for some time now, and now we need to learn the other side of things, like we need to scale up to Spark, we need to start using a better tool to manage our machine learning lifecycle, so this is exactly what O'Reilly is. It's like, oh, that's

6:21amazing to see, you know, ten years forward, there's still, although, you know, technology changes, there's still a huge need for it. Yeah, and about that, why do you think that our, like, data world keeps changing always? That's it, that's it. So, at the same time, for me, that's exciting, and it's painful, because I feel like I always need to, you know, learn something new, which is nice, but

6:45then there is this anxiety where I'm missing things, and oh, no, I need to learn back again, and, you know, being back as a junior, so what's your feeling about that? Yeah, so I like to treat what's happening in the data and machine learning space a little bit like when a tree grows, right? You have the circles, if you ever looked at, kind of like, at a tree, you see there are these circles, so

7:09it's kind of like the cycles that we are going through as a society, as an industry, so we started with solving some small problems, and then we realized, oh, we can build on top of this, and we're kind of, like, in this continuously cycle of data and machine learning, data machine learning, and even, you know, this week, some of the conversations that I had were around, you know, we have some

7:31tools to do the AI and the LLMs thing, but we now realized, again, that one of the challenges is actually data quality, because if we're injecting bad quality into the system, we'll get a bad LLMs at the end of the day, not bad, quote-unquote, but something that won't operate the way we want it to operate, so it's still kind of repeating that cycle again and again, and it's looking in the

7:55announcements also, you can see that, you know, the theme there was build your own models, and bring your own models means bring your own data to build those models. And for that, you need a certain data maturity, right? Yes. Because that's not being told there, there's a bit of a sales pitch, and I agree, everyone needs to

8:16make their business, right? Yeah. But yeah, whatever tool you are using, if you're not that amateur enough, how do you see the maturity in company evolving this past year on that sense? Yeah. Like the data foundation, really. So companies shift their strategy, there was some reduction in, you know, in resources that we saw in the EC, and now what we're seeing is with the existing resource, or

8:41you know, kind of like some more money that's coming into the world again, we're seeing companies are changing their strategy and saying, okay, now I need to build that LLM thing because it can serve my customer right now. So companies are becoming more customer-centric, customer-focused, which I believe that's the right way to go, because building a product for

9:03potentially someone to use, but then there's no really tangible customers that are going to, you know, get value out of it, it doesn't make sense. So that was a big shift is into this space, saying, okay, I know exactly what I need from my data, it's not like the modern data stack, nice to have features anymore, kind of like

9:23the, you know, cream on top, it's what do I actually need, what are the basic things that I need in order to save my customer right now, and give them the capabilities that they need. So we're kind of like, I don't like saying it, but it's true, kind of like cutting the fat on data products and focusing only on

9:44what matters. Yeah, really interesting points. I've been thinking about why LLM has been so high, and I think aside from the technology itself, which is impressive, is the thing that you mentioned, it's really customer focused. So there is this chat interface, right, that anyone, you know, can use, and it's different from

10:07a data science model, a machine learning model, sorry, in production, where it's going to, you know, is it going to consume by a microservice, which decision is going to make, is it back-end or is it exposed from front-end, which kind of delay we need in terms of answer, or is it just for prediction, so there is so much

10:26question to answer to get to that customer facing, right, and LLM is just like, you know, the chat, but like my thought is like, we're gonna have a lot of chatbots, you know, what's your feeling about this, like we saw the announcement, right, yeah. Yeah, we're going to have a lot of opinionated

10:48chatbots are going to bring some mistakes in a way to the world. We'll still have a long way to go in order to create trustworthy systems. People that are going to go only with the chatbots ones are probably going to miss the mark because they're going to be a huge, huge competition that the big companies

11:13already won, essentially. So I think the additional value that companies can give is actually around knowledge retrieval. At the end of the day, you know, if I have a dashboard, if I have some data, it's easier for me as a human being to interact with the data through English. Yeah. I write what I want, I get some

11:33information back, as long as it's accurate, that's great, and then I can make sense out of that information and add more things. So around data retrieval, it's fantastic because till now, it was very hard to do data retrieval within organizations. It exists in some places, but it's not to that level of, you know,

11:56finding the contacts and creating the relationships within different words or paraphrases or. . . Yeah, so that's like the kind of the feature they announced with the Lakehouse IQ, right? Lakehouse IQ is one of them, but they're not the only company. I mean, Native Rift is great, but they're not the only companies who's

12:17doing that, and we'll see more and more companies doing it internally for their, you know, engineers sometimes. If you think about an engineering organization, sometimes there's a lot of knowledge that it's stuck in, like, docs, you know, GitHub comments, GitHub issues, and so on, and if you want to start searching for something, we have to go over all these issues, try to put something in the

12:40search bar and hope for the best, and go over lots and lots of information and text that, as human beings, at some time, you know, we can get tired of going all this huge amounts of text just to write the next con line or figuring out what's happening. And tired about reading, right? I think there was a couple of examples where the

13:00LLM, you know, scanned a bunch of text from different sources, but instead of giving a text answer, it's giving something else, which is directly actionable, right? So we were talking about, like, I'm looking at a dashboard, and I have directly the dashboard generated, right? It's not like, okay, you should build a piece of code.

13:20And Databricks talks about the English SDK, right? So, yeah. Can you give us your thoughts about it? Yeah. So the English SDK for Spark, it enables us to write prompt and get a respond on top of, you know, and it's being translated into a Spark engine, and we get

13:42a respond on top of it. So it's interesting. It is limited to, as far as I understand, it's limited to two capabilities. One is running some operations on top of a data frame, and the second one is visualization. Both of them are something that we need, right? So instead of going over Stack Overflow, the commentations, all of that,

14:03writing a prompt and getting the answer sounds fantastic. And I think there would be a lot of learnings for people to take advantage of that in terms of how to optimize, which words to use, and so on. Because it might be that, at the end of the day, the engine would understand something different than what we want it to understand.

14:26You need to understand the Spark concept, too, yeah. Exactly. There's still need to understand what exists in it. It's a good step in the right direction. So it doesn't do, as far as I understand, it doesn't do exploration of the data. You need to know a little bit of what exists there. But it is going to simplify

14:47the lives of people into getting things done faster. So still didn't get a full, you know, hands on it. I'm still looking forward to try it out, but it seems extremely promising. And it's an interesting take, saying, you know, the next programming language is English. Yeah. Yeah. And it's really funny because I remember a few years

15:09back, Databricks, they shared the percentage of the different API calls to their service. And you could see, like, obviously, at the beginning was Scala, it was Spark, and then it shifted heavily to Python and SQL. So I think now SQL is the first interface. I didn't share here easily, like, I need to go back to the numbers. Maybe I'll put it

15:34in the post edit. But the point is, so we start from Scala and we're going to English. It's very funny, because I remember when I was writing some takeaways of the AI Summit two or four years ago, when it was a Spark Summit, I was like, hmm, I'm wondering which programming language is going to come next. You know, it's Python, SQL, but it's English.

15:56But still, you need to understand those software engineering concepts, right? I think that's the people, I think we should not be too blind about it. That's like, what, who would be the prompt engineering in data? What kind of profile should they have, do you think?

16:17That's interesting. I guess that individual should be, has some abilities to understand how the system works at a very high level. Optimization skills, like, there's still, in order to optimize the query and know that the answer that one's just got is kind of

16:39missing something or hallucinating, that individual needs to have some domain knowledge on the specific topic that they're asking. So they'll need to have that, and they'll need to have critical thinking, because if they'll just take all response, then it would be hard to

17:01trust the full outcome of such a system. So still having critical thinking, domain understanding, of course, some understanding of the framework and how it works in order to find bugs and optimize it, I think that will be critical. I think especially at the beginning, right?

17:22Because the model are still training, so they make mistakes, but yeah, the more things like people correct or say this is not correct, then the better it's going to be, right? It's the first version of everything that we're seeing. But so at the beginning, it's going to be very, we need to be very cautious about it. Do you think, I have a feeling personally

17:42I've been using GPT and GitHub Copilot. Sometimes I felt too dependent on that. As you mentioned, we get an answer, then I'm trying out, I'm not convinced about the answer, right? But I'm trying out, I have this half thinking, not critical thinking. Then like, oh no, this

18:02is not working. Then I'm heading back. And then after two or three times, back and forth, I'm rethinking about the initial question. And I was like, I think I could have done a better work myself in terms of time spent, right? So yes, it's like sometimes I'm being

18:25too lazy when using those tools. Do you have a bit of this feeling where we kind of like, yeah, avoid this critical thinking because we think, oh, it can answer everything I want? Yeah. Well, I think as human beings, critical thinking is a crucial aspect for us to create

18:49innovative and disruptive things. And there are repeatable motions that sometimes we need to go through that those can be easily automated with, you know, ChatGPT and alike. I do want to comment, I do want to say one thing about security. I think that's an interesting topic, especially when working with ChatGPT. So I don't know if you know that, but there is a story around

19:12security. It's really interesting one. So ChatGPT started suggesting or inventing code libraries, SDKs that didn't exist before. And so some hackers, black hat folks, they said, oh, you know, ChatGPT is recommending these libraries because he's making up names of libraries.

19:35Yeah. And they created, I heard that's like a library on GitHub, which was basically, yeah. Yeah. It's Trojan horse or what have you. But, you know, it depends on the library itself. But there are real risks. Right. So essentially, when we can feed information into ChatGPT, it trusts it because it's an online learning platform where it takes the information. It trusts us as human beings.

20:00the right information and kind of give it, you know, upvote, downvote, then some people could take advantage of that as well. So this is why, you know, being more opinionated. That's true. That's true. There is multiple danger. There's just the danger of making your own mistake, right? You believe what it is or you, you know, different things as a user, but without bad intention.

20:23And there's just people with other intention that, yeah, that's a really good point. During the process of writing your book, did you encounter any surprising and discoveries or insights that change your viewpoints on scaling machine learning? Yes, especially around hardware.

20:45Hardware has and still is changing. So a lot of NVIDIA's work with rapids and specific GPUs and so on. It was fascinating to see how fast they were evolving compared to other, you know, semiconductor companies.

21:04And it was interesting also how one can fine tune a lot of the work that they do with machine learning, with the lower level libraries in order to fit the exact hardware that they're using to get the most out of their scalable machine learning workloads, which I think not enough people are actually diving into that space.

21:26But it's like the connecting tissue between what actually the processor is going to run versus, you know, the code that we create and so on. So when during the compiling time, it actually fits better with the hardware. So I think that was it was a really interesting one because how fast things change all the time.

21:45Yeah, that's true that in terms of compute, a lot has been changed this past year, if you think initially on the distributed compute. What's your thought about that? Because it's true, like during the keynote, I haven't heard that much about distributed scaling.

22:06I mean, it's more scaling, yeah, but like more like insisted on distributed computes and Databricks, which are basically their core business kind of with Spark initially. So where do you see that evolving? I think it is abstracted from the user, essentially. It's not there yet in terms of, you know, people still need to configure, still need to know how many machines.

22:31But I think it would be abstracted. And it really depends on the actual case of a customer. Not everyone needs distributed compute. Sometimes it can fit within one machine. And that's great. And that's also a conversation around data fitting within one machine or the sudden compute fitting within one machine.

22:53So talking about storage versus compute, and we have the decoupled of the systems. This is also another question of, you know, how much memory do I have to actual process the data versus how many space I need on my desk in order to save the data. So these are two different aspects of that. I think it will get more and more abstracted because distributed systems are hard and core competencies of many businesses is to make it simple.

23:18Yeah, I think it's going to come definitely. It's a really good point that we still need to configure cluster, but less and less and at a higher level. And then when we figure out like all the custom things and what works for 90 percent of the use case, then we probably won't need to do it. What's next for you on the rise and another book? That's a really good question.

23:41So do you know Matt Housley and Joe Reis, authors of Fundamentals? Joe was here on the show just before. Fantastic.

23:54So we actually met and Matt and I decided to write a book on a short book now on LLMs for the rest of us, meaning really simplifying LLMs, making it accessible for people. So everyone would be able to understand how it works, how to take advantage of it without having a data survey. Cool. That's great. Any timeline?

24:19Looking into the next six months should be out. Cool. Looking forward. Yeah, I think it's really interesting to set the baseline, especially for other data profile. I have quite sometimes in my head and say, hey, I didn't jump yet on the LLM bandwagon, what do I need to read? Right. And so I think it's nice to have not something heavy, but like, you know, that set the baseline for everybody.

24:44So looking forward for that. Thank you. I wish you the best of luck for what's next and for your futures and also an amazing end of day in San Francisco. Thank you, Matt. It was a pleasure meeting you today. I really appreciate you taking the time.