#73 – Andrew Ng: Deep Learning, Education, and Real-World AI

[0] The following is a conversation with Andrew Eng, one of the most impactful educators, researchers, innovators, and leaders in artificial intelligence and technology space in general.

[1] He co -founded Coursera and Google Brain, launched Deep Learning AI, Lending AI, and the AI Fund, and was the chief scientist at Bidu.

[2] As a Stanford professor and with Corsera and Deep Learning AI, he has helped educate and inspire millions of students, including me. This is the Artificial Intelligence Podcast.

[3] If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcast, support it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F -R -I -D -M -A -N.

[4] As usual, I'll do one or two minutes of ads now and never any ads in the middle that can break the flow of the conversation.

[5] I hope that works for you and doesn't hurt the listening experience.

[6] This show is presented by Cash App, the number one finance app in the app store.

[7] When you get it, use code Lex Podcast.

[8] Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1.

[9] Broker services are provided by Cash App investing, a subsidiary of Square, a member SIPC.

[10] Since Cash App allows you to buy Bitcoin, let me mention that cryptocurrency in the context of the history of money is fascinating.

[11] I recommend a cent of money as a great book on this history.

[12] Debits and credits on ledgers started over 30 ,000 years ago.

[13] The U .S. dollar was created over 200 years ago, and Bitcoin, the first decentralized cryptocurrency, released just over 10 years ago.

[14] So given that history, cryptocurrency is still very much in its early days of development, but it's still aiming to and just might redefine the nature, of money.

[15] So again, if you get Cash App from the App Store or Google Play and use the code Lex podcast, you'll get $10, and Cash App will also donate $10 to First, one of my favorite organizations that is helping to advance robotics and STEM education for young people around the world.

[16] And now, here's my conversation with Andrew Eng.

[17] The courses you taught on machine learning of Stanford, and later on Coursera, the co -founded, have educated and inspired millions of people.

[18] So let me ask you, what people or ideas inspired you to get into computer science and machine learning when you were young?

[19] When did you first fall in love with the field?

[20] There's another way to put it.

[21] During up in Hong Kong and Singapore, I started learning to code when I was five or six years old.

[22] At that time, I was learning the basic programming language, And they would take these books and, you know, they'll tell you, type this program into your computer.

[23] So type that program to my computer.

[24] And as a result of all that typing, I would get to play these very simple shoot -them -up games that, you know, I had implemented on my little computer.

[25] So I thought it's fascinating as a young kid that I could write this code.

[26] That was really just copying code from a book into my computer to then play these cool little video games.

[27] Another moment for me was when I was a teenager and my father, because the doctor, was reading about expert systems and about neural networks.

[28] So he got me to read some of these books and I thought it was really cool.

[29] You could write a computer that started to exhibit intelligence.

[30] Then I remember doing an internship while I was in high school, this was in Singapore, where I remember doing a lot of photocopying and office assistants.

[31] And the highlight of my job was when I got to use the shredder.

[32] So the teenager of me, remember thinking, boy, this is a lot of photocopying.

[33] If only we could write software, build a robot, something to automate this.

[34] Maybe I could do something else.

[35] So I think a lot of my work since then has centered on the theme of automation.

[36] Even the way I think about machine learning today, we're very good at writing learning algorithms that can automate things that people can do.

[37] Or even launching the first MOOCs, mass open online courses that later led to Coursera, I was trying to automate what could be automatable in how I was teaching on campus.

[38] Process of education tried to automate parts of that to make it more, sort of to have more impact from a single teacher, single educator.

[39] Yeah, I felt, you know, teaching Stanford, I was teaching machine learning to about 400 students a year at the time.

[40] And I found myself filming the exact same video every year, telling the same jokes in the same room.

[41] And I thought, why am I doing this?

[42] this.

[43] Why we just take last year's video?

[44] And then I can spend my time building a deeper relationship with students.

[45] So that process of thinking through how to do that, that led to the first moves that we launched.

[46] And then you have more time to write new jokes.

[47] Are there favorite memories from your early days of Stanford teaching thousands of people in person and then millions of people online?

[48] You know, teaching online, what not many people know was that A lot of those videos were shot between the hours of 10 p .m. and 3 a .m. A lot of times, we're launching the first MOOs at Stanford.

[49] We already announced a course, about 100 ,000 people that signed up.

[50] We just started to write the code, and we had not yet actually filmed the video.

[51] So, you know, a lot of pressure.

[52] 100 ,000 people waiting for us to produce the content.

[53] So many Fridays, Saturdays, I would go out, have dinner my friends, And then I would think, okay, do you want to go home now or do you want to go to the office to film videos?

[54] And the thought of being able to help 100 ,000 people potentially learn machine learning, fortunately that made me think, okay, I want to go to my office, go to my tiny little recording studio.

[55] I would adjust my logic webcam, adjust my WACOM tablet, make sure my lapel mic was on, and then it would start recording often until 2 a .m. or 3 a .m. I think unfortunately that it doesn't show that it was recorded that later night, but it was really inspiring the thought that we could create content to help so many people learn about machine learning.

[56] How do that feel the fact that you're probably somewhat alone, maybe a couple of friends recording with a Logitech webcam, and kind of going home alone at one or two, a .m. at night, and knowing that that's going to reach sort of thousands of people, eventually millions of people.

[57] What's that feeling like?

[58] I mean, is there a feeling of just satisfaction of pushing through?

[59] I think it's humbling, and I wasn't thinking about what I was feeling.

[60] I think one thing that I'm proud to say we got right from the early days was I told my whole team back then that the number one priority is to do what's best for learners, do what's best for students.

[61] And so when I went to the recording studio, the only thing on my mind was, what can I say, how can I design my slides?

[62] What I need to draw, right, to make these concepts as clear as possible for learners.

[63] I think, you know, I've seen sometimes instructors is tempting to, hey, let's talk about my work.

[64] Maybe if I teach you about my research, someone will cite my papers a couple more times.

[65] And I think one things we got right, launched the first few MOOCs and later building Coler, was putting in place that bedrock principle of let's just do what's best for learners and forget about everything else.

[66] And I think that that as a guiding principle turned out to be really important to the rise of the movement.

[67] And the kind of learner you imagined in your mind is as broad as possible, as global as possible.

[68] So really try to reach as many people interested in machine learning and AI as possible.

[69] I really want to help anyone that had an interest in machine learning to break into the field.

[70] And I think sometimes, I actually people ask me, hey, why are you spending so much time explaining grade and descent?

[71] And my answer was, if I look at what I think the learning needs and what benefit from, I felt that having that a good understanding of the foundations, coming back to the basics, would put them in a better stead to then build on a long -term career.

[72] So we've tried to consistently make decisions on that principle.

[73] So one of the things you actually revealed to the narrow AI community at the time and to the world is that the amount of people who are actually interested in AI is much larger than we imagined.

[74] By you teaching the class and how popular it became, it showed that, wow, this isn't just a small community of sort of people who go to New Reps and it's much bigger.

[75] It's developers.

[76] It's people from all over the world.

[77] I mean, I'm Russian, so everybody in Russia is really interested.

[78] There's a huge number of programmers who are interested in machine learning, India, China, South America everywhere.

[79] There's just millions of people who are interested in machine learning.

[80] So how big do you get a sense that the number of people is that are interested from your perspective?

[81] I think the number is grown over time.

[82] I think it's one of those things that maybe it feels like it came out of nowhere, but it's an insight that are building it.

[83] It took years.

[84] It's one of those overnight successes that took years to get there.

[85] My first foray into this type of online education was when we're filming my Stanford class and sticking the videos on YouTube and then some other things we had uploaded the whole works and so on.

[86] But basically the one hour, 15 minute video that we put on YouTube.

[87] And then we had four or five other versions of websites that had built, most of which you would never have heard of because they reached small audiences, but that allowed me to iterate, allow my team and me to iterate to learn what the ideas that work and what doesn't.

[88] For example, one of the features I was really excited about and really proud of was build this website where multiple people could be logged into the website at the same time.

[89] So today, if you go to a website, you know, if you are logged in and then I want to log in, you need to log out if it's the same browser, same computer.

[90] But I thought, well, what if two people, say you and me were watching a video together in front of a computer?

[91] What if a website could have you type your name and password, have me type my name and password?

[92] And then now the computer knows both of us are watching together and it gives both of us credit for anything we do as a group.

[93] Infants feature rolled it out in a high in a school in San Francisco.

[94] We had about 20 -something users.

[95] Where's the teacher there at Sacred Heart Cathedral Prep?

[96] The teacher is great.

[97] And guess what?

[98] Zero people use this feature.

[99] It turns out people studying online, they want to watch the videos by themselves.

[100] You can play back, pause at your own speed rather than in groups.

[101] So that was one example of a tiny lesson learned, out of many, that allowed us to hone into the set of features.

[102] And it sounds like a brilliant feature.

[103] So I guess the lesson to take from that is there's something that looks amazing on paper and then nobody uses it.

[104] It doesn't actually have the impact that you think it might have.

[105] I saw that you really went through a lot of different features and a lot of ideas to arrive at the final, at Coursera, the final kind of powerful thing that showed the world that MOOCs can educate millions.

[106] And I think with the whole machine learning movement as well, I think it didn't come out of nowhere.

[107] Instead, what happened was as more people learn about machine learning, they will tell their friends, and their friends will see how it's applicable to their work.

[108] And then the community kept on growing.

[109] And I think we're still growing.

[110] You know, I don't know in the future what percentage of all developers will be AI developers.

[111] I could easily see it being north of 50%, right?

[112] because so many AI developers broadly construed, not just people doing the machine learning modeling, but the people building infrastructure, data pipelines, all the software surrounding the core machine learning model, maybe it's even bigger.

[113] I feel like today almost every software engineer has some understanding of the cloud, not all, but maybe this is a micrachronroller developer doesn't need to deal the cloud.

[114] But I feel like the vast majority of software engineers today are sort of having appreciates the cloud, I think in the future, maybe we'll approach nearly 100 % of all developers being, you know, in some way, an AI developer, or at least having an appreciation of machine learning.

[115] And my hope is that there's this kind of effect that there's people who are not really interested in soft, being a programmer or being into software engineering, like biologists, chemists, and physicists, even mechanical engineers, all these disciplines that are now more and more sitting on large datasets.

[116] And here they didn't think they're interested in programming until they have this data set and they realize there's these set of machine learning tools that allow you to use the data set.

[117] So they actually become, they learn to program and they become new programmer.

[118] So like the, not just because you've mentioned a larger percentage of developers become machine learning people.

[119] It seems like more and more the kinds of people who are becoming developers is also growing significantly.

[120] Yeah.

[121] I think once upon the time, only a small part of humanity was literate.

[122] You know, could read and write.

[123] And maybe you thought, maybe not everyone needs to learn to read and write.

[124] You know, you just go listen to a few monks, right?

[125] Read to you, and maybe that was enough.

[126] Or maybe we just need a few handful of authors to write the bestsellers, and then no one else needs to write.

[127] But what we found was that by giving as many people, you know, in some countries, almost everyone, basic literacy, it dramatically enhanced human -to -human communications, and we can now write for an audience of one, such as if I send you an email or you send me an email.

[128] I think in computing, we're still in that phase where so few people know how to code, that the codists mostly have to code for relatively large audiences.

[129] But if everyone, or most people became developers at some level, similar to how most people in developed economies are somewhat literate, I would love to see the owners of a mom and pop store be able to write a little bit of code to customize the TV display for their special this week.

[130] And I think it will enhance human -to -computer communications, which is becoming more and more important in today's world.

[131] So you think it's possible that machine learning becomes kind of similar to literacy where, yeah, like you said, the owners of a mom and pop shop is basically everybody in all walks of life would have some degree of programming.

[132] I mean capability?

[133] I could see society getting there.

[134] There's one of the interesting thing.

[135] You know, if I go talk to the mom and pop store, if I talk to a lot of people in their daily professions, I previously didn't have a good story for why they should learn to code.

[136] You know, we could give them some reasons.

[137] But what I found with the rise of machine learning and data science is that I think the number of people with a concrete use for data science in their daily lives, in their jobs, maybe even larger than a number of people with concrete use for software engineering.

[138] For example, if you run a small mom and pop store, I think if you can analyze the data about your sales, your customers, I think there's actually real value there, maybe even more than traditional software engineering.

[139] So I find that for a lot of my friends in various professions, be it recruiters or accountants or, you know, people that work in factories, which I deal with more and more these days, I feel if they were data scientists at some level, they could immediately use that in their work.

[140] So I think that data science and machine learning may be an even easier entree into the developer world for a lot of people than the software engineering.

[141] That's interesting.

[142] And I agree with that, but that's beautifully put.

[143] We live in a world where most courses and talks have slides, PowerPoint, keynote, and yet you famously often still use a marker and a whiteboard.

[144] The simplicity of that is compelling, and for me at least, fun to watch.

[145] So let me ask, why do you like using a marker and whiteboard, even on the biggest of stages?

[146] I think it depends on the concepts you want to explain.

[147] For mathematical concepts, it's nice to build up the equation one piece of the time.

[148] And the whiteboard marker or the appendist is a very easy way to build up the equation, build up a complex concept one piece of the time while you're talking about it.

[149] And sometimes that enhances understandability.

[150] But the downside of writing is that is slow.

[151] And so if you want a long sentence, it's very hard to write that.

[152] So I think there are pros and cons.

[153] And sometimes I use slides and sometimes I use a whiteboard or a stylist.

[154] The slowness of a whiteboard is also its upside because it forces you to reduce everything to the basics.

[155] So some of your talks involve the whiteboard.

[156] I mean, it's there's really not, you go very slowly and you really focus on the most simple principles.

[157] and that's a beautiful that enforces a kind of a minimalism of ideas that I think is surprisingly for me is great for education.

[158] Like a great talk, I think, is not one that has a lot of content.

[159] A great talk is one that just clearly says a few simple ideas.

[160] And I think the white board somehow enforces that Peter Abil, who's now one of the top roboticists and reinforcement learning experts in the world was your first PhD student.

[161] So I bring him up just because I kind of imagine this must have been an interesting time in your life.

[162] Do you have any favorite memories of working with Peter, your first student in those uncertain times, especially before deep learning really sort of blew up.

[163] Any favorite memories from those times?

[164] Yeah.

[165] I was really fortunate.

[166] to have had Peter Abu as my first PhD student.

[167] And I think even my long -term professional success builds on early foundations or early work that Peter was so critical to.

[168] So I was really grateful to him for working with me. You know, what not a lot of people know is just how hard research was, and still is.

[169] Peter's PhD thesis was using reinforcement learning to fly helicopters.

[170] And so, So, you know, actually, even today, the website, heli .stanford .edu, hely .companford .org, you still up, you and watch videos of us using reinforcement learning to make a helicopter fly upside down, fly loose, so it's cool.

[171] It's one of the most incredible robotics videos ever.

[172] So people should watch it.

[173] Oh, yeah, thank you.

[174] It's inspiring.

[175] That's from, like, 2008 or seven or six, like that range.

[176] Something like that.

[177] It's over 10 years out.

[178] That was really inspiring to a lot of people, yeah.

[179] What not many people see is how hard.

[180] it was.

[181] So Peter and Adam Codes and Morgan Quigley and I will work on various versions of the helicopter, and a lot of things did not work.

[182] For example, it turns out one of the hardest problems we had was when the helicopters flying around, upside down, doing stunts, how do you figure out the position, how do you localize the helicopter?

[183] So we want to try all sorts of things.

[184] Having one GPS unit doesn't work because you're flying upside down, GPS unit is facing down, so you can't see the satellite.

[185] So we tried, we experimented trying to have two, GPS units, one facing up, one's facing down.

[186] So if you flip over, that didn't work because the downward facing one couldn't synchronize if you're flipping quickly.

[187] Morgan quickly was exploring this crazy complicated configuration of specialized hardware to interpret GPS signals.

[188] Look into FPJ is completely insane.

[189] Spent about a year working on that didn't work.

[190] So I remember, Peter, a great guy, him and me, you know, sitting down in my office, looking at some of the latest things we had tried that didn't work and saying, you know, done it, like what now?

[191] Because we tried so many things and it just didn't work.

[192] In the end, what we did when Adam Colts was crucial to this was put cameras on the ground and use cameras on the ground to localize the helicopter.

[193] And that solved the localization problem so that we could then focus on the reinforcement learning and inverse reinforcement learning techniques so it didn't actually make the helicopter fly.

[194] And, you know, I'm reminded, when I was doing this work at Stanford, around that time, there was a lot of reinforced learning theoretical papers, but not a lot of practical applications.

[195] So the autonomous helicopter work for flying helicopters was one of the few, you know, practical applications of reinforcement learning at the time, which caused it to become pretty well -known.

[196] I feel like we might have almost come full circle with today.

[197] There's so much buzz, so much hype, so much excitement about reinforcement learning.

[198] But again, we're hunting for more applications and all of these great ideas that the communities come up with.

[199] What was the drive sort of in the face of the fact that most people are doing theoretical work?

[200] What motivate you in the uncertainty and the challenges to get the helicopter sort of to do the applied work, to get the actual system to work?

[201] Yeah, in the face of fear, uncertainty, sort of the setbacks that you mentioned for localization.

[202] I like stuff that works in the physical world so like it's back to the shredder and you know I like theory but when I work on theory myself and this is personal taste I'm not saying anyone else to do what I do but when I work on theory I personally enjoy it more if I feel that the work I do will influence people have positive impact or help someone I remember when many years ago, I was speaking with a mathematics professor, and it kind of just said, hey, why do you do what you do?

[203] And then he said, he actually, you know, he had stars in his eyes when he answered.

[204] And this mathematician, not from Stanford, different universities, he said, I do what I do because it helps me to discover truth and beauty in the universe.

[205] He had stars in his eyes and he said there.

[206] And I thought, that's great.

[207] I don't want to do that.

[208] I think it's great that someone does that fully support the people that do it, a lot of respect for people that.

[209] But I am more motivated when I can see a line to how the work that my teams and I are doing helps people.

[210] The world needs all sorts of people.

[211] I'm just one type.

[212] I don't think everyone should do things the same way as I do.

[213] But when I delve into either theory or practice, if I personally have conviction that here's a path for it to help people, I find that more satisfying to have that conviction.

[214] That's your path.

[215] You were a proponent of deep learning before it gained widespread acceptance.

[216] What did you see in this field that gave you confidence?

[217] What was your thinking process like in that first decade of the, I don't know what that's called, 2000s, the aughts?

[218] Yeah, I can say the thing we got wrong or the thing we got right.

[219] The thing we really got wrong was the importance of the early importance of unsupervised learning.

[220] So early days of Google Brain, we put a lot of effort into unsupervised learning rather than supervised learning.

[221] And there was this argument.

[222] I think it was around 2005 after Neurips, at that time called Nips, but now Neurips had ended.

[223] And Jeff Hinton and I were sitting in the cafeteria outside the conference.

[224] We had lunch, and Chetka, and Jeff pulled up this napkin.

[225] He started sketching this argument on a napkin.

[226] It was very compelling, as I'll repeat it.

[227] Human brain has about 100 trillion, so there's 10 to the 14 synaptic connections.

[228] You will live for about 10 to the 9 seconds.

[229] That's 30 years.

[230] You actually live for 2 by 10 to 9, maybe 3 by 10 to 9 seconds.

[231] So just let's say 10 to 9.

[232] So if each synaptic connection, each weight in your brain's neural network, has just a one -bit parameter.

[233] That's 10 to the 14 bits you need to learn in, up to 10 to 9 seconds of your life.

[234] So via this simple argument, which is a lot of problems, is very simplified, that's 10 to 5 bits per second you need to learn in your life.

[235] And I have a one -year -old daughter.

[236] I am not pointing out 10 to 5 bits per second of labels to her.

[237] So, and I think I'm a very loving parent, but I'm just not going to do that.

[238] So from this very crude, definitely problematic argument, There's just no way that most of what we know is through supervised learning.

[239] But where if you get so many bits of information is from sucking in images, audio, just experiences in the world.

[240] And so that argument, and there are a lot of known forces argument, you know, go into really convince me that there's a lot of power to unsupervised learning.

[241] So that was the part that we actually maybe got wrong.

[242] I still think unsupervised learning is really important, but we, but in the early days, you know, 10, 15, years ago, a lot of us thought that was the path forward.

[243] Oh, so you're saying that that perhaps was the wrong intuition for the time.

[244] For the time.

[245] That was the part we got wrong.

[246] The part we got right was the importance of scale.

[247] So Adam Coates, another wonderful person, fortunate to have worked with him.

[248] He was in my group at Stanford at the time, and Adam had run these experiments at Stanford, showing that the bigger we train a learning algorithm, the betterest performance.

[249] And it was based on that, there was a graph that Adam generated, you know, where the x -axis, y -axis lines going up into the right.

[250] So big you make this thing, the better performance accuracy is the vertical axis.

[251] So it's really based on that chart that Adam generated, that it gave me the conviction that it could scale these models way bigger than what we could on a few CPUs, which is where we had a stand -fit, that we could get even better results.

[252] And it was really based on that one figure that Adam generated that gave me the conviction to go with Sebastian Thurton to pitch, you know, starting a project at Google, which became the Google Brain project.

[253] The brain, you go find a Google Brain.

[254] And there, the intuition was scale will bring performance for the system, so we should chase a larger and larger scale.

[255] And I think people don't realize how groundbreaking is simple, but it's a groundbreaking idea, that bigger datasets will result in better performance.

[256] It was controversial at the time.

[257] Some of my well -meaning friends, you know, senior people in the machine learning community, I won't name, but who's people, some of whom we know, my well -meaning friends came and were trying to give me friendly friends like, hey, Andrew, why are you doing this?

[258] This is crazy.

[259] It's in the near -natural architecture.

[260] Look at these architectures are building.

[261] You just want to go for scale?

[262] Like, this is a bad career move.

[263] So my well -meaning friends, you know, we're trying to, some of them, we're trying to talk me all of it.

[264] But I find that if you want to make a breakthrough, you sometimes have to have conviction and do something before is popular, since that lets you have a bigger impact.

[265] Let me ask you just in a small tangent on that topic.

[266] I find myself arguing with people saying that greater scale, especially in the context of active learning, so very carefully selecting the dataset, but growing the scale of the dataset is going to lead to even further breakthroughs in deep learning.

[267] And there's currently pushback at that idea, that larger data sets are no longer, so you want to increase the efficiency of learning, you want to make better learning mechanisms.

[268] And I personally believe that bigger data sets will still, with the same learning methods we have now, will result in better performance.

[269] What's your intuition at this time on those, on the, this dual side, is doing your?

[270] need to come up with better architectures for learning?

[271] Or can we just get bigger, better data sets that will improve performance?

[272] I think both are important.

[273] And it's also problem dependent.

[274] So for a few datasets, we may be approaching, you know, base error rate, or approaching or surpassing human level performance.

[275] And then there's that theoretical ceiling that we will never surpass a base error rate.

[276] But then I think there are plenty of problems where we're still quite far from either human level performance or from base error rate and bigger data sets with neural networks without further average innovation will be sufficient to take us further.

[277] But on the flip side, if we look at the recent breakthroughs using, you know, transforming networks or language models, it was a combination of novel architecture, but also scale had a lot to do of it.

[278] If we look at what happened with, you know, GP2 and birds, I think, scale was the large part of the story.

[279] Yeah, that's not often talked about is the scale of the data set it was trained on and the quality of the data set because there's some, so it was like redded threads that had, they were uprated highly.

[280] So there's already some weak supervision on a very large data set that people don't often talk about, right?

[281] I find that today we have maturing processes of managing code, things like good.

[282] version control.

[283] It took us a long time to evolve the good processes.

[284] I remember when my friends and I were emailing each other C++ files in email, but then we had, was it CVS subversion, Git, maybe something else in the future.

[285] We're very mature in terms of tools of managing data and think of how to solve our very hot, messy data problems.

[286] I think there's a lot of innovation there to be had still.

[287] I love the idea that you were versioning through email.

[288] I'll give you one example, when we work with manufacturing companies, it's not at all uncommon for there to be multiple labels that disagree with each other, right?

[289] And so we would, doing the work in visual inspection, we will, you know, take, say, a plastic pot and show it to one inspector.

[290] And the inspector, sometimes very opinionated, they'll go, clearly, that's a defect, this scratch, unacceptable, got to reject this part.

[291] Take the same part to different inspectors.

[292] different, very opinion, clearly the scratch is small.

[293] It's fine.

[294] Don't throw it away.

[295] You're going to make us yours.

[296] And then sometimes you take the same plastic pot, show it to the same inspector in the afternoon as opposed in the morning, and very affinity to go in the morning to say, clearly it's okay.

[297] In the afternoon, equally confident.

[298] Clearly, this is a defect.

[299] And so what is the AI team supposed to do if sometimes even one person doesn't agree with himself or herself in the span of a day?

[300] So I think these are the types of very, very very practical, very messy data problems that my teams wrestle with.

[301] In the case of large consumer internet companies where you have a billion users, you have a lot of data, you don't worry about it, just take the average, it kind of works.

[302] But in a case of other industry settings, we don't have big data, if you're just a small data, very small data sets, maybe 100 defective parts or 100 examples of a defect.

[303] If you have only 100 examples, these little labeling errors, if 10 of your 100 labels are wrong, that actually is 10 % of your data set has a big impact.

[304] So how do you clean this up?

[305] What are you supposed to do?

[306] This is an example of the types of things that my teams, this is a landing AI example, are wrestling with to deal with small data, which comes up all the time once you're outside consumer internet.

[307] Yeah, that's fascinating.

[308] So then you invest more effort and time in thinking about the actual labeling process, what are the labels, what are the how our disagreements resolved and all those kinds of, like, pragmatic, real -world problems.

[309] That's a fascinating space.

[310] I find that actually when I'm teaching at Stanford, I increasingly encourage students at Stanford to try to find their own project for the end -of -term project.

[311] Rather than just downloading someone else's nicely clean data set, it's actually much harder if you need to go and define your own problem and find your own data set rather than you go to one of the several good websites, very good websites, with clean scoped datasets that you could just work on.

[312] You're now running three efforts, the AI Fund, LandingAI, and DeepLearning

[313].A