Allison Stanger (2020-21 CASBS fellow), professor of international politics and economics at Middlebury College, interviews former 1979-80 CASBS fellow and world renowned philosopher Daniel Dennett. He discusses his time at CASBS, his journey through academia, recent works, artificial intelligence, why Darwin’s idea is the best anyone ever had, memes, gods, and, yes, windsurfing at CASBS.
From Bacteria to Bach and Back: The Evolution of Minds
Narrator: From the Center for Advanced Study in the Behavioral Sciences at Stanford University, this is Human Centered. Today on Human Centered, current CASBS research affiliate Allison Stanger sits down for a chat with world-renowned philosopher and former CASBS fellow, Daniel Dennett. The two discuss his time at the center from 1979 to '80 and his close working and personal relationships with other fellows. We'll hear about his journey through academia, from choosing to become a philosopher to choosing to teach at Tufts. Together, they wind their way through an intellectual forest of topics: artificial intelligence and machine learning, why competency doesn't require comprehension, the controversy with Stephen Jay Gould, the power of Darwin's ideas, memes, gods, and the second best thing that CASBS gave Dennett: windsurfing.
Allison Stanger: Hi, I'm Allison Stanger, and I'm a research affiliate at Stanford University's Center for Advanced Study in the Behavioral Sciences, otherwise affectionately known as CASBS. My guest today is the philosopher, writer, and cognitive scientist Daniel Dennett. Who is University Professor and Director of the Center for Cognitive Studies at Tufts University. He is also Austin B. Fletcher, uh, Professor of Philosophy. Both of us are CASBS alums and external professors at the Santa Fe Institute. We're going to have a wide-ranging conversation that'll cover a whole bunch of topics. We'll explore in some detail Professor Dennett's latest book, intriguingly titled From Bacteria to Bach and Back: The Evolution of Minds. So welcome to the show, Dan.
Daniel Dennett: Delighted to be here, Alison.
Allison Stanger: That's great. How is the weather today in Maine?
Daniel Dennett: Right now the weather is beautiful. We're having a little Indian summer, so it's sunny and warm. That may not last, of course, but it's very nice today.
Allison Stanger: Yes, I'm here in Vermont where we've got the same gorgeous sunny autumn day with beautiful leaves. So it's nice to be here. I just wanted to— I thought I would just begin by asking you a little bit about your time in 1979 to 1980 at CASBS. And I know that you were one of a 6-member AI and philosophy group during your fellowship year and also had ongoing conversations with Hubert Dreyfus and John Searle at Berkeley. As well as Amos Tversky at Stanford. Can you tell us a little bit about your recollections of that time and then maybe how it affected your ongoing and future work?
Daniel Dennett: Yeah, it was— that was a great year, one of my favorite years in my very adventurous academic career. John McCarthy, the man who coined the term artificial intelligence, mathematician, who held the first conference at Dartmouth, did some amazing, important early work in AI, put together a group at the center. He was at Stanford at this point, but he put together a group at CASBS, 6 of us, 2 philosophers, 3 AI researchers, and 1 psychologist philosopher AI researcher Zenon Polichin, and the 6 of us spent a whole year, day after day, 5 days a week and more, thinking about, talking about AI and what its implications were, what its prospects were. It was a glorious time. It had its ups and downs. It almost failed right at the beginning. When we first got out there, And the first week we just got to know each other better. I had already known Zen and Pollution quite well. I didn't know John very well at that point. I knew John Hoagland, the philosopher from Pitt. I knew him quite well, probably knew him the best. But the others, Pat Hayes and Bob Moore, were, were new to me. And we spent the first 4 days introducing ourselves to each other and wrangling, wrangling, wrangling about some of the issues. And we thought we were getting off to a great start. And all of a sudden, John McCarthy lowered the boom on us and he said, Okay, look, you're all smart people. I wouldn't have invited you to join the group if I didn't think you were very smart. Now stop trying to prove it to each other. If you want to go on nitpicking and finding objections and sniping at each other in this way, you just go ahead and I will return to the AI lab down the hill and let you carry on for the year. I'm not going to put up with that. Well, we all— I mean, we weren't baby bunnies. We were fairly established scholars and researchers in our own right. But he really gave us a shot across the bows and said, look, I don't go for that kind of stuff. And then he told us some war stories, in effect, about how He'd helped solve some very hard problems, technical problems, during World War II. And when, when one of the— another of the experts was having trouble, instead of dumping on him and finding fault, everybody chipped in to help, help whoever it was figure out what was wrong and solve the problem. And this idea of constructive discussion. He just lay it on us real hard, and I've— as you can see, I've never forgotten it. It's influenced everything I've done since 1990. Uh, it really did have an immense effect on my life. Uh, philosophers are a nitpicky type discipline, and I have learned to hate that kind of one-upmanship, uh, sniping with a passion. And it was John who first blew the whistle on it for me. Um, but after that early warning, we settled down and things got really interesting. It was, it was a wonderful year. I can't say we solve the world's problems or even any of the serious problems, but my golly, we learned a lot and became close friends in the process. John was an amazing man. Somebody once said, "No, he's not a human being, he's a Martian, but he does a pretty good imitation of a human being." He had a puckish sense of humor, which was often in evidence. He had a bumper sticker on his car, "The Committee to Nuke the Whales." He was a gadfly and a cynic in many ways, but also a very smart and serious guy from whom I learned a tremendous amount.
Allison Stanger: Was he your connection to the Santa Fe Institute?
Daniel Dennett: No, no, he wasn't. Um, actually, maybe he was in some way, but, um, no, uh, I, I never— I don't think I ever communicated with him about the Santa Fe Institute. Uh, and I never was at the Santa Fe Institute with John as near as I can remember. Uh, but, um, uh, John thought we would we would solve some deep theoretical problems in AI and philosophy. We didn't, but at least we outlined some.
Allison Stanger: Yeah, well, that's important. Knowing what the questions are is, is so important.
Daniel Dennett: That to me, I often say that that's what philosophy is, is figuring out what the right questions are. We aren't so good at answers. We're pretty good at questions.
Allison Stanger: Yeah, that's critical. Let's talk about John Hoagland a little bit. What sort of conversations did you have with him at CASBS? Because didn't he coin the term "good old-fashioned AI," "GOFA"?
Daniel Dennett: He's the one that coined the term "good old-fashioned AI" in his book, which was published about a year after that, I think, called Artificial Intelligence: The Very Idea. John and I were dear friends, very close friends and confidants. I had known him for a few years. He'd been a student of Bert Dreyfus's at Berkeley. He wrote a very important paper. He has fledgling PhD in philosophy, but he wrote an important paper called The Plausibility of Cognitivism. Which appeared in Behavioral and Brain Sciences as a target article way back in the first few years, maybe volume 1. And I think that's how I got to know him. Maybe I'd met him before, but he moved to Pittsburgh and we became, we became dear friends. And until, until his untimely death. Yes. And he, he helped me through some of the hardest thinking I've ever done. I had started a project the year before. I had been at All Souls College and before that at Bristol. I'd had 2 years in a row off. Boy, what a privileged character I was. One year with a Fulbright at Bristol and then a visiting fellowship at All Souls. Then the year at CASBS. That year in England, I'd worked on a paper on a terrible tangle of philosophical issues, sometimes known to those who worked on it as the Propositional Attitude Task Force. This was all bound up with Russell and Frege. Quine. And I had decided, along with Steve Stitch, a philosopher who was with me in Bristol, we just tried to write a paper setting that whole issue straight. And we found that we had different ways of approaching it, so we, we got a divorce on that co-authorship. And he wrote a book on— I can't remember the title right now— plausibility of cognitive science or belief in cognitive science or something like that. And I wrote a paper that it took me really most of the year at CASBS to finish, the hardest intellectual work I think I've ever done. And it ended up with a paper called Beyond Belief. Where I tried to sort it all out. And I remember Patricia Churchland, who was a friend and confidante of mine, when she read it, she called it beyond belief and past caring.
Allison Stanger: Well, how would you defend yourself?
Daniel Dennett: He gave up on analytic philosophy before I did. So that was my attempt to put straight a very central set of issues that philosophers had been working on. This was presumably the A-team. This was the varsity philosophers who were working on this. Quine and Davidson and Kripke, and I was working on it. Finding it endlessly troubling and convoluted. And several times I almost gave up, and John would walk me up in the hills above the center and just talk to me and get me calmed down and show me that I could do it after all and encourage me. I couldn't have done it without John's help. It's not that he gave me ideas, but that he gave me confidence and gave me a new perspective on what I was doing. And I just love John for that. That was one of my great intellectual partnerships, actually.
Allison Stanger: Yeah. Well, that's a real gift, you know, when someone inspires you to persist and persevere, you know?
Daniel Dennett: Yeah, yeah, absolutely.
Allison Stanger: But what made you part ways with analytic philosophy in the end?
Daniel Dennett: Who says I have?
Allison Stanger: Well, you said that earlier, so I was just picking up on that.
Daniel Dennett: Well, well. Wondered what you meant. I'm still officially an analytic philosopher, but a lot of analytic philosophers would drum me out of the core, I guess. I just think that— It became a sort of Mandarin, hermetically sealed, taking in each other's laundry set of technical issues, which I couldn't convince myself were really worth spending much time on. One of my favorite quotes is from the great Canadian psychologist Donald Hebb, who once said, if it isn't worth doing, it isn't worth doing well.
Allison Stanger: I like that.
Daniel Dennett: Yes. And I decided that the Propositional Attitude Task Force was— wasn't worth doing. Now, I didn't just decide that and walk away. I wrote beyond belief to show, I thought, why it wasn't worth doing. I couldn't leave without pronouncing my diagnosis and epitaph. And it's still going on. And about once a decade, I look in on it to see whether they've made any progress. And I swear they're talking about exactly the same issues they were talking about back in 1980. And, uh, every now and then they reinvent one of my wheels from that paper and think it— that's some more progress. But I, for one, am not contributing to that literature anymore.
Allison Stanger: Yeah, yeah. Well, that makes sense. And you've certainly found plenty of other things to think about.
Daniel Dennett: I have indeed.
Allison Stanger: Yeah. I have an interesting quote from Brian Cantwell Smith's 2019 book. Do you know Brian?
Daniel Dennett: Oh, I know him well.
Allison Stanger: Yeah. The Promise of Artificial Intelligence: Reckoning and Judgment, with MIT Press. There he recounts that he attended a March 25th, 1980 lunch meeting of that working group we were just discussing. And here's what he writes about it. He says, at lunch on March 25th, 1980, at a workshop on artificial intelligence and philosophy at Stanford Center for Advanced Study in the Behavioral Sciences, one of the participants suggested that we would all have been better off if Kant had written his Critique of Pure Reason in Lisp. Looking up in alarm, I locked eyes with an equally shocked participant in the person of John Hoagland. So began a fast friendship and intellectual partnership that lasted until 2010, when John tragically succumbed to a heart attack at his own festschrift at the University of Chicago. Do you have any recollection of that lunch meeting that took place 3 days before your birthday?
Daniel Dennett: Oh yes, I do. I mean, we had a— we had a We had a workshop which involved a lot of people from Berkeley and around the country and even from abroad. Gareth Evans, the fiery Welsh philosopher, was there, and he was a major participant in the Propositional Attitude Task Force. And, uh, that was a landmark meeting in many ways. It was also very sad because I had invited Gareth Evans to come from Oxford to come a week early so he'd learn something about AI, because he had just taken the position in Oxford, the wonderfully named position, the Wild Readership in Mental Philosophy. A professorship of sorts. They'd offered it to me, but I couldn't afford to do it because I had a family. Salary wasn't enough. I would have loved to have been the wild reader of mental philosophy. That's great. But Gareth took it. I said, well, Gareth, you've got to come out to Stanford and I'll give you a crash course in AI. Which he did, and he, he got it, but he was not buying it. And but he— we had this amazing workshop.
Allison Stanger: What wasn't Gareth buying? Gareth Evans buying? That's interesting.
Daniel Dennett: Yes, I— it's hard to do justice to in a few sentences. Um, uh, he still had He was still a philosopher.
Allison Stanger: Yeah.
Daniel Dennett: And not an engineer. John Hoagland and I were engineers, uh, in spirit. Yeah. Um, but the tragic thing was that Gareth went back to Oxford, and I was so thrilled that he was going to be the wild reader, and he bring AI to Oxford, but he was actually dying of cancer, and he died just a few weeks or months later. He'd been ill when he was at the Stanford session. But we had some very intense conversations there. John Hoagland— I couldn't attend the Festschrift for John in Chicago But I knew that he was dying, actually, of Alzheimer's. He told me, he told me he had early onset Alzheimer's. And I had written a piece to be delivered, and Brian Cantwell Smith delivered it to John just an hour before he collapsed and died.
Allison Stanger: Wow.
Daniel Dennett: But John had the joy. I mean, I've talked to people who were there. John was hanging in there to the last moment, and he was being celebrated as the great philosopher he was. He, he really was. He was the real thing, a real philosopher, and I miss him.
Allison Stanger: Yeah. Who do you think made the lisp comment in that in that meeting?
Daniel Dennett: I can't remember. I wonder if Brian remembers. He probably does. It might have been John McCarthy.
Allison Stanger: Yeah, yeah.
Daniel Dennett: After all, John was really one of the creators of Lisp, probably the creator of Lisp.
Allison Stanger: Yeah, that's right. Yeah, that's who it was, I bet. I'm fascinated by this point you're making about being philosophers for thinking like engineers, because you have this great attraction to science, and that's what makes your work so exciting to read, is you really know the science as well as the philosophy. What do you think accounts for that? I would describe myself that way, but I was a math major as an undergraduate. You were a philosophy major.
Daniel Dennett: Yeah. Well, I grew up in a humanist family. My father was a historian. My mother was an English teacher. And the idea of being an engineer would never have occurred to any of us. In fact, if I'd said I wanted to be an engineer, I think it would have shocked my mother almost as if I'd said I'd want— I don't know, I'd wanted to be a masseur or a lion tamer. Uh, what? You know, engineers back in those days They were often viewed as sort of second-class intellectuals, but I had been building things and making things and fixing things all my life. And if I had— if there'd been an engineer in the family or something, I think I would have been an engineer. I might not have been a very good one, but I, I've always loved how things work and taking things apart and putting them back together again. So did John Hoagland. Yeah, he— while he was at CASBS— now this is the, you know, the dawn of word processing and, and the internet, and we all had, we all had, uh, ARPANET, uh, names, and we had it, we had email with ARPANET.
Allison Stanger: Really? In 1979?
Daniel Dennett: Absolutely. John McCarthy laid it on. We had, we had, um We had phone modems, 300 baud phone modems in our studies. John was fascinated with the ARPANET. To give you an idea of how much of an engineer he was, he took an old IBM Selectric golf ball typewriter. And wrote software to turn it into a printer.
Allison Stanger: That's very cool.
Daniel Dennett: And he— it was all— if you looked under the hood, he had all these bent paper clips which were in place which moved the golf ball.
Allison Stanger: And he—
Daniel Dennett: so you didn't touch the keys, you just sat there and the— and this electric ball went whippity-whippity-whippity. So he did that. He also— he also made a program that made beautiful mazes with a sort of random seed that were very tricky. I mean, the guy, he really knew computers.
Allison Stanger: He was self-taught entirely though, right?
Daniel Dennett: Oh yeah.
Allison Stanger: And were you as well, or were you—
Daniel Dennett: Yeah, pretty much. Yeah, pretty much. I'd never taken a course in computer science, but I learned a lot from various people in the field.
Allison Stanger: What was the path that led you to philosophy as a major?
Daniel Dennett: Well, when I was an undergraduate, when I was a freshman, I had a wonderful philosophy teacher. That was it. I had two wonderful philosophy teachers at Wesleyan. One was a historian of philosophy named Louis Mink, and with him I read Descartes' Meditations and thought, hmm, This is fantastic, but I think it's wrong. I think I'll spend an afternoon, see if I can figure out how to refute it. You know, 60 years later, I'm still working on it. And then there was a young logician there. I took a math course from him, and he was under the mistaken impression that I was a math prodigy. Because I'd had 2 years of calculus at Exeter. I wasn't a math prodigy, I'd just been well taught. And, uh, but I took a mathematical logic course from Henry Kuyberg, which was certainly the strangest and most daunting introduction to logic any undergraduate's ever had. But that's where I discovered Quine late one night in the math library reading Quine's book From a Logical Point of View, which was actually in words, not just in formulae. I spent all night reading the book and decided in the morning I'd transfer to Harvard because I had to do battle with this Quine guy. He was pretty amazing. I thought he was wrong, but I wanted to go there and study with him, which I did. I transferred to Harvard to work with Klein— work against Klein, really.
Allison Stanger: Yeah.
Daniel Dennett: Then, then when I got to graduate school, I discovered I was the village Kleinian. I accepted so much more of Klein than any of them did.
Allison Stanger: But it's interesting that you could read it and like it but still see things that you thought were wrong with it. Usually it takes undergraduates a bit of time to to, to disagree with something that's very well constructed, you know.
Daniel Dennett: Well, yeah, I guess that's right. But I always tell my students, you know, I love it when a student comes to me just on fire. They've got a refutation of Kant or Hume or Descartes or somebody, and I say, yeah, Great. Here's what you do. You write it up. Now, you may discover as you're writing it that you've somehow underestimated your quarry or misunderstood your quarry, and that your reputation isn't quite as solid as you thought it was. Don't worry. Finish the draft, then go back and write a new first paragraph which says, "Some folks might think that the following was a refutation of Kant or Hume or whatever. Here's the argument, but they're wrong." And that's a real contribution. That's a real contribution if you can do that.
Allison Stanger: That's an extension, right?
Daniel Dennett: It's your safety net.
Allison Stanger: Yeah, that's great.
Daniel Dennett: Yeah, so that's, that's been a, a favorite gimmick of mine over the years. I call it a gimmick, but it's a pretty deep— it's a pretty deep approach. Yes, basically I tell students, if you think you've got a refutation of some famous philosopher, the chances are like about 99 to 1 that you're wrong. They're great for a reason. But go ahead, see what you can make of it. But just remember, these folks, we're probably not idiots to put these idiots on the syllabus.
Allison Stanger: It sounds like you're a great teacher. I'm curious, what led you to Tufts? I know you were at UC Irvine.
Daniel Dennett: I was at Irvine for 6 years, loved it. But we wanted to come east because we were from the east, and we bought a farm in Maine and wanted to get back closer to Maine. Doing a 3,000-mile commute every summer did not strike me as a good idea. Yeah, and Tufts had the best invitation. Those were still sort of— it's still sort of a seller's market in the philosophy world. I actually had had 3, 3 offers with tenure, so I was able to choose Tufts, and it was the perfect place for me.
Allison Stanger: What made it perfect?
Daniel Dennett: A couple of years later, I was a visiting professor at Harvard for the year where I'd been an undergraduate, and when I'd been an undergraduate, I resented the fact that some of my professors were very good at hiding And I thought, hmm, shame on them. When I was a visiting professor, I discovered you have to be very good at hiding because Harvard students will eat you alive. They'll— they think they're entitled to world-class people at the top of their form every day, all day. And if you don't find some way to hide They will drive you into the ground. And, uh, one year of that was a fabulous year in many ways, but it was enough for me. I could go back to Tufts and I didn't have to supervise dissertations. And I— the, the best of the Tufts students were as good as the best of the Harvard students. But they weren't as sure of themselves, and they weren't as spoiled. They didn't think that they had a right to 12 hours a week of your personal time. I could give them all they could handle with a fraction of the preparation. That I had to reserve for Harvard students.
Allison Stanger: That's so interesting. Because I teach at Middlebury, and there's— and I've taught at Harvard, and I compare the two experiences this way, that if you have a guest celebrity academic coming in from outside at Harvard, and there's a dinner afterwards, the Harvard students will knock each other over to be able to sit next to that person at the dinner. When it comes to middle grade students, they'll all be politely hanging back because they're convinced someone else deserves the spot next to the speaker.
Daniel Dennett: Yeah, that's— that rings true.
Allison Stanger: And it's nice though. It's nice for the reasons you, you suggest.
Daniel Dennett: I could never have written the books I've written if I had been a Harvard professor because I would have been swallowed up. Harvard works their professors very hard and I guess being a big fish in a little pond was good for me.
Allison Stanger: Yeah, you've certainly done well. I mean, did you find it helpful to test drive some of your ideas on the undergraduates? Because you're such a lucid writer.
Daniel Dennett: Oh, every one of them. Every— this was an absolute— I discovered early on that— here's how Dan writes a book. When I've got a penultimate draft, I teach a seminar on the book, on the penultimate draft. And the students are there to keep me honest, to say what they don't understand. We go through it chapter by chapter. They all get thanked in the foreword, in the preface. They all get thanked in the preface and they get an autographed free copy. And otherwise, you know, their parents pay. It's the old Tom Sawyer whitewash fence trick. They paid big bucks. To fix my books.
Allison Stanger: That's great. Sam Huntington used to do that too. He used to say that if you want to write books for a more general audience, teach undergraduates because they'll tell you when you're not making sense. He says graduate students are too subservient. You know, they want to please you.
Daniel Dennett: Oh, absolutely right. That's been a theme of mine for years. If you can't explain what you're doing to a bright undergraduate, you don't understand what you're doing. And in many fields, especially in philosophy, where you're just talking to graduate students and colleagues, you get these little, uh, internecine, hermetically sealed battles that aren't worth anything. They're not productive, and nobody else can understand them. And the, the participants may even take a certain amount of pride in that.
Allison Stanger: But yeah, you know, it's, it's just Well, your books appeal to so many different disciplines too, I think, from that approach too, from being at a liberal arts university.
Daniel Dennett: Well, you know, I've often admitted this. I said if Consciousness Explained, which was my sort of first trade book aside from The Mind's Eye, which I did with Doug Hofstadter, if that had been a runaway bestseller, you know, and a Pulitzer Prize winner, but it hadn't been taken seriously by the people in the relevant cognitive sciences, it would have been a failure. What I had discovered was a really deep systemic fact about interdisciplinary research. When experts talk to experts, they always err on the side of underexplaining.
Allison Stanger: Yeah.
Daniel Dennett: Because to overexplain is to insult. And there's no cure for that except get a bunch of right undergraduates and have two experts come and teach them. Well, one expert teaches while the other one overhears, eavesdrops. Then the other expert teaches while the other one eavesdrops. Nobody worries about over-explaining because they're explaining the The undergraduates are the target audience. And in fact, if you do it that way, everybody wins because that's actually the right level for the experts too.
Allison Stanger: Yeah. No, I agree with you. That's a great way of describing it. Yeah. You've been described by Stephen Jay Gould as a Darwinian fundamentalist.
Daniel Dennett: Hahaha. Yes. Yes.
Allison Stanger: What is he misunderstanding when he calls you that, or would you proudly accept the label.
Daniel Dennett: In a way I would, but Steve got off in a world of his own. I think, actually, I've just been writing about this, I think he may have been done in by his early successful battle with cancer. There were a few years when he was widely reputed to be dying of cancer, and nobody wanted to criticize him. And after those, he went into full remission. But after that, he'd become used to not being— being above criticism. And he was saying all sorts of things that just weren't true. And he was indulging his— some of his ideological brickbats and hobbyhorses. In ways that were really, I think, unconscionable. He was misrepresenting opposing views— mine, but not just mine. Also a lot of evolutionary biology he was misrepresenting, and ecology, and sociobiology. And he, he, he had a political agenda which he could not distinguish from his scientific agenda. And although he was famed as the evolutionist laureate of America, he was actually selling America a false bill of goods. And a lot of people knew it, but he was so powerful and so vindictive and so vicious in his attacks Nobody wanted to bell the cat until I came along, and I just thought, I'm going to do this. Somebody's got to do this. While the evolutionary theorist says, "Will, hold your coat, Dan," I went after him. I have a whole story about that in my memoirs. I'm not going to tell it all to you here. Let's just say that, um, he went ballistic after my book appeared.
Allison Stanger: Yeah.
Daniel Dennett: And, uh, but not at first. For a couple of years, he, he just— when anybody asked him about it, he just said it was beneath discussion. It was just trash.
Allison Stanger: Really?
Daniel Dennett: But oh yeah. Then Maynard Smith, the dean of evolutionary theorists, John Maynard Smith in England, wrote a glowing review in the New York Review of Books, which was Steve's periodical. That's where he wrote his reviews. And Steve was so hurt by that review that he launched his two-part diatribe on Darwinian fundamentalism. He didn't dare really take on Maynard Smith. Maynard Smith was a hero to everybody. But he sure took after me, and he took after Steve Pinker, and a few others. But, you know, I'm a philosopher. I don't have NIH funding or NSF funding. He couldn't hurt me. So, it was okay. I could weather the storm. And in fact, in the wake of that, I was invited to give named lectures at a number of the top evolutionary biology programs in the country.
Allison Stanger: Their way of saying, "Thanks, Dan." What do you think his most egregious misrepresentation was, if you had to pick one thing that he misrepresented?
Daniel Dennett: His punctuated equilibrium was a misrepresentation. Because everybody from Darwin on realized that evolution could go fast or go slow. Most of the time it goes slow, but there are these punctuations, as people have recognized for years. So that was a, that was a tempest in a teapot. His attack on adaptationism in the famous or infamous Spandrels paper with Dick Lewontin That was a huge misrepresentation of adaptationism. And I had criticized him at length in Behavioral and Brain Sciences about that.
Allison Stanger: Yeah.
Daniel Dennett: So those were both misrepresentations. And he misrepresented the Cambrian explosion in Wonderful Life. And? Although I never went into this, there were a lot of people who wanted me to go after him for The Mismeasure of Man, which destroyed the reputation of Burt. Turned out that Burt wasn't the guy who fudged the skull measurements, it was Gould.
Allison Stanger: Oh dear.
Daniel Dennett: This came out in Science a few years later. Gould never made any changes. So, no, it's a sad story.
Allison Stanger: Yeah, that is sad. On a more happy note, your book is great. From Buck—
Daniel Dennett: Thank you.
Allison Stanger: Yeah, this latest one. It's such a distillation of so many different tributaries of thought.
Daniel Dennett: Well, for me, certainly that's what it was. It was trying to put it all together.
Allison Stanger: Is that— would you consider it a good distillation of your thought? Is that how you would situate it among your many contributions?
Daniel Dennett: Yeah, I think it ties off a lot of the loose ends pretty well and defends against some misrepresentations. I'm still working on consciousness, have some more ideas, more things to say about that.
Allison Stanger: I look forward to hearing them. I just want to ask you a little bit, and for our listeners, I thought we'd just wet their appetite for this book a bit, because in the book you write that you'd like the concept of manifest image and scientific image to migrate to fields other than philosophy. I found that very useful.
Daniel Dennett: These are terms from one of my heroes, philosopher Wilfred Sellars. The manifest image is the world we live in. It's the world of colors and haircuts and dollar bills and rainbows and so forth. The scientific image is the world of photons and quarks, atoms and molecules, and in which there really isn't any color. There's wavelengths of electromagnetic radiation, but atoms aren't colored, molecules aren't colored. So what evolution has done is simplified the behavioral world of organisms by giving them the manifest image, which is a sort of metaphorical, stripped-down, the physical world for dummies way of getting through the day. And only the things that matter to the particular organism are highlighted. The pigeon can peck crumbs of food off a piece of newspaper, but it can't read the newspaper. Doesn't need to read the newspaper. That's just background noise to that. And human speech is background noise. To every creature but us. We— what the manifest image consists of is what the late great psychologist Jimmy Gibson called affordances. They're the things that afford you doing things, like chairs afford sitting, windows afford looking out of or jumping out of holes. Afford putting things in them, doors afford opening and closing and walking in and out of, a knife affords cutting, so forth. A hardware store is a museum of affordances, and so is a furniture store, so is sort of everything in the concrete world that we inhabit. When I pick up my glass, it affords drinking. I don't have to think about it. I have a sense of its heft. I know it's got a back and a front. I know that it won't leak out the bottom. I know how far to tip it. All of these things are, as Heidegger would say, ready to hand. And that's, that's the world that we live in. And it is a sort of user illusion. It is like the user illusion of your cell phone. You don't want to know the dozens of layers of complexity of software that go into your cell phone. Heavens no. You want to click and drag and squeeze with your fingers. All those— that's a beautifully engineered, designed user illusion. That makes you an adept but basically clueless user of the cell phone. You don't know what's going on under the hood.
Allison Stanger: Yeah.
Daniel Dennett: And the same thing is true of your mind. Your mind is your user illusion of the brain. Your mind— evolution has given you a stripped-down digest a metaphorical way of rendering all the things that matter to you in a form that's easy for you to handle without too much exercise.
Allison Stanger: But my brain is stuck on something. My brain is stuck on where machine learning fits into all this, because would that represent the quintessential scientific image?
Daniel Dennett: Well, well, no. That's very nice. That's a good question. It's in between. Some people would say that deep learning is neither manifest image nor scientific image, it's engineering, and that's all it is. I once wrote a paper called Cognitive Wheels. Wheels are wonderful, but you don't find any in nature. Well, that's not true. It turns out there are some molecular-level wheels, little rotors in the flagella of bacteria, for instance. Some amazing little ratchets and wheels when you get down at the molecular level. But there aren't any animals with landing gear. No four-wheeled iguanas. So deep learning is a stunning application of big data. And fundamentally sort of Darwin-esque methods. But what we get from them are sort of wonder fabrics. They're amazing systems that can find needles in haystacks and find patterns that are otherwise all but invisible. But, but nobody knows how to build them into a, into a an intelligent agent yet.
Allison Stanger: You know, when I think about machine learning and I think about your description of our consciousness not being something that we can understand by looking under the hood, with machine learning you've got a similar sort of black box. So is this a move, like, is this science or is it something else?
Daniel Dennett: Aha, yes indeed, good question. Yes, it's science. It's a new kind of science.
Allison Stanger: Yeah.
Daniel Dennett: It's a kind of science which may be very disturbing because we may be seeing the end of the age of deep comprehension where we may give up comprehending some things and just get tools that can solve the problems without our comprehending them, which is, to a person like me, deeply unsettling. I want to understand, but some people just want to control, and they may be able to control without really having deep understanding.
Allison Stanger: Maybe you can help me understand. Do you think, do you think that machine learning is in a way an obliteration of the very idea of an Archimedean point, you know, as your favorite Descartes would have understood it, or Aristotle, or—
Daniel Dennett: It isn't, but that point's got to be demolished anyway. My way of saying it is there's no homunculus at the center of you. There's no Cartesian theater. There's no place where it all comes together. All of that is accomplished by a distributed, in space and time, network of lesser agencies that manage to hold you together and give you the benign illusion that you are in charge. And in fact, you are. But you isn't a thing that's distinct from your body. You is the way your body controls itself.
Allison Stanger: Yeah, that, that's a very fundamental insight, and some of the recent findings in cognitive science would very much back up your understanding of consciousness. But I— the thing I'm really curious about is, is, um, this isn't something you've written about, but I wonder where mental illness figures into this notion of consciousness. I mean, what Is that just another manifest image as opposed to a real image? When someone's psychotic, are they—
Daniel Dennett: What do you think?
Allison Stanger: I mean—
Daniel Dennett: Well, you know, some years ago, Nick Humphrey and I spent a year investigating multiple personality disorder. We wrote a long essay on that that was destined for the New York Review of Books, but then the New York Review of Books refused to publish it for mysterious reasons. We published it elsewhere, but it's in my collection of Brain Children. It's called Speaking for Ourselves, which was also the title of a briefly published newsletter for and by multiple personality disorder sufferers. Speaking for ourselves. They do speak for themselves. And the thing is that multiple personality disorder is a real disorder, or dissociative identity disorder it's called now. The question of the extent to which it's iatrogenic, caused by the doctor, is still an open question, and we're not sure. But there are certainly some quite robust cases. Of multiple personality disorder where there's 2 or 3 selves inhabiting a single body, or even more. But they have a somewhat different manifest image.
Allison Stanger: Yeah.
Daniel Dennett: Yeah.
Allison Stanger: I just wanted to get back, back to your book though, because I like this very much and I thought we could share it with the people listening, where you distill what I might call the essence of Darwin as In order to make a perfect and beautiful machine, it is not requisite to know how to make it. Could you just elucidate that concept? And I'm interested in the Turing concept too.
Daniel Dennett: Yeah, that's a quote from one of my favorite critics of Darwin, a man named Beverley, who in high dudgeon wrote about Darwin's strange inversion of reasoning. And, uh, and that's it. That's right, you don't have to understand. Knowledge comes later, comprehension comes later. In the beginning, there is mindless, purposeless natural selection, and it can actually design beautiful machines by the Darwinian process. It's slow, but it can speed up. Along the way, it makes many— it evolves just the way human artificers have made tools and then use those tools to make better tools and use those tools to make better tools. So evolution has, has improved its game over the millions of billions of years, adding new steps which enhance the capacity of natural selection to explore regions of design space and come up with yet further intricate designs. It's, it's to me the most wonderful idea of all ideas, and, and it's true. It works. We know it's true, and it turns everything on its head.
Allison Stanger: And then you have Turing's contribution: in order to be a perfect and beautiful computing machine—
Daniel Dennett: You don't have to understand arithmetic. And then there's Hume's strange inversion of reasoning too. He sees that the secondary qualities, so-called, the qualia that philosophers love to talk about, Or let's take that his idea of causation. It's not that causation is something we see with our eyes. It's not perceivable. It's something that our mind in effect projects. He says the mind spreads itself on external things. And that's the manifest image. We've had these revolutions, the Copernican Revolution, The Einsteinian revolution, the Darwinian revolution, and now we got one more which might be called the Humean revolution, or it might— or the AI revolution. It's coming to see that meaning and mind is not the original source, you know. In the beginning was the word. No, words are very recent. They've only been around for a few hundred thousand years. In the beginning was evolution and understanding, comprehension, explanation, and design, intelligent design. Is a very, very recent phenomenon, and it's not always that intelligent.
Allison Stanger: Well, you tell us that, and I think quite rightly, that both Darwin and Turing have discovered something truly unsettling to a human mind: competence without comprehension.
Daniel Dennett: That's right, yes. This is an idea that its time has come. I think people are more and more seeing how true this is. Let's start with a, with a simple example. A bacterium may competently move the right direction in order to obtain nutrients or to escape from a toxic element. It doesn't know why it's doing that, doesn't have to know why. It's just endowed. It is, it is gifted with a little bit of competence that it doesn't have to comprehend. A butterfly has spots on its wings that look like eyes, and it can scare off a bird by flapping its wings, and the bird may think this is a predator in the branch, not a, not an edible butterfly. The butterfly is clueless about that. Everywhere we look in nature, everywhere, and in ourselves, from the, from the motor proteins in ourselves on up, everywhere we look, we see these brilliant adaptive solutions to problems and reactions to things by things that don't understand what they're doing. And the nerves that send the signals from your retina to your visual cortex. They don't know what the heck they're doing, but they're tremendously competent. They send messages which are coded and decoded by decoders that themselves don't have to understand what they're doing. So in the end, we can see that understanding is not the beginning, it's the end. It's a very, very recent phenomenon. And we have an understanding that no other creature on the planet has— not the dolphins, not the elephants, not the dogs, not the wolves, not the chimpanzees. Our kind of understanding is not the beginning, it's the very most recent chapter. And if we don't use it better, it'll be the last chapter.
Allison Stanger: Well, that's— I just wonder if what follows logically from that is that today's evolutionary biologists and computer scientists have competence without comprehension, and maybe the same applies to you and me.
Daniel Dennett: In many ways it does, but we shouldn't get too frazzled about that. There's lots of good reasons for things that you do every day that you've never thought of. You don't have to. Maybe you do them because other people do them. It's just habit. But if you were to stop and get persnickety and recalcitrant and do things a different way, you'd soon discover, no, it was better to do it the way everybody was doing it. They don't have to understand the reasons for this. They just do it. Those are reasons that are endorsed by cultural natural selection, not by genetic natural selection.
Allison Stanger: You know, we haven't talked about memes at all, which you refer to as informational viruses that govern infectious habits. You see it— yeah, you see that on page 173. And words are the best examples of memes, you say further on, on 207.
Daniel Dennett: Absolutely, yes.
Allison Stanger: And you also point out that we're willing to die for meme complexes, that is, memes made of memes. Examples like freedom, democracy, and truth.
Daniel Dennett: Yes, and any number of religious memes that have inspired people to go to their deaths and sometimes take a lot of other people with them.
Allison Stanger: Yes.
Daniel Dennett: Memes. So memes, one of the— I think the great insight in Dawkins' concept of a meme is that it has its own fitness. I mean, a lot of people want to write about cultural evolution and about cultural items as being adaptations passed on by cultural, not genetic, means. Just like a hammer or an axe is like strong biceps. It's passed on not genetically but culturally. But it gives the owner a clear advantage in adaptation. But you have to understand that memes, some of them are that way. Some of them are very helpful. Some of them we couldn't live without. But some memes are absolutely parasitic and suicidal to us. I mean, they render us suicidal.
Allison Stanger: What meme? Give an example of that kind of meme.
Daniel Dennett: Well, the meme of suicide. Every now and then there will be these rashes of suicides where the idea catches on. Or how about school shootings? Or name your terrorist atrocity. These aren't good memes. Or spray painting graffiti.
Allison Stanger: You know, you describe memes as transmitters of culture. And I just wonder where power fits into this mimetic world, which can upend and refashion cultures as we see happening today with our own culture.
Daniel Dennett: Well, power is gained and lost by the acquisition of memes. You can't count There are plenty of deleterious memes out there. And I mean, I think that's becoming more and more obvious. There are people out there whose minds have been crippled, crippled by the memes that they have adopted.
Allison Stanger: So with that view, would ideology be a meme?
Daniel Dennett: Well, different ideologies are memes, yes. I mean, A meme, the best way to think of it is it's like a gene except it's not expressed. Its vehicle is not DNA, it is some cultural vehicle like a sound or a picture or a word. Or one of my favorite examples is a wagon with spoked wheels. It not only carries grain from town to town, it carries the idea of a wagon with spoked wheels from town to town. And that's how it gets there. That's how wagons spread. It's a good idea. And spray painting graffiti is probably a bad idea, all things considered, but it spreads the same way. People see it and they adopt it. And sometimes we adopt them without even wanting to adopt them, like new pronunciations of words. We don't even notice it. Or we see, we see the extinction of a meaning of a word and its replacement with another. Um, beg the question is gone. Farewell. It doesn't mean what it used to mean.
Allison Stanger: Yeah.
Daniel Dennett: Um, incredible doesn't mean what it used to mean.
Allison Stanger: That's true.
Daniel Dennett: When, when somebody says, oh Dan, that was an incredible lecture, and I say, gee, I'm sorry, I really wanted you to believe it.
Allison Stanger: That's true. Or something sick, and that's good. Yeah, there's a bunch of those things.
Daniel Dennett: Yeah, there's a lot of them.
Allison Stanger: Yeah. Much has been made in the media about your fierce atheism. And I have always wondered, why are you an atheist rather than an agnostic? How can you be so sure there is no God?
Daniel Dennett: Oh, I think it's plain obvious. I mean, I mean, unless you have a concept of God that's so watered down that— right, yeah. One time I was debating a right-wing religious radio host, and he said, you mean you don't believe in an eternal all-powerful force that makes the world possible? I said, oh, I do, I do, I do. He says, "You do?" I said, "Yeah, I call it gravity." I thought you were going to say love. No, gravity. It's more powerful than love.
Allison Stanger: Yeah? But you can't prove that's so. So why wouldn't you just suspend disbelief? This is like a Kierkegaardian attack on your consciousness.
Daniel Dennett: Look, I don't think it's worth arguing about. I think it's so obvious. Obvious that religions, they are the fables and musings of often illiterate, uneducated people trying to make the most of the world they live in. And they come up with these ideas. It's the most natural thing in the world. I have a whole section in, in, uh, Breaking the Spell about how I think the idea of God evolved. And I think it evolved from the idea of lesser gods, that is to say leprechauns, fairies, gremlins, all the other little invisible creatures of the world, the talking trees and so forth. And I think that what happened is that those are— they're all triggered by the hyperactive agent-detective device, as one writer puts it. When we're, you know, in a perhaps anxiety-provoking situation out in the dark and we hear a rustling noise, who goes there? Not what's happening, who, who. We want— we are first, and this makes evolutionary sense, If there's an agent out there, I want to know because he may want me. What do you want? And when the snow falls off your eaves and lands with a thud outside the window, your dog jumps up and growls, who goes there, who goes there? But he goes back to sleep. But a human being, it's going to spin around in their head because they got language and they got thinking, well, maybe there's a Maybe there's an abominable snowman out there. And pretty soon they're off to the races and it feeds on itself. So pretty soon every, every community has these, uh, local— a local menagerie of invisible creatures that everybody believes in because it's so cool to believe in them and so hard to disprove them. Those are the wild memes. Then they get domesticated by priests and kings and harnessed to improve the power of priests and kings and others. And I think to me that the, the story of the growth of religious doctrine is just transparent You know, I devoted, as one outraged critic said, less than 6 pages of Breaking the Spell to arguments for or against the existence of God. And the main reason for that was that when I asked people who were deeply religious, they said that wasn't the important thing. And I believed them.
Allison Stanger: Hmm. What was the important thing?
Daniel Dennett: Community. Love, helping. I've always liked the line from Robert Frost's poem: Home is where when you have to go there, they have to take you in. That's what religions do for many people, right?
Allison Stanger: But you believe in those things too.
Daniel Dennett: I believe in those things, and I don't know if there is a secular substitute that will do that thing. As well as religions do, but I don't want my religions to be doctrinal at all. I think all the irrationality— I think anything that valorizes irrationality is dangerous.
Allison Stanger: The clerics are dangerous, you know, not necessarily the religion.
Daniel Dennett: Not just the clerics, but the churchgoers who valorize irrationality.
Allison Stanger: Yeah, I like that definition. That's good. Let me ask you, and I'm taking up a lot of your time, and I'm grateful to you for it. I wanted to ask you about your view on artificial intelligence, where you say, "The real danger, I think, is not that machines more intelligent than we are will usurp our roles." our role as captains of our destinies, but that we will overestimate comprehension of our latest thinking tools, prematurely ceding authority to them far beyond their competence. That's on page 402, and I think it's great. Can you elaborate on that?
Daniel Dennett: Yeah, I think that's a real danger. I think people who worry about superintelligences and the singularity I think that's a distraction and, and a potentially dangerous distraction from the real problems, which are much closer to hand. We already have AI systems which can do a lot of harm by being granted too much authority. And it's a very tricky thing. Let's take medicine for a moment. There are now diagnostic systems in medicine Watson is geared up to become a virtuoso diagnostic system that can probably outdo any human diagnosticians. Well, now, what's the doctor going to do? I don't know about you, but I'm not sure I want a doctor who has the chutzpah to say, "I trust my own judgment over that of this machine." Mm-hmm. It is their own sense of responsibility that will lead them to cede authority, in some cases, to the machines. And that will be dangerous because it might be a runaway phenomenon. If we start ceding authority to our machines, then we'll have less authority to cede. And the next crop of medical students will not be anywhere near as good diagnosticians as the previous generation, because they didn't have to be. And pretty soon, what you've got is medical people who are basically valued for their bedside manner. They're glorified doormen at the apartment building, pushing buttons and wishing you good day and cheering you up. If that's what the life of the doctor becomes in a few years, who will want to be doctors? Yeah. Yeah.
Allison Stanger: Yeah, we need to be mindful of that. For researchers of human consciousness today, What are the big unanswered questions, and are they being asked?
Daniel Dennett: Oh yes, they're being asked by some of us.
Allison Stanger: What are they?
Daniel Dennett: Well, what I've been saying for 30 years is that there's no Cartesian theater, there's no homunculus in the center. But what, what's there to replace it? And what I've said is that all the work and play done by the homunculus has to be broken up and distributed among lots of lesser agencies, um, cell assemblies that are not themselves conscious but that can do some of the appreciating, some of the understanding, can do this. Now, how do you get close to a trillion cells to organize itself into a team that can do that. Uh, yes, the brain is a computer, but it's not a digital computer. It's not a von Neumann machine. And quite frankly, nobody knows what the operating system of such a computer should be. In fact, in a certain sense, it doesn't have an operating system. It couldn't. It's got to be controlled by a coalition of competitive collaborative agents, sub-agents, that don't even know who you are. But how that's done, that's, you know, one of my favorite lines about philosophy, but not just philosophy, is that one of the great foibles is mistaking a failure of imagination for an insight into necessity. I don't know how to imagine answers to this question. I'm working on it. I'm working on it every day. I have clues. I have glimmers. And there are people today, the Paris Group, Stan Dehaene and his colleagues, the Global Workspace Group, the Bayesian networks groups, the the predictive processing groups. I think they've, they've got parts of the answers. Trying to put it all together into something that isn't just hand-waving is the next step, and we're getting closer. It's very exciting.
Allison Stanger: Are you familiar at all with Geoffrey West's book on scale?
Daniel Dennett: Oh yes, I know Geoffrey and I've read his book. It's a brilliant book. Yeah, that's one of those great Santa Fe ideas. Mm-hmm.
Allison Stanger: It's definitely a complexity idea.
Daniel Dennett: Yeah, yeah. No, I think Geoffrey's work is on my list of things. I've got a bunch of authors whose work I'm trying to incorporate into my own thinking, and he's one.
Allison Stanger: What would you like to be remembered for most? It's a hard question, but I can't resist asking it. You've had such an illustrious career, so many contributions. What are you most proud of?
Daniel Dennett: I guess for helping to solve the problem of consciousness.
Allison Stanger: That's a small thing.
Daniel Dennett: With an evolutionary— by bringing evolution, I just keep discovering more ways in which Darwin's idea is the best idea anybody ever had. And it's funny how many people hate that message. I mean, people who aren't religious, but they still hate it. Really unsettling to many people.
Allison Stanger: Yeah, it seems like cognitive science is increasingly backing you up.
Daniel Dennett: Oh yeah.
Allison Stanger: Yeah, which is fascinating.
Daniel Dennett: Oh yeah, I'm— I'm getting to say, "I told you so," in something like that.
Allison Stanger: Well, enjoy that. That's a good feeling.
Daniel Dennett: Well, yeah, but it's mean. It's an ignoble sentiment. So I try to bite my tongue.
Allison Stanger: It's an exciting time. Let's just call it that.
Daniel Dennett: Let's call it that. Yeah, I've been— after all, I've been working on that since 1965, so—
Allison Stanger: You deserve a little feeling of satisfaction, I think. Final question. 3 books that you would consider your desert island books. Or 3 books that you've recently read that you would recommend to other people.
Daniel Dennett: Well, Daniel Dorr's book, The Instruction of Imagination. And Jeffrey Westbrook's Scale is a very— Then put that right up there. And, um, well, if they haven't read it, they should read The Origin of the Species. It's beautifully written. Yes, it's a page-turner. It's wonderful. And you get, you get to know the man in his humility and his brilliant, his enthusiasm and his In some ways, innocence. Oh, can I add a fourth book?
Allison Stanger: Yes, you're allowed.
Daniel Dennett: Richard Dawkins, some years ago, put together a wonderful volume that very few people have read. It's called The Ancestor's Tale. And it starts at us and it works back through the tree of life going from human beings to chimps and bonobos, and it goes back all the way to the beginning of life down our particular paths. It's— and of course it's based sort of on Chaucer's Canterbury Tales in a way. And it is— interesting— it is full of wisdom and brilliant sketches of amazing life forms. It will enrich your sense of the planet. It's Richard Attenborough in a book. Yeah, I highly recommend it.
Allison Stanger: Yeah, that's wonderful. Well, thank you so much for a really stimulating, wonderful conversation. I really enjoyed speaking with you.
Daniel Dennett: And we didn't talk very much about CASBS.
Allison Stanger: Should we talk some more about CASBS? Please tell us about Lake Lagunita, your distress about Lake Lagunita.
Daniel Dennett: I'd been at the center for several months and there was this sort of ugly brown patch of ground down on the Stanford campus down below my hilltop window. And one weekend or so, one Monday I came in, it was filled with water. Winter rains had come, and not only was it filled with water, but there were Stanford students out there swimming and windsurfing. This was the beginning of the windsurfing age, and I couldn't take my eyes off them. And I'm a sailor from way back, and I I went down the hill on my bicycle and got over there and found out, hey, I could rent a windsurfer. I had a Stanford faculty pass because I was at CASBS, and I was going to come back the next day. So I went, I told John McCarthy, I said, tomorrow I'm bringing a swimsuit and I'm going to go down and learn how to do that. And he said, 'Do you mind if I come along?' And I said, 'No, no, come along.' And he sat there on a bench on the shore while I fell and fell and fell and fell and fell and fell and fell for probably an hour and a half. And then I paddled back in, and John said, 'Well, I guess you got that out of your system.' I said, oh no, I'm going to be back tomorrow. And I did. And the very next time I went, I got it and became a competent and even somewhat comprehending windsurfer. So windsurfing was one of the memes that I was infected by at CASBS and It led to hundreds and hundreds of hours of joy.
Narrator: Thanks to Daniel Dennett and Alison Stanger for a great conversation. And if more great conversation is what you're after, head over to our website at casbs.stanford.edu. Or you can find us on Twitter, we're @casbsstanford. As always, we've got links in the episode notes for those of you following us in your podcast app of choice. And for those of you who don't, there's a whole world of interesting topics we explore, sometimes one-on-one with individual fellows and sometimes engaging panel discussions with diverse voices. So fire up that podcast app on your phone, search for Human-Centered, and give us a follow. You'll be glad you did. Until next time, from everyone at CASBS and the Human-Centered team, thanks for listening.