Dan Kelly, 2018-19 CASBS Fellow, is an associate professor of philosophy at Purdue University who focuses on the intersection of philosophy, cognitive science, moral theory, and evolution. Dan chats with host John Markoff talk about implicit and algorithmic bias, social norms and morality, and ethics in technology.
Dan Kelly's book “Yuck! The Nature and Moral Significance of Disgust”
Two fascinating papers coauthored by Dan:
“Minding the Gap: Bias, Soft Structures, and the Double Life of Social Norms”
“Who’s Responsible for This? Moral Responsibility, Externalism, and Knowledge about Implicit Bias”
CASBS’s project on Creating a New Moral Political Economy
Center for Advanced Study in the Behavioral Sciences
@casbsstanford on twitter
John Markoff: From the Center for Advanced Study in the Behavioral Sciences at Stanford University, this is Human Centered. I'm John Markoff. Today we talk with Dan Kelly, a 2018-19 CASBS fellow. Dan is an associate professor of philosophy at Purdue University who focuses on the intersection of philosophy, cognitive science, moral theory, and evolution. We'll hear Dan's thoughts on cognitive bias, social norms, and ethics in technology. You know, after our last conversation, one of the things that I, I wanted to start with actually is your path into philosophy. Where did it start?
Dan Kelly: It started that I was one of those kids who lived inside their own heads and overanalyzed everything. Yes, so I'd never taken a philosophy course. I was always sort of a voracious reader and, you know, academically, like, interested in a lot of things. And I got to my orientation at college and I was interested in philosophy and I tried to sign up for a philosophy course and it was all full. But the guy next to me in line ended up being a dean, and he, he signed me into the class, like, to overflow the class. So I took this first class, I loved it, I did well, I picked up a major. And then when I was trying to figure out what to do with my life, I was still sort of overthinking everything, and I just decided that it would be good if I found some way to organize all of this hyper-analytic tendencies I had. So I went to Tufts, and I sort of did really well in their MA program and got into Rutgers, which was a great PhD program, and here we are.
John Markoff: And the cognitive science thread, because I mean, there's philosophy, and then, but you sort of focus on cognitive science kinds of issues to an extent.
Dan Kelly: That's right. So I was also an undergrad, I was also a major in English literature, so I've always been fascinated by people and what makes them tick and why they can be great sometimes and horrible other times and sort of the whole human condition thing. And I got to Tufts where Daniel Dennett is, and he He does philosophy of mind and cognitive science, but kind of with a very humanistic face to it. And that just sort of lit up my entire switchboard. And that's where I got really interested in that. And then at Rutgers, cognitive science is also one of their main strengths, and I just kept going down that pathway.
John Markoff: So before Dennett, as an undergrad, did you find yourself identifying with any particular thread in philosophy? I mean, you know, like, were you an existentialist or a humanist?
Dan Kelly: Yeah, I was in philosophy. At the same time I was doing philosophy, I was taking all these classes on Shakespeare and, you know, modern literature and British poetry. And so oddly, the stuff in philosophy that I really liked was the philosophy of science side of it. Sort of how science has given us this tool to understand nature and what the virtues of it are. But then there's a thread in there about naturalism and the role that the theory of evolution can play in sort of giving us a scientific understanding of human nature. So the literature is giving us one way of understanding humanity and our place in the world, and evolution is giving us another one, which people seem to think is not compatible with the one coming out of a lot of the humanities. But I never bought that. I always thought that they were sort of consistent, and we just need to think harder about how they can mutually inform each other.
John Markoff: And what's your current project?
Dan Kelly: Yeah, my current project is I'm writing a book on culture and human nature, and specifically human morality. Moral nature, so how our sort of cultural capacities have allowed us to become extremely social and the way it's allowed us to build more and more sophisticated social organizations and how that is sort of inflected with all forms of different kinds of morality and then how that shapes our very identities.
John Markoff: And you're also here on— you're at CASBS visiting, working on a different project, a paper you're working on with— collaborating on?
Dan Kelly: Yeah, so it's an idea that we've had for a couple of years. We've sort of batted it back and forth and this is Michael Brownstein. He's a CASBS fellow this year. And then the working title of the paper is The Ethics of Consistency. So we're thinking through how this value of consistency, which you might think of as primarily a logical value, like different sentences or different claims can be compatible or incompatible with each other, is also something we prize in people. Like if someone is consistent over time, or if the different facets of themselves are consistent with each other, when that's the right way to evaluate them, or when using that as a standard of evaluation can actually lead us astray from a moral point of view.
John Markoff: So the obvious question is, where are you guys on the question of moral progress and whether we are making—
Dan Kelly: what did you guys conclude? So I think there's a pretty good case that along a lot of sort of global measurement dimensions, moral progress has been made in the last couple of centuries, maybe the last couple of millennia. Child mortality rates are down. There's not a country in the world anymore in which slavery is legal. That doesn't mean it doesn't happen anymore, but there's progress that it's not even like countenanced as being legal anymore. Violence, depending on how you measure it, may be down. On the one hand, I think there's good reason to think that moral progress has been made on some dimensions. That doesn't mean it's perfect or that we're all the way where we want to be. Part of what we raised there was more thought needs to be put in, from philosophers anyway, more thought needs to be put into this question of how, how sort of collective level advances in moral progress might come at the expense expensive, more personal-level values that have a moral valence to them. So the sorts of community and meaning that really make lives worth living. A lot of the complaint you hear is, as the world gets globalized and all this, you know, whatever sort of institutions allow the sorts of higher-level or more abstract global kinds of moral progress might come at the expense of human, very sort of human day-to-day kinds of ethical values. And thinking about the trade-offs between those two and if we can get both of them at the at the same time is something which I think really deserves more thought.
John Markoff: Where does your thinking line up with— I mean, I guess, sort of from a popular point of view, this has been framed by Steven Pinker for most of us. Do you break with Pinker on any points, or—
Dan Kelly: I think he's a little too optimistic, and he focuses too much on what Michael and I thought of as these sort of higher-level, more abstract ways of measuring what moral progress might amount to. I know there's all kinds of criticisms about even the way that that stuff has been measured You know, maybe there's been stacking the deck in one way or another with all the statistics, and I just don't take a stand on that. But it's the lack of focus which is paid to a plurality of values, some of which are much more, again, sort of based on the day-to-day characteristics of a life and what emotional and moral well-being amounts to for a person rather than a corporation or something which can be captured with the statistics over the course of different generations.
John Markoff: You know, one thing that occurred to me that you would have an interesting perspective on— you came from, you grew up with a generation which I think is when you were in college, 20, are you 20 years out of college?
Dan Kelly: Yeah, that's right, just over 20.
John Markoff: So it's about a generation and, and you know, I ran into this summer because I was back to, I went back to a summer camp that I had attended as a kid and I'd worked there and so I ran into the young staff who were in their 20s who were probably I was thinking they're probably about the age of the students you're teaching now. Right. And I was stunned. The particular thing I ran into was the issue of cultural appropriation, which is a subject we don't need to go into, but they were so different as a generation, and I'm just wondering your view about the students you're teaching versus the students you went to school with in terms of the kinds of things you study. Is it really dramatically different?
Dan Kelly: Yeah, it's a great question, and it's after not having to teach last year and going back this year. I don't know if it's just because I was away for a while or if there's— there is, you know, we're nominally on the cusp of the shift from millennials to Gen Zers and the new generation. So this year's students are digital natives. There was no time at which they weren't online. And if there was, if there was sort of a tipping point on these sorts of sensibilities. So cultural appropriation is a good one. They seem to be much more sort of aware of group membership and identity and ways in which people from different groups can help each other out or be offensive to each other, and in a way which I just was certainly not when I was in college. It was something I went to a fairly homogenous— not homogenous. There wasn't a lot of diversity at the college campus I went to or even the high school that I went to. But we also didn't have the internet, which was bringing perspectives from all over the globe just sort of at your fingertips. And so I think this is a generation which being on screens since they've been awake basically makes them aware of cultural differences and that we need to think more carefully about how we navigate those differences in a way that was unprecedented when I was young.
John Markoff: I overheard you use the term "homonormicus" while you were here at CASBS. I'm curious, what is homonormicus?
Dan Kelly: What is homonormicus? It's a word I came up with. So, okay, the idea here is something like this. There's an image of what humans are and how humans make decisions and what drives human behavior, which has become baked into a lot of our theoretical imaginations, particularly in, in the West. But it— call it Homo economicus, and it sees humans— it's sort of this, this image of humans as behaving in ways such that they're trying to maximize self-interest. And the conception of rationality at the heart of it is sort of an economic conception of rationality. And that this is basically what drives human nature, or sorry, human behavior, and it's something, you know, at the heart of human nature. And I think it is, and I'm not alone in this in any way, shape, or form, but it's a vast simplification. So it's in part right, and you know, it gets something right about humans and human psychology, but it is very much incomplete, and to the extent that we rely on it too much, we get a distorted picture, and we build policies and institutions for humans which aren't aren't sort of sensitive to and harnessing all of our best impulses. So human normicus is this, uh, is an idea. So I do a lot of work on the cognitive psychology of social norms, uh, and human normicus is just, you know, if you need something which is catchy and is sort of a contrast class to human, uh, to homo economicus, homo normicus would be something like, well, another really key part of human psychology and human nature is that we respond to social norms and we cognize social norms, and this is a very large part of what drives how we behave and how we how we understand ourselves and how we interact with each other. So humans are able to cooperate on these sort of scales which are unprecedented in human— sorry, in nature. And what, you know, we can build cities and we can send people to the moon and we can build internets. And part of what allows us to do this is we're extremely sensitive to norms, to rules, which govern social life. And when people violate a norm, then other people sort of trans— they punish them, but they tell you how to go about your day, and they help us. I mean, some morality— moral norms are a key part of moral norms.
John Markoff: And were these concepts and thoughts sort of how you were drawn into the moral political economy project here at CASBS?
Dan Kelly: Yeah, so the moral political— building a new moral political economy project— I got pulled in via some conversations with Federica and some other members of that, that group, because I wrote, I wrote something about the plurality of human values and the cultural variability of human values. And so you might think that human homo economicus is an expression of human values and what humans value, which is very much centered on individualistic Western picture, and it's at the heart of the kind of political economy, which has dominated for a while, which comes out of Adam Smith and a little bit more Thomas Hobbes. And, and this connects back to some of what I was talking about with the moral progress stuff. I think those sorts of values and those sorts of norms are one set of norms, and they give us a certain kind of good, and they allow us to understand human, human nature and human morality in one way, but it's only one facet of a much larger and more diverse picture. And so human— the idea of Homo normicus is give us a picture of human psychological nature which allows for that sort of variability and which can accommodate a wider range of values that we might want to try and build a moral political economy such that it can accommodate and be sensitive to a much wider range of values.
John Markoff: I wanted to ask about your, your earlier work. I think it was in 2011 you wrote a book about disgust. That's a wonderful title, Yuck. Did you plan it for a popular audience, or did it have that kind of an impact? Did you reach a popular audience with this discussion of this kind of—
Dan Kelly: um, it's very easy to connect to, right? So everyone is familiar with a first— you don't need any theoretic background to know what it's like to be grossed out by, like, stepping in dog crap or something. Um, so on the one hand, there's, there's this easy connection to something everyone can conceptualize. Yeah, the title, it's almost like a Malcolm Gladwell-y title. It's a single word with an exclamation point after, which may have been a little misleading. So I tried to write the book in a way which was accessible for anyone willing to put in a little bit of work. You know, it's a little bit more in the trenches than a New Yorker article or something. But I very much wanted to reach an interdisciplinary audience. And so you can't— when you're doing that, you can't just assume assume familiarity with technical lingo that's, you know, from one particular discipline. So I didn't write the book, to answer your question explicitly, I didn't write it aiming for a popular audience. But I wrote it in ways which I could see it sort of tipping in that direction. And so did it reach a popular audience? You know, I remember seeing it on the bookshelf at the Strand in New York, and I considered that a huge win. So, you I think it was read by some people who weren't merely academics. Maybe they got, you know, sort of false-advertisingly seduced in because of the title was a little bit sexier than otherwise. But it got reviewed in places beyond just the professional journals. So it was reviewed in New York Review of Books and the Times Literary Supplement. So I think that's part of why, if people who read it weren't just in the academy, that was part of it.
John Markoff: You explored the evolutionary roots of this notion or this quality, and I was wondering, did it evolve? And I mean, if we had looked at disgust a century ago or two centuries ago, was disgust a cultural thing?
Dan Kelly: Right. So part of the reason I got fascinated with disgust is exactly because I think this is a— it's a fair question and it's one which doesn't have an obvious answer. So there's people out there— I mean, I know a historian who thinks discussed was sort of culturally assembled about 300 years ago. I very much disagree with that view, but I can see that there's— it's not sort of totally beyond the pale. And so my view on disgust is it did indeed evolve. It's an emotion, or it's a cognitive mechanism, or it's a piece of human psychological nature, which is unique. So I think you can see the component parts of disgust. You can see it analogues of them in other— our primate ancestors. But it only sort of congealed in— and part of the book is a story that I'm telling about how this came to crystallize in the human mind but didn't crystallize in other animals' minds. And what I think it is, is in part it's a psychological response to a pair of threats. One of those threats is eating things which are poisonous. Eating things which are going to disrupt your gastrointestinal system. And part of the reason that's important for humans is because we're omnivorous, so we don't just eat eucalyptus leaves and we don't just eat other— you know, we're not like dogs where we have a scavenger's gut, which is you can eat anything, which is nasty, and it's just going to neutralize us. We can eat a lot of things, we inhabit a lot of different environments, but some of the stuff out there which is potential food is going to mess us up. And so what disgust is, is it's sort of a way to learn and prevent us from eating this stuff, which is going to mess us up. The other side of disgust, and one of the two sort of core components or main directives of this emotion, is it's the first line of defense for our immune system. So it's not T-cells, you know, that fight microbes which have already infiltrated the body. What disgust is, is it's a sensitivity to the potential for infection and contagion in the environment. So it's part of our behavioral immune system rather than our sort of internal microbial immune system. And so things which are disgusting are, you know, other people who are obviously sick. Like, we just have an avoidance response and kind of a visceral reaction to someone who has pussy sores on their face. And so what disgust in humans is, is it's sort of the melding of both of these, of the machinery which protects us against both of these threats into this emotion, which has this very recognizable facial expression, the sort of gape face or the yuck face. There's reason to think— I think there's good evidence that that's a cross-cultural universal. So that's like, you can go to another culture, and if you make that face, they'll know immediately that, like, you're disgusted by something, and they'll probably want to know what it is that you're disgusting— disgusted by, so they can avoid it as well. It's a very useful and clear signal, sort of social signaling. And then some of the things which are disgusting are also universal. So phenotypic abnormalities is one way to talk about it. That— but then the sort of rough draft of the idea of what's universally disgusting can get specified and calibrated and made more precise via cultural information. So there's also a wide range of cultural variation in what's disgusting. And so there's a role for for culture to sort of take this piece of our psychology and shape it in ways which will make it more effective given the environment that the person lives in.
John Markoff: When I was in New York, when I lived in New York, I used to swim in this very crowded swimming pool. There were 8 people in my lane. It was just a pain in the ass. And there was at one point where I was stuck for a couple weeks, it seemed like, with a guy who was swimming with only one leg. And I had some sort of visceral reaction. I never really thought about it, but I mean, it was clear. Where did that come from? Where? You know, I just— it just was hard to deal with.
Dan Kelly: I think that's exactly it. It's things— and so what is normal phenotypically, that can vary from culture to culture. So it can even be something like skin color, that, you know, people around here look like this. Here's someone who doesn't look like this. And that shapes this particular response. And it's just this weird aversion. It's also one of these pieces of our psychology, which seems like it's in weird ways immune or recalcitrant or sort of compartmentalized off from what we think explicitly, from what we know. So, you know, the famous examples like this is, you know, chocolate or fudge, which is shaped to look like dog turds. There's a sense, you know, there's a sense in which people like, you know, that it's just fudge, like that the conscious explicit part of your head knows that there's nothing disgusting about it. But just knowing that can't totally shut down the aversive response you have. So I'm guessing you probably had something similar with this, with your experience, where I know there's nothing wrong here, but that just knowing that doesn't shut down the sort of this lower-level piece of our emotional repertoire.
John Markoff: I also wanted to ask you about methodology. You sound, in talking to you, you sound like a social scientist, but you're not. You're a philosopher. So what was the methodology? I mean, how do you describe the methodology for doing the research for Yuck?
Dan Kelly: So my home base for me as a philosopher is very much, it's on— I'm a consumer of a lot of social science and cognitive science. To some extent that I'm making theoretical claims, like I claim that disgust is a unique part of human psychology and that other animals don't have it. So that's, that's an empirical claim, right? But the, the argumentative basis there isn't just reflection on concepts or something. I'm, I'm trying to synthesize what we know from a number of different channels in the behavioral and cognitive sciences. So sometimes, you know, people who are working in the trenches and doing, doing the very hard and important work of, you know, experiments and crunching data and publishing studies, um, don't, don't have the time to take a step back and make the larger claims or try to fit all the pieces together. Of course, they do do that sometimes as well, but that, to the extent that that's part of the project here, it's again, it's very much I'm trying to work from an evidential basis to make theoretic claims having to do with the structure of the mind and what human nature is like. I also, in the book, at the end of the book, I go on to draw some, some implications for different disputes in moral theory. So not how the mind is, or not what disgust, not what disgust necessarily does, but what it should do, or how we should treat our feelings of disgust, given what we know about, like, descriptively and explanatorily about this piece of human nature. Right, so there's these debates, sort of large debates in philosophy about the role that intuitions should play in our construction of moral theory, like what, like our sort of gut responses to various cases, whether or not we should give them moral or epistemic credibility. And so the specific one was this question of if you are disgusted by something or if a wide range of sort of the population is disgusted by something, so a practice like same-sex marriage or human cloning or stuff like that. Should, should the disgust response or the fact that a lot of people have feelings of disgust toward the practice count in favor of thinking that it's morally wrong? So what should our moral theory say about it? And the position I took on that was I just sort of took the most extreme position and saw how far I could defend it. And the position was absolutely not. Disgust is totally irrelevant from the point of view of morality. And then to the extent that we can, we should sideline it. We should minimize the role that it has in figuring out what position we want the law to take on these practices. We should sort of set that, set that aside. So it's a piece of human nature, but it's not one we should celebrate when we're trying to do more reflective morality. And we also know that disgust tends to dehumanize its object. When its object is other people. And now there's, I tell a story about why that is, and I just sort of make the next move and say, oh, by the way, we shouldn't use disgust as a social tool because it dehumanizes people, and we don't want to dehumanize people, obviously.
John Markoff: The other thing that's happening at Stanford and in many academic centers around the country now is this real intense engagement with technology and ethics. It's become part of the curriculum there. I like to say that there are ethics courses littered around Stanford campus now. You apparently have been engaged in ethics and technology at Purdue. Do you have a perspective— well, two questions, I mean, many questions actually. So, well, let me get into it in one way to ask you about one of the things that Harvard's done, the computer science department. There is now, in every CS class at Harvard, there is a module taught by a graduate student from the philosophy department.
Dan Kelly: Oh, that's great.
John Markoff: Which is really interesting, so that people are forced to engage themselves with, with this thing. It's just happened. Is it going to make a difference, I guess, is the question.
Dan Kelly: Yeah, yeah. Is it going to make a difference? I, I mean, I wish I knew the answer to that question. You certainly hope so. Purdue's doing similar things, so there's you know, there's data mine stuff happening and there's a movement from the engineering, which is the huge part of Purdue, from the engineering college that no one graduates with a degree without having taken some ethics courses. And then how to manage that and how much the philosophy department is going to contribute to it is also in the mix. But we've been thinking very hard about, as a department, how to teach those courses in a way which will make a difference. And I think that to the extent that there's promise there, it's that we're learning that the right way to teach those classes is not the way you would teach those classes to someone who's a philosophy major or someone who's just taking an intro to ethics class. And we're changing them so that they're very much case— it's case studies rather than just here's utilitarianism, here's the Kantian ethics view, and like at a very abstract level. And so we have classes where people are working through through, you know, cases which went afoul of the sorts of moral and ethical codes that you would expect govern how you build a building or what sorts of, you know, how you write code such that it's not biased one way or another. And I think that has the promise to be more, to actually be effective, that you're getting kids to think through the ethical issues on their home turf, essentially. And it's not the way philosophy classes often get taught, but we're We're hoping that this, you know, adapting to the times will provide some fruit.
John Markoff: Have you had the chance to engage directly with engineers or software engineers about algorithmic bias in particular, or—
Dan Kelly: I was on a grant on, I think the title was algorithmic bias, a couple years ago with the PI there was someone in the computer science department at Purdue, and you know, we had a lot of conversations and It got me thinking about this stuff a lot, and there's, there's a growth industry, a small growth industry, but I think it has legs in philosophy of people doing philosophy of information and algorithmic bias and oppression as through various data structures. It got me thinking about how— so that's not my specialty at all, right? So part of where my efforts got devoted was, we need to hire someone who does this, like, this is their area of specialty, not just competence or anything like that, to help us think through this. But part of what that got me thinking about from sort of my own research home base in human nature was there's this, there's this intuitive distinction between sort of intelligence and wherewithal, which is natural versus artificial. And I think that distinction is untenable, actually, because I think, I think human intelligence from the get-go, what makes us different from other species, how we, how we got here from an evolutionary point of view, is that we externalize our intelligence. We build gadgets which solve problems, and, you know, we build technology, and that's— we have technological minds, and the boundary between what's inside and what's outside is extremely porous. But I think that distinction between the artificial and natural intelligence feeds into this fallacy that is that is still with us, that we tend to grant what we think of as artificial intelligence or algorithms or anything which has like a mathematical basis way too much epistemic authority way too soon. So a good example of this would be someone who's driving around in their car and they have their GPS telling them what to do, and instead of like trusting the evidence of their senses, they just turn when the GPS tells them to turn. And you know, there's all kinds of cases where people drive into lakes or through like the front door of a building or something. And so they've just sort of outsourced this, the problem of navigating, to a GPS, to an algorithm basically. And then they stop thinking about it. They just think it's got to be right. You know, it's objective or it has some sort of epistemic authority, which we don't have. And I think this has happened on a societal level already to some extent. So you see places where recidivism rates are calculated by algorithms and then just sort of given this sort of trust, like it must be right, it's math, it's algorithm, it has this objectivity which humans don't have. But we know that there's already, there's bias baked into the datasets that they're learning from. The algorithms can just amplify that bias and propagate it in various ways. So that's my interaction with people thinking about algorithmic bias and the sort of thing, that's where it sort of led me me on the path that's led me down.
John Markoff: I really want to ask you about machine autonomy. I mean, because I tend to worship at the church of a computer scientist whose name is Ben Scheidemann, who actually argues against machine autonomy for ethical reasons. He argues that the danger in autonomy, in building these autonomous systems, is you separate the machine from human responsibility.
Dan Kelly: Totally agree.
John Markoff: But that's not a popular perspective yet. I mean, we're still I mean, in Silicon Valley, there is still a lot of investment in building and, you know, setting these machines free. That is part of the culture of Silicon Valley. And I think that perspective that— so I guess then, so how do you think about embedding an ethics in a machine? I guess, I mean, in the end, they're human ethics, right?
Dan Kelly: But yeah, we want them to do what we want them to do, not what they want to do. Yeah, it's, it's a, it's a thorny issue. I totally agree. But one, one of the things you said resonated with me, that we just want to set them free. But we also— there's this way of thinking about what autonomy is. Is it something that we can just build and then sort of impute in them? Like, you know, if we just make the algorithm work the right way, or if we built the, build the sensory apparatus and the motor the right way, then they'll, then they'll have autonomy. It'll just emerge from what they have. But I think a better way to think about that is autonomy is something that we collectively grow grant, right? So this is how we treat people. Like, you know, there's little kids, and they don't really have legal autonomy until they're 18 years old. And then once they have this autonomy, well, then a certain set of duties and a certain set of responsibilities are sort of granted to them. And then they're held accountable when they don't comply with those duties or responsibilities. And I mean, this is vague and big picture, but my sense is that that's got to be the way that we're treating whatever the autonomous vehicles or whatever autonomous algorithms or AIs are, is that they don't have autonomy till we give it to them. And right now we're giving it to them way too soon. There's this other question of whether or not that's the ideal that we should be aspiring to. This might have been what you were speaking to, is that embedded in Silicon Valley is just, we want to build this stuff and then turn them loose into the world. That's what you said.
John Markoff: That's like Frankenstein. Yeah, yeah.
Dan Kelly: So yeah, in some sense, is that really what we want to be doing is just to turn them loose? I mean, what we want to be doing is not have to think about the stuff that we want them to do anymore. But I think that's just—
John Markoff: so Schneiderman makes this distinction that I really like between autonomy and automation. And automation is, you know, within the framework of human control, human-in-the-loop kinds of ideas, you can still automate tasks, right? The human is— that's his model of thinking about it.
Dan Kelly: I, I love that. Yeah, yeah, it seems like it's pointing to a really crucial distinction, um, right? And again, if we're trying to build things which are automated versus autonomous, and whether or not we want to grant anything complete autonomy, are, yeah, connected questions. But I, my sense is that, look, we have all these norms in place, we have all this, this ethical, you know, a buildup of centuries and millennia worth of ethical apparatus that we use to, to to assess and keep each other in line. We have all the social technology of morality. And what we do when we raise a child is we sort of induct them very slowly into it before we treat them as an autonomous agent who's going to be governed by these norms. Why think that it's going to be any different with AIs or other autonomous kinds of technology?
John Markoff: That's a good analogy. As a philosopher, have you been dragged into the trolley problem?
Dan Kelly: Oh, of course.
John Markoff: Of course.
Dan Kelly: Yeah, yeah.
John Markoff: I hesitated to bring it up, but it's, I mean, it's the prism through which we've looked at self-driving vehicles.
Dan Kelly: Right, right. And so there's a paper that came out a year or two ago that looked cross-culturally at different ways that different cultures respond to the trolley problem and different variations of the trolley problem and everything. Yeah, I mean, I don't, my opinion on the trolley problem, as a philosopher is that, you know, save 5 lives versus save 1 life. It's you always go for the 5. But how to implement that in the sensitivities or what people are sensitive to when they think there's important differences between variations of the trolley problem. So, right, if you're just flipping a switch versus pushing someone off a bridge or how to build that into various AIs. I mean, there's not even consensus amongst humans about what the right thing to do is in the various cases. How are we going to deal with that?
John Markoff: The way you framed it, I just realized that something I've struggled with. So there's, in the artificial intelligence and autonomous weapons debate now, the principal argument in favor of them goes back to about 20 years ago. I did some reporting with one of the engineers, he was at SRI International, who had been one of the designers of the first smart bombs. He was French, and he explained his personal ethical rationale for what he did. It was because he grew up in a French village. His village, as he described it, had during World War II been subject to both Allied and German bombing. They were trying to destroy a bridge in that village. They spent the entire war trying to hit the the bridge, and they hit everything in the village but the bridge. And so his argument was, his justification for designing smart weapons is, if you can kill the bridge with one bomb, that is morally preferable. And I've struggled, I can't get it out of my head because, I mean, you know, I'm raised as a pacifist basically. Yeah, yeah. But obviously that he's dealing with a real-world phenomenon, and people have transferred that argument into a defense for AI-based weapons. AI weapons, you know, will be more intelligent, and they will also, the argument is, not commit war crimes. You could design an AI so that it wouldn't get angry or—
Dan Kelly: Right, right, right. More surgical, more precise, less collateral damage. Oh, that is a great argument. So that was in context with the trolley problem that made me think of this this weird disparity, which may— the way people respond to these may be informed by this. So with the trolley problem, you have these two cases where one is you can, you can push someone off a bridge and it'll stop the trolley, and so one person dies, but it stops the trolley so the five other people don't die. And another one is where you're sort of off distant from the action, and you can, you can flip a switch and divert the trolley from hitting the five people so that it goes on to another track and you hit one person and they die. And sort of the common response is that it's okay to flip the switch, but it's not okay to push the person off the bridge, even though from a numbers point of view, you're exactly the same thing is happening. And the rationale that a lot of people give is that there's something about— well, an explanation for this divergence in response is that there's something morally icky about the up-close-and-personal pushing of something off the bridge, but then it's it's okay if you're just sort of using technology to intervene and if you're far off. And so that licenses this use of technology to do something. On the other hand, a lot of people have this sort of icky kind of yuck response to the idea of drone warfare or of like technologically mediated interventions and killing like this, which is strange because they thought it was okay in the one case but not okay in the other case. Yeah, so I think maybe the one conclusion to draw from that is is that people have incoherent views about how technology should be used in these sorts of cases and in warfare, when you know you're intending to bring about death one way or another, or destruction, I guess, in one way or another. And the argument that your friend gave is maybe one of, it's giving really good voice to some of the contradictory impulses that people have behind these. Yeah, it's, oh, I'm gonna wrestle with that for a while now.
John Markoff: Finally, I got the sense that Silicon Valley, despite being beat on culturally and politically in the last— remains, I think, largely a community that's techno-optimistic.
Dan Kelly: Yes.
John Markoff: I got the sense that maybe you're not so much. And that I know people now, it's fashionable to talk about unintended consequences. So they've made a step to understand that you have to think about things. Take everything away, they still— it's a hammer and nail kind of situation. If you have, you know, like, now it's AI. AI will solve all problems. It's sort of the sense of it.
Dan Kelly: Uh, I, I think on the one hand, I used to be just sort of blatantly techno-optimistic, um, and just figured, you know, we'll have problems and then someone will be able to make a fortune by solving them, and so that'll induce them to go out and solve them. Um, I just wonder now if we're like problems are on a scale at which we don't even understand them that well, and they require not just what we usually think of as technology, you know, building new gadgets and having scientific breakthroughs, but what they're going to— like climate change, for instance. That requires not just advances in science and sort of physical material technology, but we clearly need some sort of advance in our political technology as well. It requires us to like take political action, and it's a collective action problem, which a new gadget or a new like, you know, way to— well, we need that as well. And that, and my understanding of the current state of affairs is what's maybe dampened my enthusiasm for sort of purely technological solutions, is that again, the scope of the problem is just enormous. That said, I kind of do have faith in like the cultural evolutionary process of maybe no one individual mind or one individual valley is going to come up with things which are going to be game changers, but you know, we are all working together and we're all connected to each other in a way which is unprecedented in human history. So maybe, maybe collectively we'll be able to do something that we hadn't been able to do in the past and at a speed speed that is going to allow us to address those sorts of challenges in the timeframe that they need to be addressed.
John Markoff: Yeah, that was one of the original framing arguments of Silicon Valley, actually. Doug Engelbart, who was one of the original pioneers in '62, had this notion of intellectual augmentation. He set out to build tools that would accelerate the collaborative abilities of small groups of intellectual workers. I mean, that was the founding principle of Silicon Valley in some ways. Certainly was the founding principle of the ARPANET.
Dan Kelly: It's almost the founding principle of this place, right? That's true.
John Markoff: That's right. Yeah, yeah, yeah. That's absolutely true. You're right. I hadn't thought about that. That's a good point to end on. Thanks for spending time with us. I'd actually love to continue the autonomy question with you separately. I mean, as a reporter. So maybe we can have that conversation again. To learn more about the topics in this episode, check out the show notes. There you can find links to works from our guests and relevant articles, including Dan's entertaining and engaging book, Yuck! The Nature and Moral Significance of Disgust, by MIT Press. Human-Centered is a show from the Center for Advanced Study in the Behavioral Sciences at Stanford University and is produced by Michael Jitani, and Joseph Monzel. I'm John Markoff. Thanks for listening.