WordPress.com

 Risky BusinessBarry: It isn’t very often that the Supreme Court gets a chance to put a number on how many innocent people the police are allowed to arrest. But they’ve come close. One time involved a 1999 case.Its 3:00 in the morning and three friends were headed to a party in Baltimore. Donte (Partlow) was driving, Joseph (Pringle) was riding shotgun, and Otis (Smith) was in the back seat. Donte is pulled over for speeding and agrees to let the officer search the car.Tape: And found drugs packets for distribution.Tape: Five glassine baggies of cocaine.Tape: A large amount of cash in the glove compartment.Tape: After all three men denied ownership of the cocaine and money, the officer arrested each of them.Barry: That’s Chief Justice William Rehnquist reading the facts of the case. Eventually at the station, Joseph, Joseph Pringle, the guy riding shotgun, confessed that the drugs and money were his. He hid it from his friends when he got into the car. It's why Donte let the officer search the car, he didn’t think anything was there. The officer let Donte and Otis go that night. Joseph was charged and eventually convicted.Tape: The Court of Appeals of Maryland reverse Pringles' conviction, concluding that Pringles' confession should've been suppressed as the fruits of an illegal arrest because the officer did not have probable cause to make a warrantless arrest of him.Barry: The argument that won Joseph the appeal in Maryland was that at the time of the arrest, there was only a 1/3 chance that Joseph was the owner of the drugs, and 1/3 is not nearly probable enough for probable cause. The Supreme Courts Justices were fascinated by this argument; the Justices, both conservative and liberal, start pressing Gary Bair, who is arguing for Maryland to affirm Joseph’s conviction. Tape: What if they're been four people in the car?Tape: I don't if four people would change things.Tape: How about six?Tape: I think within the con-Tape: Or what if it was a mini van and there were eight in the mini van?Tape: I'm not sure is changes it significantly, Your Honor. I think that most signi-Tape: You think with eight people in the mini van you could arrest all eight and hold them over for trial?Tape: Probably.Tape: [Laughs] Probably. Intro:Barry: Where do you think you would draw the line? Most people I talk to think you can arrest 2 people, but you can’t arrest 10. But almost everyone says something like “it depends on the context,” which is another a way of saying that maybe you can arrest 10, maybe even 20 as long as the context is right. Context though is usually code, its actually hard to formulate what that’s supposed to mean without getting into racial, class, and neighborhood profiling.And its hardly an academic question; because it turns out that probable cause, and not proof beyond reasonable doubt, is the defacto standard by which most of offenders are incarcerated today. Well on today’s show we’re going to look at how the legal system is having it both ways, using statistical evidence to arrest and incarcerate people while at the same time refusing to let statistical evidence constrain the actions of law enforcement. I don’t think its going to last.Algorithms are quickly moving to replace human decision making in American law. There’s bipartisan support and lots of money pushing for it. The Supreme Court is fully aware that a major case is coming. Chief Justice Roberts has said as much. I think that when the day comes, all of the invisible lines of justice, reasonable suspicion, probable cause, reasonable doubt will need to be revised. Today on Hi-Phi Nation, we’re going to look at the arguments before they hit the courts, and before algorithms take over the justice system.But first, back to the Pringle case and the Supreme Court’s missed opportunity.Scalia: What about five? Are you gonna arrest all five?Barry: Justice Antonin Scalia is questioning Gary Bair in 2003.Scalia: You know it gets worse and worse. There are ten of them now, so the chance that any one individual did is 10%, that's still enough.Bair: I think that you cannot quantify probable cause. In those circumstances-Scalia: It doesn't mean probable.Bair: No, it does not mean probable clearly.Scalia: Why do we call it probable cause?Bair: I think there's a bit of misnomer there, but clearly from the case of law of this court, it means a fair probability, it means something greater than reasonable suspicion under-Scalia: If you had to reduce it to a percentage figure, what would you call a percentage required for probable cause?Bair: I don't know that I could, You Honor. Barry: As hard as the justices seem to be on Bair, he actually has history on his side. Case after case has shown the Supreme Court to be resistant to numbers, opting instead to let purely qualitative descriptions of evidence be the constraints on probable cause. What’s unique to Pringle is their interest in setting a numerical lower bound, dividing the line between legitimate and illegitimate use of police powers to arrest. It's momentous, because the court knows that formulating a numerical probability as a threshold for probable cause means explicitly stating how many innocent people police are given license to arrest and put in jail, even for a long term period. Here’s Justice Ginsberg questioning Sri Srinivasan who is arguing for the U.S. in Pringle.Ginsberg: Supposed Pringle hadn't messed up and he exercised his right to remain silent? Then you might have prolonged period, assuming they couldn't make bail. Three people stuck in the brig and two of them are innocent.Sri: Yes Justice Ginsburg, it's possible that innocent persons will be arrested and bound over in circumstances like in this case, but the probable cause standard accepts that possibility as the cost of insuring the effect of enforcement of the criminal laws as simply the cost of a functioning criminal justice system.Barry: In the end, the court ruled against Pringle. And they did it by answering the boring question rather than the interesting one. This is Justice Souter, questioning Nancy Forster who was arguing for Pringle, with a line of reasoning that ended up winning the day.Souter: Do you have a roll of bills exposed in your glove compartment?Forster: At times I do, Your Honor.Souter: You do?Forster: Yes.Souter: You better be careful.Tape: [Laughter]Souter: Most drug dealers do not go around in there place of business, the car, with people who are totally innocent of drug activity. Doesn't that inference support amount to probable cause?Rehnquist: The probable cause standard is incapable of precise definition or quantification into percentages because it's deal with probabilities and depends on the totalities of the circumstances.Barry: That infuriating piece of gibberish is the decision being read by William Rehnquist, a unanimous decision that took only a month to reach. The court thought that the qualitative description of the circumstances of arrest was enough for probable cause, and that you can’t reduce those circumstances to any numbers. But that’s preposterous! Justice Souter was making claims about what “most people riding in cars with drug dealers” are like, and “what most people who keep stashes of cash in their glove compartment do.” I’m sorry but saying “most” is a claim about numbers, you’re saying that Pringle had a greater than 50% chance of being guilty. If that’s why the officer had grounds to arrest him, fine, but then don’t leave it open that he would still have grounds if the chances really were 1/3, or 10% or even lower. Like what if statistics showed that most people riding around in cars with drug dealers aren’t actually guilty of anything, or that 100 times more law-abiding business owners keep cash in their glove compartment than drug dealers? Do these numbers play a role or don’t they?The court missed their chance. They didn’t end up stating whether 1/3 was high enough for probable cause, or even whether there was ever a numerical threshold too low for probable cause. And if you don’t think this is important, because arrests aren’t convictions, and anyway innocent people will have their day in court and be vindicated, then you are just mistaken about the way the justice system works.Megan: My name is Megan Stephenson, I'm a law professor at George Mason University.For many years the sort of standard model of making the pretrial custody decision is a person gets arrested, they get booked, and then usually within 24 to 48 hours, they go and they have their bail hearing. But a hearing is probably an overstatement, cause they usually last only about 30 seconds. And in those thirty seconds the judge will you know kind of take a look at the police report, they'll take a look at the charges, maybe take a glance at the criminal history, and they'll set a bail amount. Anything from a very high monetary bail to bail denied all together. But most often, it's a relatively low monetary bail amount in the range of a couple hundred to a couple thousand dollars. If you can't afford to pay it you basically sit in jail until your case is resolved.Raphling: The judges have an institutional interest in having people locked up pre-trial.Barry: This is John Raphling, a senior research lawyer at Human Rights Watch. Raphling: People who are in custody plead guilty very fast because people who are in custody are given the choice, if you plead guilty today, you get out, if you plead not guilty, then your trial is gonna be in three months and your sitting in jail for three months. So what're you gonna do?People plead guilty to get out of jail and everybody knows this is how the system works, this is the logic of the system, we pretend that it's about justice, it's not. It's about processing casing and that's why this is one of the engines of the mass incaceration system. Pretrial incarceration greases the wheels because if you're locked up, you're gonna do anything to get out and if that is in the short term you're gonna get released, you're gonna take that deal because you want to get out of jail, like 90% of people do.Megan: And the scale of pre-trial detention is just massive in the United States I mean people that are detained before trial, basically people that are still presumed to be innocent, actually make up a really big portion of the entire incarcerated population. Something like one in five are serving time pretrial. You know you have people sitting in jail basically because you know a police officer believed at least the probable cause was met, but very little protections beyond that.Barry: In July of 2017, Senator’s Kamala Harris and Rand Paul, introduced a bill in the senate moving to replace the entire cash bail system with a data collection and risk assessment system. It followed the lead of some states in the last few years, New Jersey, Kentucky, California and others, that sought to reduce pretrial incarceration. These moves have been widely seen as progressive reforms.Megan: Algorithmic risk assessment tools are these kind of statistical formulas that predict the likelihood that somebody is going to be arrested, convicted, re-incarcerated, or fail to appear in court in the future. They're designed by statisticians who you know kind of crunch a bunch of big data sets and say okay these are the big predictors, you get some points add it on for the things that are risk predictive, such as the number of you know prior arrests. Sometimes you get points taken off for things that predict that you'd be at low risk, such as somebody being you know older 50, 60 something like that. And so for the end of the day, you have a number. That number gets converted into what is usually something like three different risk bins, low risk, moderate risk, high risk. And then this information gets conveyed to the judge or some other criminal justice decision-maker who's going to use that information to help decide whether you should be detained or whether you should be released.It's a way of equalizing the you know the criminal justice outcomes for people that, at least on paper, look alike. It's a way of getting rid of the idiosyncratic variants, based on the judge, the judges mood, you know ways that the defendant might present themself in court. Some people argue it's a way of getting rid of or minimizing racial disparities or racial bias, although that's that's contentious. The other reason people really argue for these things is that they think human beings are not that good at predicting. If we can use computers to analyze historical patterns, you can do a better job at predicting who's going to commit crime in the future, so that you put those people in jail, and then you put the people that have a low risk of committing crime in the future you can release them.Barry: Let's talk very briefly about what goes into the input of these risk assessments. Ahat do they use as data and, more importantly, what are they prevented from using as data?Megan: The only thing that is really not used is actual race. But beyond that pretty much anything anything goes. The more controversial ones are the direct socio-economic indicators. So if someone's poor, if they're homeless, if they don't have a job, if they're unmarried, if they live in certain zip codes--higher crime zip codes--all of these things can be used as factors that increase your risk level and make you more likely to be detained, more likely to be incarcerated, so on and so forth. This is very uncomfortable for many people to think that you would directly increases the chances of somebody being incarcerated because they're poor. Barry: I had this misconception going in to reading this literature that had to be corrected and your work really helped to correct this. When a certain procedure generates a risk score, that score is different from a measure of the probability of a re-arrest, or the probability of a failure to appear. High-risk doesn't mean high probability, low risk doesn't mean low probability. could you explained to people why that is? What that means?Megan: Yeah so, I mean high risk means higher probability, but it doesn't necessarily mean a high probability. When you put somebody in the high risk for violent crime category, they actually only have about an eight percent chance of being re-arrested for a violent crime within the time period that the the tool is looking at. And I think most people when they hear the phrase "high risk," they're thinking numbers that are a lot bigger than 8 percent, because 8 percent means 92 percent fail to be rearrested for a violent crime. this is part of the the tricky thing about adopting these things and implementing them in a meaningful fashion. Not everybody comprehends probabilities and percentages that well, so it's like, we'll make it simple, it's low, moderate, and high. But by making it simple you can also make it misleading.Barry: There’s the magic number, 8%. 8% of people labelled as high risk are rearrested within 6 months. That number comes from the Arnold Tool, the tool used in New Jersey, Kentucky, and lots of municipalities and its public so you can look it up. Other tools have similar numbers, others have better numbers. One used by the federal government, PTRA, is 10%. The rearrest for violent offense is much lower at 2.9%. In Florida, their instrument has high-risk offenders rearrested at sixteen percent. One of the reasons for these discrepancies is the interpretation of high risk.Raphling: But it basically gives you a score on a scale of 6. 6 being highest risk, 1 being lowest risk. What's a 3? Every jurisdiction that I'm aware of, the judges control how the scoring system works. Well let's see, too many people are out of custody and demanding their trial, because if you're out of custody it's easy to file motions and investigate your case and push and fight and get continuance to litigate. It's slowing my court calendar down, so let's make a 3 high risk. Let's make a 3 correlate with stay in jail. There's nothing objective about that.Barry: What’s more troubling is that even without any judge’s discretion, even if we only make the highest risk people subject to automatic custody, the vast majority of them aren’t being rearrested for new crimes, which is the whole justification for automatically locking them up in the first place. If you think 8% is too low, then where should the line be? Is it 16? Is it 33%? This is the same issue as in the case of Pringle. How many innocent people should it be okay to jail so that we prevent a truly dangerous person from being free? Is it 2 out of 3, or 92 out of 100?We’re in the midst of a justice system slowly transforming into one where statistical data is setting the standard for how we treat people under the law. It operates on precisely the principles that the unanimous Supreme Court wanted to deny in Pringle, numerical quantification and thresholds.When we come back, bias, racism, bad data, these aren’t even close to the most serious problems facing these statistical algorithms. There are far deeper ones.Break Barry: There’s this little puzzle that they teach you in law school and it's actually based on real life cases.You’re walking home at night and from a distance you see a taxi cab clip your parked car and then it drives away. Its dark, you can’t really see the color of the cab or the license plate. But you find out that in fact, there are only two taxi companies that run in town, the Yellow Cab company and the Orange Cab company. It turns out that the Yellow Cab company operates 75% of the cabs and the Orange Cab company operates the other 25%. You know it’s a cab, other witnesses can confirm that for you, but none of you could see the color. The question in the law is whether you have enough evidence to sue the Yellow Cab company for damages and win?Almost everyone thinks that the answer is no, and in fact that’s true, the law typically states that this kind of statistical evidence just isn’t good enough.That should be very surprising! It should be so surprising that you ought to be amazed and furious and incredulous at the same time. First of all, the standard of proof in these kind of civil lawsuits is preponderance of evidence, which many lawyers have told me means its greater than 50% likely that the Yellow cab company is at fault. But it is more than 50% likely. For some reason it still isn’t enough. In fact, if we made the numbers higher, like 99% of the cabs in the city are Yellow ones, that still isn’t good enough. You're taught in law school that statistical evidence like this is supposed to be bad evidence.But nowadays it’s good enough evidence to put people in jail! We’re incarcerating people pretrial at a far far lower standard than 75% likely to be an offender or reoffender. Statistical evidence is supposed to be bad, but then why suddenly is it fine?Georgi: My name is Georgi Gardner and I'm a junior research fellow at st. John's College Oxford University.Barry: At first I thought maybe this little paradox arose because we’re talking about civil law not criminal law. Like statistical evidence is bad evidence to sue people but not to jail them. But Gardiner disabused me of this right away.Georgi: A very classic example there would be that a a riot happens in a prison yard, and there's a hundred people in the in the prison yard, and then the sort of blurry video footage shows that one person refused to riot. So now you have this bare statistical evidence that ninety-nine out of a hundred people are prisoners in the prison yard rioted and one out of a hundred didn't.Barry: It’s the same case. Its now 99% likely of every individual that they’re guilty, but it doesn’t pass the threshold for reasonable doubt. It doesn’t pass it, not because the threshold is higher than 99%, but because evidence like this is not supposed to be good evidence even in the criminal law. There’s the puzzle. Statistical evidence is not sufficient to determine liability in civil cases. Statistical evidence is not sufficient to determine guilt in criminal cases. But it is being used to determine danger to society, enough to justify jailing people in the one part of justice system where evidential standards in the courtroom don’t apply. Good or bad.If I were a prosecutor and I wanted to make a case in trial, now we're talking about trial, that a person did or did not commit a crime, can I use the following argument: this person is black and young, making him much likelier to have committed the crime than somebody who is white and old. Can I use that argument in trial?Raphling: There's all kinds of little sneaky ways that prosecutors might make that argument to a jury-Barry: I'm talking to John Raphling at Human Rights Watch. Raphling: -using symbolism, using code words, but right you can't just say that.Barry: Can I say this individual has this many priors, therefore they are likely to have committed this crime. Can I use that as an argument in trial? Raphling: That's actually a very complex question. It's as a rule excluded, meaning you can't use it. Barry: Can I use questions and evidence that you use on risk assessments as evidence in trials and it seems like there are strong prohibitions about it in trial, people sneak it in because they are inclined to believe things on the basis of that information but there are prohibitions on that in trial right?Raphling: I would say at this stage, in a jury trial in front of a jury, a risk assessment score would not be admissible.Barry: There are moral reasons why that's not admissible, I think people recognize that the justice system when you're putting people on trial for a particular crime, that certain kinds of evidence around evidence are not acceptable evidence for this particular crime right.Raphling: It goes to show to some extent what a fraud this system is. And I mean that's actually a really interesting and good point that, in theory, in the trial you're being judge by who you are as an individual, what you've done, what the evidence shows you've done. A tiny of cases go to trial, the vast vast majority, over 97% of cases get resolved through some kind of a plea deal or some other non-trial resolution.Barry: This I think is the real issue; either we’re going to have to admit statistical evidence everywhere, or nowhere. You can’t have it admitted in one place but not another, used to jail 97% of people who don’t go to trial, but then all of a sudden for the 3% who end up in trial, all of these supposedly legitimate reasons used to jail and process you all along suddenly become inadmissible. If people are going to be defaulting to technology to streamline the justice system, there’s no way to avoid the fact that most of the technology is statistical. When we start using facial recognition software to determine a likely perpetrator from video or photographs, it’s statistical. When do you arrest, when it says 35% match, 50% match? Is that match going to be admissible evidence at trial, like eyewitness testimony, or is it only good enough for probable cause? These I think are the real questions behind the use of statistical algorithms. Is statistical evidence legitimate or not?Renee: So what a statistic tells you is the preponderance of a feature across a population.Barry: Renee Bolinger, philosopher at the Australia National University.Renee: So when you apply the statistic across any member of the group, then you're not going to be attentive to the facts about the individual that aren't picked up by the population the way that you've defined it. One of the things that we care about is matching the sentencing of an individual to their actions and not to actions of other people like them. And the big problem with statistical evidence is that it takes into account what other people like them have done when deciding how to treat this person. And that's not normally a way that moral interactions should go.Barry: What would be the right kind of evidence? Because even good evidence might not be decisive right?Renee: Yeah so the right kind of evidence would be what the courts generally refer to as individualized evidence. A person gets hit by a bus and in the area where they were hit, 90 percent of the market share for that route is run by the blue bus company, so the victim sues the blue bus company. That statistical evidence won't actually hold up on just that the plaintiff won't win. Contrast that with a similar sort of case by suppose that there's an even market share between two bus companies, the blue bus company and the green bus company, and there's an eyewitness who says that the bus was blue. And we know one other thing about this eyewitness--he's only about 80 percent accurate. So it's 80% likely on this body of evidence that the blue bus company is at fault. But that kind of evidence is absolutely going to win the day and the plaintiff would come away awarded with the compensation. Barry: Have you thought about responding to the challenge that that's just an incorrect way of thinking about evidence, like actually this distinction is just bizarre?Georgi: Defenders of merely statistical evidence-Barry: Georgi Gardiner of Oxford University.Georgi: -People who advocate using merely statistical evidence might well press this kind of objection--look isn't all evidence merely statistical, because eyewitness testifiers might be accurate 95 percent at the time or something like that. But I think that this kind of response does miss an important detail which is, when for instance an eye witness testifies says oh it was a red taxi, other things are going on epistemically. It's not merely a dice throw.So one way to unpack that might be to say, if the eyewitness testifier said that it was a red taxi and it wasn't, something's gone awry, something's gone wrong, something needs to be explained. Now we can contrast that with where it's merely statistical evidence, so in this case there was just 80 percent of the taxis on the road were red and so in that sense arguably it was probably a red taxi. Well now suppose it was a non red taxi, it was a green taxi, well nothing's gone wrong, there's nothing to be explained. It was just it was just a kind of lottery to get go, you win some you lose some. So that's the kind of epistemic weakness of merely statistical evidence is that just sometimes it's the other case.Barry: Gardiner’s working hypothesis is that the concept of being normal is what makes individualized evidence importantly different from statistical evidence. Statistical evidence makes something likely to be true, individualized evidence makes something likely and normally true. If you’re playing poker, and you have four tens. Its statistically very likely that your adversary doesn’t have four aces. That’s a highly improbable hand, but it’s still a normal hand, if she displayed it you wouldn’t be surprised. Now suppose she displayed five aces. You know something is wrong, she’s either a cheat or the cards are screwed up. But notice that its not impossible for her to do that. People do cheat, cards are made in factories, there are mistakes and misprints. In fact, you could imagine that the likelihood of getting five aces due to cheating or a factory mistake to be the same as the one of getting a Royal Flush of Spades, something like 2.5million to one. You’d be surprised but not outraged if she displayed a Royal Flush, because that’s a perfectly normal hand unlike five aces, which is improbable and abnormal. It's because individualized evidence makes something likely and normally true that makes it special on Gardiner's view. But what it means to be normal is still a mystery. Like consider the difference between eyewitness identification and computerized face recognition. We know that eyewitness identification isn’t perfect, and neither is computerized face recognition, but only the computerized one is explicitly statistical. It's tested on a range of faces and spits out some probability of a match, like 60%. And if its wrong, that’s just luck, we knew we would be wrong 40% of the time. On the other hand, of Gardiner's right, when an eyewitness gets it wrong, as they often do, we demand an explanation, like did the police pressure them, were they stressed? This demand for an explanation shows that we think of eyewitness testimony as normally supporting guilt, its individualized evidence, but should it be?Have you thought about whether the need for explanation is just some bias that human beings have, like sometimes we just want some kinds of minority cases explained and other times we're just happy with minority cases just being the luck of the draw.Georgi: I mean I think that there is something very epistemically important about things like assurances, testimony, what we can normally expect and rely on. I think that there are important epistemic features of explanations and narratives and it's not just a bias. There's features like when we grasp an explanation or we grasp a coherent story, we can tell if something doesn't quite make sense or if some information appears to be missing or some information seems more important or revealing. So I think there might be a lot of epistemically important goodies that get missed if we're just understanding things as a robot might. Ultimately our sort of epistemic lives aren't just a matter of probability and statistics and you win some you lose some. Now whether that's just the hope of mine I'm not sure that I can speak to that.Barry: The state of play in the epistemology of law is that we don’t know what marks the difference between individualized and statistical evidence, whether one is problematic but the other legitimate. People have their theories, they're working it out in journals. We need to figure it out, because all of the boundaries protecting us from illegitimate arrest, prosecution, and punishment depends on it. The philosophers haven't figured it out yet, but practice is preceding theory in the real world.Tape: Are you ready to go?Tape: May it please the court...Barry: Risk assessment algorithms are playing a more and more prominent part at the other end of the justice system, sentencing.Tape: How do we know the judge went too far here?Barry: You’re hearing the arguments in the Wisconsin Supreme Court in the Loomis case. Eric Loomis was charged with being the driver in a drive-by-shooting, and like 97% of these cases, he pled out to lesser charges. He continues to deny having been the driver. At the sentencing, the judge had Loomis's risk assessment score.Tape: You got a bar chart that says this guy's a high risk.Barry: The judge explicitly stated that because of the high-risk score, he was going to sentence Loomis to 6 years in prison, rather than probation.Tape: I do not think that the trial court ought to be using any if those risk scales to say that this ought to be in prison or not in prison.Barry: Michael Rosenberg is arguing on behalf of Eric Loomis at the court. His central argument is that using the algorithm, Compas, amounts to using an unconstitutional factor in sentencing.Rosenberg: The use of gender. The judge doesn't even know the gender's being used and it is because you can't have the risk assessment for recidivism without it.Barry: And of course why wouldn’t you? Men as a matter of fact are at higher risk to commit violent crimes than women.Rosenberg: It also is not an individualized sentence. A defendant is being sentenced based upon showing up with similar qualities as some other group.Barry: The conflict between individualized and statistical evidence is in full force in the Loomis case. On the one hand, yes, Loomis is being judged on the basis of what other people did in the past, that’s statistical science, making a prediction about what you’re likely to do based on what people who are just like you have done. Now think about what used to count as individualized sentence? You had a judge who had less information than Compas does, and she forms a judgement or intuition about how dangerous the defendant seems. You don’t know how the judge got there, like what went into her decision, is she basing it on other people she’s encountered, are they representative? No one used to challenge it because that’s how its always been done. Why is this worse?Remington: To assess a risk of a person in front of you requires circuit courts to make assumptions.Barry: Christine Remington here is arguing for the state.Remington: There was a question about how courts are notoriously bad at making those assumptions. This is a system that has been peer reviewed by scientific literature to be proven more accurate than judges assessment of risk alone.Barry: Loomis lost the case. The Wisconsin Supreme Court upheld the ability of the trial courts to use Compas for sentencing purposes. And everyone I talk to tells me the practice is spreading. There’s actually a lot more to the story regarding Compas. Compas is owned by a private company, who claim that their algorithm is proprietary, so it's impossible for any defendant to cross examine it, they could only challenge the data that goes in. And then three years ago, an investigative team at Propublica found that Compas was giving higher risk scores to the class of blacks who didn’t reoffend than they did to whites who didn’t reoffend. This meant that far more low-risk black people end up in jails than low-risk white people. A controversy ensued and Megan Stevenson decided to try and reverse engineer the algorithm to see if she could figure out the workings of the black box, and what she found opened up everything.BreakMegan: We don't have the algorithm-Barry: Megan Stevenson, law professor at George Mason.Megan: But there's data sets that are available that you know on an individual level will say okay well this is what this person's compass score is and these are some things we know about this person--like this is their age, this is their gender, this is their race, and so forth. What you can do is once you have that data set, you can just regress the score on these different factors and see which of these factors are really explaining most of the variation and the risk score. And so one of the things that was really striking to us is that almost 60% of the variation in these risk scores was just explained by age. Age dwarfed any other factor that was in there. It dwarfed race, it dwarfed criminal history, dwarfed juvenile justice, dwarfed gender.Barry: Are you saying that age actually dwarfs prior convictions of violence also or?Megan: Oh yeah, by far.Barry: By far?Megan: Yeah. The data doesn't have violent convictions, but it definitely dwarfs prior convictions.The label at high risk of violent recidivism is very stigmatic. That might be that might be a sort of legitimate feeling if that is that person earned that label because they've engaged in lots of violent acts in the past, but it's totally not legitimate if people are getting it mostly because they're 18,19 years old.Yes, on a statistical level it might be true that an 18 year old with no history of violence but has you know had a couple of run-ins with the law, maybe they've got some marijuana arrests, this is the type of person that could wind up getting this high risk of violent recidivism label. Like in a very general statistical sense, they might have a high risk of violent crime, but, more importantly, they haven't done anything. Like you know really they're just a teenager.Barry: The considerations you had that make age a concerning feature of getting a high risk score seems to me to apply to a lot of things that could give you a high risk score right?Megan: Yes. So I mean there's lots of things that statistically correlate with future arrests or future conviction, but are non culpable. Like we were talking about previously, you know if you're homeless, if you didn't graduate high school, these are things that have some statistical correlation with future arrests but they're not things that you want to blame someone for.Barry: There's this paradox worth considering that comes from the philosophy of moral responsible. It says that as our ability to predict a person's behavior goes up, their culpability seems to go down. The more a situation can predictably lead people to do something wrong, the more excuse they have from being blamed for that action and the more unjust it is to punish them. It's the reasoning behind excuses for entrapment in the law. Like if cops set up a sting operation offering free heroin at a narcotics anonymous meeting. Now if this is true, it goes directly contrary to the fundamentally justification for jailing high-risk people. Nothing in the current algorithms take into account whether a person is at fault for the things that make them high-risk. And why should it? No one asked the algorithm to do that. And yet we're using them in the one area of government where we're assessing people for fault and responsibility and punishment. If Megan Stevenson is right and the highest predictive factors of future crime are the things that people have least control over--their age, their juvenile criminal history, their race--then precisely the opposite should be happening.Which brings us to what we should be doing with these algorithms. Because the fact is, without them, people will continue to make the decisions, and if you think algorithms are discriminatory, there are a few judges I’d like to tell you about. Now there are three views of this, optimistic, pessimistic, and eliminativist.Seth: No my view actually on this is quite optimistic. I think-Barry: Seth Lazar is the head of the school of philosophy at the Australian National University and he is heading up a project called Humanizing Machine Intelligence, an attempt to seed machine learning with morality. Seth: At every stage of these algorithms they incorporate and reproduce the values that we already have and they also give us the opportunity to impose the values that we actually want to have. The advantage of this sort of algorithmic decision-making is that it does allow us to provide a clear variable for each of the things that we care about. And I think it's a lot easier to do that than it is if what you're trying to change is individual behavior, because whenever you make a decision, you can then subsequently represent it in the way that makes you look best. But it's ultimately opaque to other people, there's no way of really getting into your head. And you often yourself don't know, like this is why we call it implicit bias, unconscious bias.Barry: Megan Stevenson.Megan: And you know there's lots of reasons to be concerned about these judges making these very ad hoc quick decisions, trying to predict the future, over-responding to the skin color of the defendant that's sitting in front of them. Is the adoption of risk assessment better or worse than the status quo? If so, then sure why not. I mean you know I I guess I'm not hugely ideological in that I'm like if it works it works, you know we've got all kinds of dumb things in our world. You know who am I to question what's working if it works.Raphling: The right answer is to take control and get rid of the tools.Barry: John Raphling of Human Rights Watch.Raphling: The tools are dehumanizing, the tools are racially biased and the tools are a powerful driver of mass incarceration in the hands of the existing power structure.Barry: Are you saying that it's actually now than you know cash bail?Raphling: At least with bail a certain number of people can actually bail out. With the new system, people are stuck. If you're found to be high-risk or the judge otherwise thinks you should be stuck in custody, you're stuck in custody.There are models for reforming pretrial decision-making that don't use money bail and don't use risk assessment tools. It's called preserving the presumption of innocence.Barry: Preserving the presumption of innocence means all of the people charged with a crime, except the most serious felonies, are never placed in custody before a trial, regardless of risk. It's never a bargaining chip prosecutors or judges have to hold over the merely arrested.Raphling: For people with more serious charges, then you would be entitled to an actual hearing, not the thirty second jobs that they do in court right now in which their requirements that in order to lock someone up, they have to prove that this person is in fact a specific danger and not some statistical estimate of a danger.B: Do you trust that the judgments that are going to come out of these hearings are going to be more fairer to the individuals than the computerized thing? J: You know judges are often unfair, but there are rules, there are standards, there's review. Having procedures, due process, rights that people have is going to increase the chance that we're going to get to justice.Barry: I think what we’ve learned over the past two episodes is that every stage of the criminal justice system is transitioning over to the use of computerized statistical evidence. Every stage, that is, except the one stage that supposed to determine guilt and innocence, the one stage that gives the accused the most rights of due process, an actual trial. We’re going to need to resolve this contradiction soon, and it can go one of two ways; we’ll have to allow statistical profiles as a way juries reason about someone’s guilt, or we’ll have to place the same constraints on using them outside of trials as we do inside of them. There’s only so much longer the courts can drag their feet about statistics in criminal justice. Maybe they’ll get wiser when all of us do as well.EndHi-Phi Nation is written, produced, and edited by Barry Lam, associate professor of philosophy at Vassar College. For Slate podcasts, editorial director is Gabriel Roth, senior managing producer is June Thomas. Senior producer is T.J. Rafael. Production assistance this season provided by Jake Johnson and Noa Mendoza-Goot. Visit for complete show notes, soundtrack, and reading list for every episode. That's .Barry: If you ever wonder what it would mean to have machines run algorithms morally as opposed to have moral people design algorithms with morality in mind, Seth Lazar is leading just such a research project.Seth: So one idea that's been pursued by some researchers connected with the ANU and the CSIRO is the notion that you can have dueling algorithms. Where on the one hand you have one algorithm that aims to kind of clean the data that it receives, to remove the association with particular ethnic backgrounds or what-have-you from other demographic indicators, and then the other algorithm tries to sort of predict from the data that it receives what ethnic or the demographic background of the people are that it's receiving. And essentially it's a sort of a duel. The more the one is aiming to achieve kind of accurate results, the other algorithm is aiming to sort of figure out how much bias is kind of is carried within the within the data that's being sent through, and they kind of loop around and you have opportunities as the operator to kind of tweak the values that are going in so we can reach a judgment that we find reflectively worth endorsing.Barry: Listen to the rest of the conversation as well as more with Megan Stevenson on the Slate Plus version of this podcast. You'll also get ad free and bonus content for all Slate podcasts. Just go to hiphiplus or click the link in the show notes. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download