老澳门开奖结果

One of the biggest civil liberties issues raised by technology today is whether, when, and how we allow computer algorithms to make decisions that affect people鈥檚 lives. We鈥檙e starting to see this in particular in the criminal justice system. For the past several years the 老澳门开奖结果 of Idaho has been involved in a fascinating case that, so far as I can tell, has received very little if any national coverage, but which raises fascinating issues that are core to the new era of big data that we are entering.

The case, K.W. v. Armstrong, is a class action lawsuit brought by the 老澳门开奖结果 representing about 4,000 Idahoans with developmental and intellectual disabilities who receive assistance from the state鈥檚 Medicaid program. I spoke recently with Richard Eppink, Legal Director of the 老澳门开奖结果 of Idaho, and he told me about the case:

It originally started because a bunch of people were contacting me and saying that that the amount of assistance that they were being given each year by the state Medicaid program was being suddenly cut by 20 or 30 percent. I thought the case would be a simple matter of saying to the state, 鈥淥kay, tell us why these dollar figures dropped by so much.鈥�

What happens in this particular program is that each year you go to an assessment interview with an assessor who is a contractor with the Medicaid program, and they ask you a bunch of questions. The assessor plugs these into an Excel spreadsheet, and it comes out with this dollar figure amount, which is how much that you can spend on your services that year.

But when we asked them how the dollar amounts were arrived at, the Medicaid program came back and said, 鈥渨e can鈥檛 tell you that, it鈥檚 a trade secret.鈥�

And so that鈥檚 what led to the lawsuit. We said 鈥測ou鈥檝e got to release this, you can鈥檛 just be coming up with these numbers using a secret formula.鈥� And then, within a couple of weeks of filing the case, the court agreed and told the state, 鈥測eah, you have to disclose that.鈥� In a ruling from the bench the judge said it鈥檚 just a blatant due process violation to tell people you鈥檙e going to reduce their health care services by $20,000 in a year for some secret reason. The judge also ruled on Medicaid Act grounds鈥攖here are requirements in the act that if you鈥檙e going to reduce somebody鈥檚 coverage, you have to explain why.

That was five years ago. And once we got their formula, we hired a couple of experts to dig into it and figure out what it was doing鈥攈ow the whole process was working, both the assessment鈥攖he formula itself鈥攁nd the data that was used to create it.

Eppink said the experts that they hired found big problems with what the state Medicaid program was doing:

There were a lot of things wrong with it. First of all, the data they used to come up with their formula for setting people鈥檚 assistance limits was corrupt. They were using historical data to predict what was going to happen in the future. But they had to throw out two-thirds of the records they had before they came up with the formula because of data entry errors and data that didn鈥檛 make sense. So they were supposedly predicting what this population was going to need, but the historical data they were using was flawed, and they were only able to use a small sub-set of it. And bad data produces bad results.

A second thing is that the state itself had found in its own testing that there were problems鈥攄isproportionate results for different parts of the state that couldn鈥檛 be explained.

And the third thing is that our experts found that there were fundamental statistical flaws in the way that the formula itself was structured.

Idaho鈥檚 Medicaid bureaucracy was making arbitrary and irrational decisions with big impacts on people鈥檚 lives, and fighting efforts to make it explain how it was reaching those decisions. This lack of transparency is unconscionable. Algorithms are often highly complicated, and when you marry them to human social/legal/bureaucratic systems, the complexity only skyrockets. That means public transparency is vital. The experience in Idaho only confirms this.

I asked Eppink, if Idaho鈥檚 decisionmaking system was so irrational, why did the state rely on it?

I don鈥檛 actually get the sense they even knew how bad this was. It鈥檚 just this bias we all have for computerized results鈥攚e don鈥檛 question them. It鈥檚 a cultural, maybe even biological thing, but when a computer generates something鈥攚hen you have a statistician, who looks at some data, and comes up with a formula鈥攚e just trust that formula, without asking 鈥渉ey wait a second, how is this actually working?鈥� So I think the state fell victim to this complacency that we have with computerized decisionmaking.

Secondly, I don鈥檛 think anybody at the Medicaid program really thought about how this was working. When we took the depositions in the case I asked each person we deposed from the program to explain to me how they got from these assessment figures to this number, and everybody pointed a finger at somebody else. 鈥淚 don鈥檛 know that, but this other person does.鈥� So I would take a deposition from that other person, and that person pointed at somebody else, and eventually everybody was pointing around in a circle.

And so, that machine bias or complacency, combined with this idea that nobody really fully understood this鈥攊t was a lack of understanding of the process on the part of everybody; everybody assumed somebody else knew how it worked.

This, of course, is one of the time-honored horrors of bureaucracies: the fragmentation of intelligence that (as I have discussed) allows hundreds or thousands of intelligent, ethical individuals to behave in ways that are collectively stupid and/or unethical. I have written before about a fascinating paper by Danielle Citron entitled 鈥淭echnological Due Process,鈥� which looks at the problems and solutions that arise when translating human rules and policies into computer code. This case shows those problems in action.

So what are the solutions in this case? Eppink:

A couple years ago after we鈥檇 done all that discovery and worked with the experts, we put it together in a summary judgment package for the judge. And last year the court held that the formula itself was so bad that it was unconstitutional鈥攙iolated due process鈥攂ecause it was effectively producing arbitrary results for a large number of people. And the judge ordered that the Medicaid program basically overhaul the way it was doing this. That includes regular testing, regular updating, and the use of quality data. And that鈥檚 where we are now; they鈥檙e in the process of doing that.

My hunch is that this kind of thing is happening a lot across the United States and across the world as people move to these computerized systems. Nobody understands them, they think that somebody else does鈥攂ut in the end we trust them. Even the people in charge of these programs have this trust that these things are working.

And the unfortunate part, as we learned in this case, is that it costs a lot of money to actually test these things and make sure they鈥檙e working right. It cost us probably $50,000, and I don鈥檛 think that a state Medicaid program is going to be motivated to spend the money that it takes to make sure these things are working right. Or even these private companies that are running credit predictions, housing predictions, recidivism predictions鈥攗nless the cost is internalized on them through litigation, and it鈥檚 understood that 鈥渉ey, eventually somebody鈥檚 going to have the money to test this, so it better be working.鈥�

As our technological train hurtles down the tracks, we need policymakers at the federal, state, and local level who have a good understanding of the pitfalls involved in using computers to make decisions that affect people鈥檚 lives.