On Machine Intelligence 3 - 懂你英语 流利说 Level7 Unit2 Part1

I have a friend who developed such computational systems to predict the likelihood of clinical or postpartum depression from social media data.

The results are impressive.

Her system can predict the likelihood of depression months before the onset of any symptoms -- months before.

No symptoms, there's prediction.

She hopes it will be used for early intervention. Great!

But now put this in the context of hiring.

So at this human resources managers conference, I approached a high-level manager in a very large company,

and I said to her, "Look, what if, unbeknownst to you, your system is weeding out people with high future likelihood of depression?

They're not depressed now, just maybe in the future, more likely.

What if it's weeding out women more likely to be pregnant in the next year or two but aren't pregnant now?

What if it's hiring aggressive people because that's your workplace culture?"

You can't tell this by looking at gender breakdowns. Those may be balanced.

And since this is machine learning, not traditional coding,

there is no variable there labeled "higher risk of depression," "higher risk of pregnancy," "aggressive guy scale."

Not only do you not know what your system is selecting on, you don't even know where to begin to look. It's a black box.

 It has predictive power, but you don't understand it.

"What safeguards," I asked, "do you have to make sure that your black box isn't doing something shady?"

She looked at me as if I had just stepped on 10 puppy tails.

She stared at me and she said, "I don't want to hear another word about this."

And she turned around and walked away.

Mind you -- she wasn't rude. It was clearly: what I don't know isn't my problem, go away, death stare.

Look, such a system may even be less biased than human managers in some ways.

And it could make monetary sense.

But it could also lead to a steady but stealthy shutting out of the job market of people with higher risk of depression.

Is this the kind of society we want to build, without even knowing we've done this?

because we turned decision-making to machines we don't totally understand?

*

What does the system developed by Tufekci's friend do? It predicts the likelihood of depression.

Why did the manager refuse to answer Tufekci's question? She didn't want to address the potential ethical issues with the system.

Why is Tufekci concerned about letting machine intelligence hiring employees? The system may be biased in unexpected way.

To be at the onset of something means… to be at the beginning of it.

To weed something out means…to get rid of them.

*

Not only do you not know what your system is selecting on, you don't even know where to begin to look. It's a black box. It has predictive power, but you don't understand it.

Is this the kind of society we want to build, without even knowing we've done this, because we turned decision-making to machines we don't totally understand?

Another problem is this: these systems are often trained on data generated by our actions, human imprints.

Well, they could just be reflecting our biases,

and these systems could be picking up on our biases and amplifying them and showing them back to us,

while we're telling ourselves, "We're just doing objective, neutral computation."

Researchers found that on Google, women are less likely than men to be shown job ads for high-paying jobs.

And searching for African-American names is more likely to bring up ads suggesting criminal history, even when there is none.

Such hidden biases and black-box algorithms that researchers uncover sometimes but sometimes we don't know, can have life-altering consequences.

In Wisconsin, a defendant was sentenced to six years in prison for evading the police.

You may not know this, but algorithms are increasingly used in parole and sentencing decisions.

You wanted to know: How is this score calculated?

It's a commercial black box. The company refused to have its algorithm be challenged in open court.

But ProPublica, an investigative nonprofit, audited that very algorithm with what public data they could find,

and found that its outcomes were biased and its predictive power was dismal, barely better than chance,

and it was wrongly labeling black defendants as future criminals at twice the rate of white defendants.

So, consider this case:

This woman was late to picking up her godsister from a school in Broward County, Florida, running down the street with a friend of hers.

They spotted an unlocked kid's bike and a scooter on a porch and foolishly jumped on it.

As they were speeding off, a woman came out and said, "Hey! That's my kid's bike!"

They dropped it, they walked away, but they were arrested.

She was wrong, she was foolish, but she was also just 18.

She had a couple of juvenile misdemeanors.

Meanwhile, that man had been arrested for shoplifting in Home Depot -- 85 dollars' worth of stuff, a similar petty crime.

But he had two prior armed robbery convictions.

But the algorithm scored her as high risk, and not him.

Two years later, ProPublica found that she had not reoffended.

It was just hard to get a job for her with her record.

He, on the other hand, did reoffend and is now serving an eight-year prison term for a later crime.

Clearly, we need to audit our black boxes and not have them have this kind of unchecked power.

*

Why is it important to audit machine intelligence? To make sure its decisions are accurate and objective.

Why might machine intelligence be biased? It is trained on data generated by humans. 

*

To amplify something is...to increase its effect. 

To audit something is…to closely examine it. 

If something is dismal, it is…not successful or of very low quality.

*

These systems could just be reflecting our biases, and these systems could be picking up on our biases and amplifying them and showing them back to us,