Bias in AI: Lessons for Publishers

by KnowledgeWorks Global Ltd.

During the Frankfurt Book Fair, we teamed up with Springer Nature to host a panel debate entitled “AI 2.0: Machine-Generated Content, Intelligent Automation, and the Future of Academic Publishing” in which speakers from KGL (Cenveo at the time), HighWire Press, and Springer Nature explored ways in which publishers are implementing artificial intelligence and machine learning.

During the discussion, the panelists were asked if there are any drawbacks for publishers relying on this technology to automate processes. Jim Longo, VP of Product Management for HighWire, noted that one of the challenges that has come up is the issue of bias. Because humans are at the heart of writing the code or training machines to perform tasks, the technology is reliant on the perspectives and experiences of those individuals creating the algorithms that could inherently lead to unintended bias.

Some examples of this have been seen in other industries:

Banking and Credit

Just last week, founder and CTO of Basecamp David Heinemeier Hansson tweeted that the Apple Card, a credit card created by Apple and issued by Goldman Sachs, gave him 20 times the credit limit that it gave his wife despite them have shared assets, jointly filing taxes, and more.

The New York Times noted that Steve Wozniak, who invented the Apple-1 computer with Steve Jobs voiced a similar occurrence with he and his wife. “The same thing happened to us. I got 10x the credit limit. We have no separate bank or credit card accounts or any separate assets. Hard to get to a human for a correction though. It’s big tech in 2019.”

Though Goldman Sachs claims that each application is examined separately, the algorithm behind generating credit limit seems to have created this disparity.

Healthcare

In New York, there is an investigation into UnitedHealthcare Group’s use of an algorithm called “Impact Pro” which identifies which patients could benefit from complex health procedures. According to an article in Business Insider, “the algorithm predicted black patients would cost less, which signaled to medical providers that their illnesses must not be that bad. But, in reality, black patients cost less because they don’t purchase healthcare services as much as white people on average.”

Another example in healthcare begins long before machine learning and AI came into play, but will grow worse when reliant on technology to do the work alone—the issue of images of rashes on people of color. Because the majority of images on websites and in books tend to be of people with light skin, it is hard to determine and diagnose in people of color. As the healthcare system turns to using technology to consult more patients, Dr. Roxana Daneshjou, a dermatology resident at Stanford University said it best in the piece on National Public Radio, “‘If we don’t fix the biases we have in healthcare before we jump to training a computer to do it, we’re just going to actually perpetuate those biases.”

Employment

Last fall, the news broke that Amazon was discontinuing the use of a recruiting engine that discriminated against women. The system rated candidates with stars from one to five and, according to this article in Reuters, “Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.” The algorithm was taught to look for specific job functions and locations, but was found to focus on terms found predominantly on the resumes of male candidates.

Once Amazon discovered the bias, it altered the algorithm and currently uses a different version, but there are multi-national companies using similar algorithms developed by third parties or in-house which may be producing similarly skewed results.

Translation

In July 2018, Vice reported on a strange issue users of Reddit were finding in Google Translate. When people would put gibberish in for Google to translate into another language, it would share religious texts. This story sparked much debate as to what the translation was pulling from, if it was a joke by Google employees to cause controversy, or something more ominous, but the answer was actually very simple according to head of Google Translate Macduff Hughes—the machine was often taught these languages using religious texts. Though not a situation of bias, this illustrates how bias in AI can take place.

Academic Publishing

During the Frankfurt Book Fair, Unsilo presented the findings from a survey it had conducted this autumn about the use of AI in academic publishing. In the panel that outlined the findings of the survey, the issue of bias was addressed.  From a blog post recapping the panel discussion: “One worrying finding from the survey was that fewer than 10% of publishers using AI tools were checking for bias. Such a figure seems remarkably low compared with the number of times ‘bias’ is mentioned in an AI discussion today.”

Bias can potentially be an issue for scholarly publishing in areas such as peer review and recommended articles. A similar question came up in a recent Society for Scholarly Publishing webinar, “Are There Finally Real Use Cases for AI?” Use cases that were explored included automating language quality evaluation of manuscripts in copyediting for journals, and using a custom taxonomy to train concepts for indexing, search, and recommending article reviewers. Presenters reported bias in representation of topics (industry vs. academic, theoretical vs. applied), but less so in post-acceptance production processes where the system is not making judgments and if anything, language automation allows more papers to enter the publication process, especially for non-native English-speaking authors.

Taken together, experiences like these suggest that as publishers look to implement AI more in their businesses, it is important to consider all the implications, learn from unintended biases that have arisen in other industries, and devise ways to counteract the inevitable human fallibility that technology can sometimes exacerbate.

KnowledgeWorks Global Ltd. (formerly Cenveo Publisher Services) is the industry leader in editorial and production services for every stage of the content lifecycle. We are your source for intelligent automation, high-speed publishing, accessibility compliance, digital learning solutions and more. Email us at info@kwglobal.com.