Is AI biased in recruitment?

When faced with piles of resumés, the idea of using software to find the best candidates is understandably appealing.

But while the efficiencies and speed of applicant tracking systems have been promoted far and wide, news that Amazon has abandoned a recruitment tool that turned out to be biased against women has renewed concerns around the potential for bias in artificial intelligence (AI) and machine learning.

The computer program had issues around gender, along with too many unqualified candidates. It screened applicants by looking at patterns in resumés submitted to Amazon over 10 years, but most came from men so, essentially, the system taught itself that male candidates were preferable, according to Reuters.

AI is not yet perfect, said Lisa Stam, founder of Spring Law in Toronto.

“Amazon was just pulling what it thought was neutral information from existing resumés, but the existing data is not really part of the new paradigm that we’re looking to get to,” she said. “We just need to be more forgiving along the way as we figure it out.”

 “It’s a very powerful tool; it’s going to compute things way better than humans most of the time, but we also need to put that upfront work into the machine in the first place. It doesn’t replace human judgment.”

There’s a race towards trying to develop and use these new technologies, but at what cost? said Petra Molnar, research associate in the International Human Rights Program at the University of Toronto Faculty of Law in Cambridge, U.K.

“There’s definitely more questions than answers still, and when it comes to really complex, nuanced areas, like hiring, where we already know that discrimination and biases exist, it’s troubling that we are moving toward the use of these new technologies before we really think about what that’s going to look like,” she said. “Really, these technologies have profound ramifications on people’s basic rights, and… implementing these technologies without thinking about the impact and an accountability framework is really a bad idea.”

What’s the problem?

With AI and machine learning, it’s like anything computer-wide, it has to be trained, said Marilynn Kalman, CEO of HireGround Software Solutions in Calgary.

“And if it’s mistrained, then it gives out wrong data. So, you’re always working at getting a better set of algorithms that give accurate data,” she said.

“It’s always a work in progress and… you’re constantly trying to refine those algorithms and getting them better.”

The more data you have, the more accurate the results, said Kalman.

“So that’s your objective, to feed it more data and get better results, and refining the data source so you can start to slice and dice your data a bit better and refine it so those algorithms are more accurate.”

People are expecting perfection, but machine learning is still ultimately a product of human input, said Stam.

“We hope for but don’t expect perfection from humanity yet, and it’s only as good as the input. It still leads to great conversations about what is neutral; we are trying to take out human bias, and it forces organizations to have this conversation, to think about what criteria are creating these traditional barriers.”

Part of the problem with the recruiting AI is we are trying to create this new paradigm of gender and pay equality, she said, but “there’s no pre-existing models or information or data to build upon necessarily, and so machines are only as good as their input, they’re not a crystal ball. So we will have a few steps yet to go before it can be truly neutral because it can only base its learning on existing information, or have humans at the table to tweak that input to reflect where we want to go.”

There’s an emphasis on AI to overcome biases by designing objective and beta-driven algorithms, with little recognition of how biases are baked within data sets, said Nicole Leaver, Boston-based fellow at the Jeanne Sauvé Foundation in Montreal.

“Of course, the goal should be finding ways to manipulate algorithms to produce equitable recruitment practices, that’s a good thing, but where we missed a step, where it’s rapidly going forward, is it’s not really recognizing the types of data that’s being fed into the algorithm in order to really understand and comprehend the roots of existing biases and how they’re maintained and perpetuated, especially when it’s applied to labour rights.”

Google, for example, invested millions of dollars into a lab where it tried to figure out a way to create the perfect team, only to find out that perfect team synergy didn’t really exist in a recipe, she said.

“That’s what I think really complicates AI is there’s an emphasis on moving towards this automated efficiency and unbiased world, without the emphasis on increasing and protecting the workers themselves,” said Leaver.

“To lay responsibility on coders or programmers would maybe be a jump or not necessarily fair. I think the responsibility extends beyond that into the data that’s being used and the data that’s not being used, especially when it comes to its application.”

Fixing the bias

Part of the solution involves working with AI to lead it into a more positive and equitable direction, she said, “and at least posing questions during its development phase, because that’s been one of the major issues with a lot of automated systems that were suddenly integrated into workplaces was that the… ethics weren’t mainstreamed throughout the process — it was an afterthought, and obviously when ethics are an afterthought, that will lead to major grievances.”

It’s also about undoing the premise that technology and AI are neutral, said Molnar.

“A lot of people make the mistake if you introduce new technologies, they’re going to be more accurate and less biased than a human being. But when you think about it, AI or algorithm just simplifies. It’s a bit like a recipe, so if you have a biased recipe, you’re going to have biased cake.”

Humans have bias and discrimination in their decision-making, she said, and “that data will be used by the machine that it learns on, so we are basically replicating issues we have already.”

Greater diversity at every stage of the process, such as the design and oversight mechanism, also makes sense, said Molnar. And that can mean interdisciplinary conversation too “because there’s a lot of expertise out there about how technology needs to be thought about. But we’re not having conversations across different disciplines and expertise — we need to have policy-makers, academia, civil society, technologists, communities around the table to think about how we’re going to be moving forward with these technologies.”

And it’s also possible there are certain areas where this technology is just a no-go zone, especially when basic fundamental human rights are at stake, such as immigration or employment, she said.

“You definitely need checks and balances… We need to have a broader conversation as a society on what it means to be augmenting or even removing human decision-makers at any given moment, because if that’s where we are heading, we really need to think about what that is going to look like and also what the mechanisms for appeal are going to be.”

Everybody needs to be realistic about the purpose of AI, and what kind of tool it is, said Stam.

“It’s a great way to bring some efficiencies to process, to synthesize or distill information in front of you — in this case, recruiting. I’m not sure that it should always be the final decision-maker, especially in HR, when you really are looking to build a workplace culture,” she said.

“A workplace culture is stronger when there’s a great diversity there, but there’s still some spidey sense that comes into play here.”

We’re all still learning how to do it; it’s still in its infancy, said Stam.

“It’s the adage of ‘garbage in, garbage out,’ or your putout is only as good as your input. We’re learning what we need to put into the machine learning to punch out the right information, or punch out neutral information,” she said.

“So the AI is a tool (to use) alongside human judgment but I do love the role AI can play in terms of really helping to weed out as much human bias as possible, but not replace humans.”

And while it’s possible discrimination claims may arise around the use of AI, it would also be interesting to know how much worse the Amazon output would have been had humans done the same process, said Stam.

“We just notice it more with AI because we’re spitting out this formula and we can pick up the algorithms or the patterns a little more, but if you still had a whole decentralized army of human resources professionals doing this, I’m not convinced that it would have been substantially better, it’s just that we’re expecting higher standards with AI. So, I definitely don’t think we should throw the baby out with the bath water, it’s still a step-by-step process.”