Artificial Intelligence (AI) has immense potential for advancing society, across various sectors. However, as AI systems are further integrated into more industries, and as open-source models become more advanced, there are concerns about relying too heavily on AI outputs.
Once thought to be objective tools to bypass human error, certain real-world examples would suggest otherwise and highlight the dangers of relying too heavily on these systems.
An early, example showing the imperfections of AI comes from Amazon. In 2014, the company had begun developing an AI system to review job applications and recommend top candidates.[1] The goal was to allow the program to sort through hundreds of applicants, to recommend top candidates, and therefore streamline the hiring process. Instead, the hiring tool displayed gender bias, as it was found to prefer male candidates over females and downgrade candidates with resumes including the word “women”, and individuals who attended all-women’s colleges.[2] The company eventually abandoned this program.
Still, this highlights a scary reality as more and more companies integrate AI across sectors, and particularly as they rely on AI systems in their hiring processes. A 2025 comparative study was conducted to look at bias in language AI models (such as ChatGPT).[3] Nearly identical prompts were input into various language models, with the only difference being specifications indicating if the user was male or female.[4] The results were striking. When looking at the AI outputs, they found gender bias, specifically when the programs were asked about recommendations on salary negotiations.
For example, when ChatGPT’s o3 model was prompted to advise a female job applicant on how much compensation to request during salary negotiations, it recommended she request a salary of $280,000. The same prompt for a male applicant recommended they ask for $400,000.[5] The greatest salary disparities between male and female prompts were observed in the fields of law and medicine.[6]
The study also investigated how the models would advise different users on career field choice. Again, the results show that models outputs varied depending on if the prompt stated the user was male versus female. [7] This further demonstrates the perpetuation of gender stereotypes present in the workforce, in the outputs of these models.
While the results of this study are shocking, given AI is supposed to be an unbiased, objective tool, even more disturbing is how these programs came to learn these biases.
Most large language models (like the ChatGPT o3 model looked at in the study) learn based on large datasets mostly found on the internet. Human trainers then engage in conversations with the models, before humans again step in to rank outputs, further refining the models learning. This means that the models have learned these stereotypes from the internet and available historical data.
The reality is, despite several provinces having pay equity legislation (e.g. Manitoba, Nova Scotia, New Brunswick, PEI, Newfoundland and Labrador) and others having laws to protect pay equity (e.g. Ontario and Quebec), there are still gender pay gaps across Canada.[8] Specifically, the hourly average gender wage gap across Canada was 12% in 2025.[9] This means that the hourly earnings of all working women are 12% less than the hourly earnings of all working men.
Although AI models learn from the internet and real-world statistics such as those above, perhaps this is an opportunity for legislation to step in to ensure AI is not perpetuating harmful discrimination and furthering gender bias with regards to the work force. It is also an opportunity for AI companies to refine their programs to educate people, rather than perpetuate out-of-date stereotypes (i.e. that women in the workforce deserve less than men).
[1] Jeffrey Dastin, “Insight – Amazon scraps secret AI recruiting tool that showed bias against women”, Reuters. https://www.reuters.com/article/world/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK0AG/
[2] Xinyu Chang, “Gender Bias in Hiring: An Analysis of the Impact of Amazon’s Recruiting Algorithm”. Advances in Economics Management and Political Sciences 23(1), p 134-140.
[3] Aleksandra Sorokovikova, Pavel Chizov, Iuliia Eremenko, & Ivan P. Yamshcikov. “Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models” (2025). Cornell University. Accessed from: https://arxiv.org/pdf/2506.10491
[4] Aleksandra Sorokovikova, Pavel Chizov, Iuliia Eremenko, & Ivan P. Yamshcikov. “Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models” (2025). Cornell University. Accessed from: https://arxiv.org/pdf/2506.10491
[5] Aleksandra Sorokovikova, Pavel Chizov, Iuliia Eremenko, & Ivan P. Yamshcikov. “Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models” (2025). Cornell University. Accessed from: https://arxiv.org/pdf/2506.10491
[6] Sion Geschwindt, “ChatGPT advises women to ask for lower salaries, study finds” (11 July 2025). TNW. https://thenextweb.com/news/chatgpt-advises-women-to-ask-for-lower-salaries-finds-new-study
[7] Sion Geschwindt, “ChatGPT advises women to ask for lower salaries, study finds” (11 July 2025). TNW. https://thenextweb.com/news/chatgpt-advises-women-to-ask-for-lower-salaries-finds-new-study
[8] Pay Equity Office, “The Gender Wage Gap: It’s More Than You Think”. King’s Printer for Ontario, 2025. https://payequity.gov.on.ca/the-gender-wage-gap-its-more-than-you-think/
[9] Pay Equity Office, “The Gender Wage Gap: It’s More Than You Think”. King’s Printer for Ontario, 2025. https://payequity.gov.on.ca/the-gender-wage-gap-its-more-than-you-think/