A recent study from the London School of Economics and Political Science (LSE) has revealed that artificial intelligence (AI) tools used by over half of English councils may be downplaying women’s health issues. The research found a concerning gender bias in how these AI systems, specifically large language models (LLMs), summarize case notes for adult social care. This bias could lead to unequal care provision for women.
The study, which used real case notes from 617 social care users, inputted the same information into AI models, changing only the gender. The results showed that language used to describe male patients’ health issues was significantly more serious than for women with similar needs. For instance, a man’s case might be described using terms like “disabled,” “unable,” or “complex,” while a woman’s case with identical circumstances would have her needs either omitted or presented in less severe terms. One example cited a male patient as “unable to access the community” while a female patient with the same needs was described as “able to manage her daily activities.”
The LSE report, led by Dr. Sam Rickman, highlights that since access to care is determined by perceived need, this algorithmic bias could directly result in women receiving less support than men. The study particularly flagged Google’s “Gemma” model as having a more pronounced gender-based disparity compared to other models tested. This finding is especially worrying given that councils are increasingly turning to AI to ease the administrative burden on social workers, yet there’s a lack of transparency about which specific models are being used and how they’re impacting care decisions.
The findings underscore a long-standing concern about gender and racial biases in AI systems. The models, trained on vast amounts of human-generated data, absorb and perpetuate existing societal biases. This study serves as a critical wake-up call for regulators and local authorities to address these issues head-on. The report concludes that there must be mandatory measurement of bias in LLMs used in long-term care to ensure algorithmic fairness. In response, Google has stated that its teams will examine the study’s findings.
This research highlights the urgent need for robust oversight and transparency in the deployment of AI in public services to ensure that new technology enhances, rather than undermines, fairness and equality in care.
