'It doesn't actually measure what it sets out to measure'
It’s no secret that employers are paying a lot of attention to employee mental health, especially in light of COVID and other outside factors that have added to the stress and anxiety of workers trying to navigate through work and life.
One tool many organizations have employed to gauge the mental health of their workplace is the psychosocial work survey. This is usually administered to employees, who are asked to assess how they are feeling in a variety of work circumstances such as workload, engagement, skills training and how they are being treated while at work.
But one widely used survey may have some problems with its effectiveness, according to a joint study by the Institute for Work & Health (IWH) and Occupational Health Clinics for Ontario Workers (OHCOW).
The Guarding Minds at Work guide was launched in 2009 for Canada Life’s Workplace Strategies for Mental Health from research done by Simon Fraser University in Vancouver; it is hosted by the Canadian Centre for Occupational Health and Safety (CCOHS) and was used as a reference for the National Standard of Canada on Psychological Health and Safety in the Workplace.
Concerns around duplication
The IWH and OHCOW study was conducted between Feb. 10 and March 5, 2020, and 1,006 Ontario workers were administered the survey online. It included 65 questions over 13 different dimensions.
(The survey was updated in 2020 and includes even more questions, according to the Guarding Minds website.)
After the online surveys were done, Peter Smith, president and senior scientist of IWH and John Oudyk, occupational hygienist at Occupational Health Clinics for Ontario Workers (OHCOW), employed psychometric analytical techniques to understand how each of the questions relate to the other questions within the dimension they’re meant to be in, and to see if the questions in those dimensions separate themselves from the questions in the other dimension.
Peter Smith
For example, if one question was under civility and respect, it should not be duplicated in another dimension, such as engagement, says Smith.
“When we want to talk about psychometric properties… questions within a particular dimension should be more related to those other questions than they are to say other questions in the survey, and… in the case of the Guarding Minds at Work survey, that wasn’t always the case. In fact, it wasn’t often the case,” says Smith.
It’s important that there is no spillover between the different dimensions or categories of questions, he says.
“There’s a question about ‘If you’re informed about important changes that impact the way your work is done’ — which is related to involvement and influence — and there’s another question which is asked: ‘I’m informed about important changes at work in a timely matter,’ which is around clear leadership and expectations. And, clearly just based on the wording, you’d imagine those questions probably get the same response, even though they’re put in different dimensions.”
The guide measures psychosocial dimensions of the work environment, however, it is unable to measure each of the 13 dimensions in isolation, according to the IWH/OHCOW study.
“What we found is measures that were meant to be asking one question meant to be focused on a given dimension would actually relate to questions that were asking about other dimensions of the work environment,” says Smith
“The reason that’s important is because if you wanted to isolate engagement, compared to growth and development, we don’t think the current questionnaire enables you to be able to do that.”
But the dimensions were not meant to be statistically distinct, said Heather Stuart, professor in the department of public health sciences, department of psychiatry and the School of Rehabilitation Therapy Bell at Queen’s University in Kingston, Ont. in a written response on June 14.
“Instead, they were viewed as categories of risk mitigation that impact the psychological well-being and safety of workers. Because efforts of organizations are often intertwined, overlap between the dimensions was expected. One example they gave was that a psychologically safe organizational culture could be perceived as promoting psychological support, it would also be strong in civility and respect.”
When two areas do bleed over, they do not necessarily become invalid, according to another written rebuttal by Merv Gilbert and Dan Bilsker, directors of Vancouver Psych Safety Consulting, and Martin Shain, principal, Neighbour at Work Centre in Caledon Village, Ont. on May 13.
“The perception of managers’ ability to identify psychologically distressed workers will likely be associated with perception of workers’ awareness of their own psychological distress. An organization perceived as strong in organizational culture promoting trust is also likely to be perceived as strong in general level of civility and respect. An organization evaluated as strong in recognition and reward is also likely to be rated as strong in employee involvement and influence. And so forth, ad tedium.”
Survey still valuable
While the Guarding Minds guide is “good” and employers can still employ it to gain valuable information about the overall psychosocial work environment, some tweaks may have to be done for it to be truly effective, says Smith.
“The Guarding Minds at Work group have done a lot to raise the profile of the psychosocial work environment and the importance of the psychosocial work environment, but there isn’t actually a lot of information on how reliable [it is], both in terms of if you are someone that surveys from day-to-day, what would their changes be? And how valid [would they be]? So it doesn’t actually measure what it sets out to measure.”
While the questions in the Guarding Minds survey are generally valid and could yield valuable information on their own, he says, “as part of developing a questionnaire, usually there’s a process you go through with questions that seem to map on to two or three or four dimensions. You would normally remove those because they’re not tapping on to the dimension you need them to tap onto.
“So each question should pick up a little bit of unique information and a bit of shared information within a given dimension.”
But employers should not stop using this survey, says Smith.
“Maybe there’s a bit of overlap, maybe there’s a bit of redundancy, maybe it can’t isolate the 13 dimensions that you would expect it to but it’s better information than nothing in terms of thinking things are going OK. It will give you an indication of how well you are going in terms of the psychosocial work environment as a whole.”
And keeping tabs on employee mental health remains vitally important, he says.
“There is an increasing recognition that hazards at work, and not just those that you can see… are related to things you can’t see, but still as relevant to the health of your employees in terms of their mental health, as well as potentially in terms of their physical health,” he says.
“So I think it is important to understand, from your employee’s perspective, how are things going around their engagement in work? How are things going around the emotional demands that are expending at work? Do they feel like they’re treated with respect? Do they feel like there’s a transparent decision-making process?”
Workers are three times more likely to suffer from depression if the psychosocial environment is unhealthy, according to Australian research.
