Mr. Jones was very excited about Mark, the new customer service representative who was starting Monday. Mark had interviewed well, demonstrated incredible enthusiasm, and his references were glowing. At last, they had found someone who could potentially move up to the supervisor role that would be available in two years.
But six months in, Mr. Jones is beginning to seriously question his ability to judge character. Not only is Mark not a candidate for the supervisor role, he is on the verge of being terminated for poor performance. Why didn’t he catch this in the interview?
This scenario is more common than most of us would like to admit. According to HR academic John Sullivan, professor of management at San Francisco State University, research shows that as many as 20 to 50 per cent of new hires fail to live up to the expectations of their employer. Many of these individuals either end up leaving the organization voluntarily or are fired. If this problem is plaguing your organization, you may be asking what can be done to improve the selection process.
An increasing number of organizations are looking at assessment testing to enhance the interview process. Advances in technology over the last 10 years or so have greatly increased the accessibility of assessments as part of the selection process. These same advances, however, have also complicated the process by making so many options available, and opening up questions related to ethics and administration.
The challenge to organizations that are looking to include assessment in their selection process is three-fold: what to include in the assessment, how to manage the assessment, and how to handle output from the assessment.
The assessment process
While assessments had been used in the military, primarily in the United States, it was not until the 1980s that researchers really identified the potential of assessments to predict job performance. At that time they identified two individual differences that were key determinants of performance: general cognitive ability and conscientiousness.
Research indicated that these two characteristics alone could account for 20 to 40 per cent of performance difference across any job category. It became obvious to many recruiters that assessing these two characteristics during the selection process could improve the quality of the hire.
Cognitive ability refers to how one’s brain processes information, or more practically, a candidate’s thinking skills. There are a wide variety of tests available that measure attention, concentration, memory, abstract-thinking, problem-solving, judgment, language skills, ability to interpret information from senses (hearing, vision, touch), ability to control fine motor skills in the hands, intelligence, academic skills and emotional functioning. Most common for general selection are problem solving, judgment and, more recently, emotional functioning (or EI).
Conscientiousness is part of an individual’s personality. The most common theory from which most personality assessments are based is the “Big Five,” identified by J.S. Wiggins in the 1996 book, The Five-Factor Model of Personality. This narrows down the many variables related to personality to five that are most likely to predict success: conscientiousness, extroversion, openness, agreeableness and natural reactions (neuroticism).
A more recent addition to the assessment for selection package is the “job sample” test, or simulation. These assessments are designed specifically for an organization or a role within an organization. While these types of assessments do increase the reliability of the selection process, they are costly to design and require expertise to ensure validity (the ability to actually measure the desired skill or behaviour) and non-bias (does not negatively impact a designated group). This addition to the assessment portfolio, however, is well worth the investment if an organization is hiring a large quantity of people for similar roles.
The issue of design
One of the most accurate ways to predict how a candidate will perform on the job is to combine an assessment of personality traits and cognitive ability with a job sample type of assessment.
The most credible assessment protocols meet the following criteria: reliability, validity, relevance to business needs, data-based and practicality (see sidebar below).
Credible assessment protocols include assessments that have been tested for reliability and predictability. This information is generally available from the test producer. While there are many excellent “off-the-shelf” options, it is important to map the attributes, skills or competencies assessed to a validated profile for the role in question.
Job sample tests are often customized to the organization and the specific job. While previously these have been “low-tech” (such as role plays), more recent iterations that have taken advantage of technology include job-related scenarios that present candidates with several multiple choice answers. The candidate is asked to identify which of the choices represents the best strategy. These assessments are easy to administer online and are scored using a key that indicates the level of effectiveness associated with each strategy for handling the situation described in an item.
The advantage of a technology-based job sample test is that the organization can build in scenarios related to the job, the values of the organization and the culture the organization has or is trying to develop.
Technology eases test administration, but watch for fraud
The next challenge for an organization is how to administer the assessment. One of the advantages of technology is ease of access and administration. Candidates can access the assessment from anywhere there is an Internet connection. This increases availability to candidates in remote locations and reduces the cost of recruitment as it can reduce travel costs. However, this ease of administration increases the risk of fraud, where the candidate may have someone complete the assessment for them.
A recent example of this risk is a Canadian retail organization that included an online psychometrics assessment in addition to an interview and in-basket exercise in an assessment. Several participants inadvertently revealed that they had included a spouse in the completion of the online assessment. Innocently enough, the spouse’s role was to help define words that the participant was unfamiliar with, but they also, almost inevitably, provided input and recommendation to the response to the question.
In answer to this concern, some recruiters have created satellite assessment centres where candidates go into an office, often at a local university or college, an affiliated recruitment agency or a library, to complete the assessments in a supervised environment. Many organizations, however, are going on the “honour” system, trusting the candidate to be honest, and expecting that dishonesty will be picked up somewhere else in the assessment process.
While technology does make administration easier, organizations run the risk of screening out candidates who are not technologically adept. This matters if computer skills are not a prerequisite to the role in question. Alternative administration methods must be available in cases like this.
Reports easy to run, but often flat in tone
Another of the great benefits (and potential flaws) of many online assessment options is automated reporting. Technically speaking, the narrative component of the report is derived by the system selecting from a library of phrases and paragraphs based on responses to the assessment.
The upside of this technology is the simplicity of reporting, which can often be the most costly part of an assessment. (Consider several hours to write a report at several hundreds of dollars per hour).
The downside is you have to live with the narrative report as it is written. In a recent executive assessment for an international transportation company, an assessment on the president resulted in a phrase in the report stating: “Unable to make logical decisions.” Even if the comment were true, the wording is not constructive. And, given that this is a self-assessment, the definitiveness of the wording is not appropriate. Unchecked or un-validated this type of comment can have serious ramifications.
Some online assessments provide an alternative. Hogan Assessments provides two options for the reports on many of its assessments, a more detailed report that is designed to hand over to the individual, or a less-costly and less narrative “coaching” report. A trained individual can use the results in the coaching report with data from other assessments, interviews and other sources to create a role and organization specific summary report.
Hiring better staff
Research shows that adding some form of assessment into the selection process improves the predictability of on-the-job performance. In fact, authors Hough, Oswald and Ployhart, in a 2002 article in the International Journal of Selection and Assessment, reported that adding personality, cognitive and job sample assessments increases predictability of success by 36 per cent. This improvement in selection benefits the organization by increasing confidence in data for selection purposes, reducing recruitment costs and increasing productivity by getting the right employee in the right place quicker. This evidence is important and worthy of further exploration.
Technology, such as the Internet, has made assessment an easy addition to the recruitment process — sometimes too easy, however. Due diligence is required on the part of hiring organizations to ensure assessments used are in fact finding the right people for the organization. There are all sorts of anecdotal and quantitative data out there, often overlooked, of organizations that have added an off-the-shelf assessment to the recruitment process only to find years later that the addition of the test has done nothing to improve the quality of recruiting.
The best selection processes are the ones that can link each step of the recruitment process to a well-defined profile. And don’t let the advent of technology override the old-fashioned approach of talking to the candidate. Getting to know them can be a useful way to validate the results from the assessment.
Assessing assessment tools
Look for technology that meets the following criteria:
•Reliability — selection procedures and decisions are applied consistently, with consistent results;
•Validity — selection decisions are justified based on a demonstrated relationship between selection procedures and job performance, and ensures legal compliance of the system;
•Relevance to business needs — the selection system needs to reflect the behaviours that differentiate the organization’s business from the competitors’ business;
•Data-based — evaluation of candidates is based on objective data from validated selection procedures, and is thoroughly documented; and
•Practicality — processes are time- and cost-efficient, and straightforward to implement.
To develop a credible assessment protocol that is relevant to an organization, reliable, valid and legally defensible, begin with some form of job analysis or profiling. The most comprehensive and reportedly most dependable include both skills and behaviours for the job, as well as ethics and cultural fit.
Mary Marcus is a senior managing consultant at the Toronto office of Right Management Consultants, a global organizational consulting and career transition firm. She can be contacted at (416) 926-1324, [email protected] or visit www.right.com/ca.
But six months in, Mr. Jones is beginning to seriously question his ability to judge character. Not only is Mark not a candidate for the supervisor role, he is on the verge of being terminated for poor performance. Why didn’t he catch this in the interview?
This scenario is more common than most of us would like to admit. According to HR academic John Sullivan, professor of management at San Francisco State University, research shows that as many as 20 to 50 per cent of new hires fail to live up to the expectations of their employer. Many of these individuals either end up leaving the organization voluntarily or are fired. If this problem is plaguing your organization, you may be asking what can be done to improve the selection process.
An increasing number of organizations are looking at assessment testing to enhance the interview process. Advances in technology over the last 10 years or so have greatly increased the accessibility of assessments as part of the selection process. These same advances, however, have also complicated the process by making so many options available, and opening up questions related to ethics and administration.
The challenge to organizations that are looking to include assessment in their selection process is three-fold: what to include in the assessment, how to manage the assessment, and how to handle output from the assessment.
The assessment process
While assessments had been used in the military, primarily in the United States, it was not until the 1980s that researchers really identified the potential of assessments to predict job performance. At that time they identified two individual differences that were key determinants of performance: general cognitive ability and conscientiousness.
Research indicated that these two characteristics alone could account for 20 to 40 per cent of performance difference across any job category. It became obvious to many recruiters that assessing these two characteristics during the selection process could improve the quality of the hire.
Cognitive ability refers to how one’s brain processes information, or more practically, a candidate’s thinking skills. There are a wide variety of tests available that measure attention, concentration, memory, abstract-thinking, problem-solving, judgment, language skills, ability to interpret information from senses (hearing, vision, touch), ability to control fine motor skills in the hands, intelligence, academic skills and emotional functioning. Most common for general selection are problem solving, judgment and, more recently, emotional functioning (or EI).
Conscientiousness is part of an individual’s personality. The most common theory from which most personality assessments are based is the “Big Five,” identified by J.S. Wiggins in the 1996 book, The Five-Factor Model of Personality. This narrows down the many variables related to personality to five that are most likely to predict success: conscientiousness, extroversion, openness, agreeableness and natural reactions (neuroticism).
A more recent addition to the assessment for selection package is the “job sample” test, or simulation. These assessments are designed specifically for an organization or a role within an organization. While these types of assessments do increase the reliability of the selection process, they are costly to design and require expertise to ensure validity (the ability to actually measure the desired skill or behaviour) and non-bias (does not negatively impact a designated group). This addition to the assessment portfolio, however, is well worth the investment if an organization is hiring a large quantity of people for similar roles.
The issue of design
One of the most accurate ways to predict how a candidate will perform on the job is to combine an assessment of personality traits and cognitive ability with a job sample type of assessment.
The most credible assessment protocols meet the following criteria: reliability, validity, relevance to business needs, data-based and practicality (see sidebar below).
Credible assessment protocols include assessments that have been tested for reliability and predictability. This information is generally available from the test producer. While there are many excellent “off-the-shelf” options, it is important to map the attributes, skills or competencies assessed to a validated profile for the role in question.
Job sample tests are often customized to the organization and the specific job. While previously these have been “low-tech” (such as role plays), more recent iterations that have taken advantage of technology include job-related scenarios that present candidates with several multiple choice answers. The candidate is asked to identify which of the choices represents the best strategy. These assessments are easy to administer online and are scored using a key that indicates the level of effectiveness associated with each strategy for handling the situation described in an item.
The advantage of a technology-based job sample test is that the organization can build in scenarios related to the job, the values of the organization and the culture the organization has or is trying to develop.
Technology eases test administration, but watch for fraud
The next challenge for an organization is how to administer the assessment. One of the advantages of technology is ease of access and administration. Candidates can access the assessment from anywhere there is an Internet connection. This increases availability to candidates in remote locations and reduces the cost of recruitment as it can reduce travel costs. However, this ease of administration increases the risk of fraud, where the candidate may have someone complete the assessment for them.
A recent example of this risk is a Canadian retail organization that included an online psychometrics assessment in addition to an interview and in-basket exercise in an assessment. Several participants inadvertently revealed that they had included a spouse in the completion of the online assessment. Innocently enough, the spouse’s role was to help define words that the participant was unfamiliar with, but they also, almost inevitably, provided input and recommendation to the response to the question.
In answer to this concern, some recruiters have created satellite assessment centres where candidates go into an office, often at a local university or college, an affiliated recruitment agency or a library, to complete the assessments in a supervised environment. Many organizations, however, are going on the “honour” system, trusting the candidate to be honest, and expecting that dishonesty will be picked up somewhere else in the assessment process.
While technology does make administration easier, organizations run the risk of screening out candidates who are not technologically adept. This matters if computer skills are not a prerequisite to the role in question. Alternative administration methods must be available in cases like this.
Reports easy to run, but often flat in tone
Another of the great benefits (and potential flaws) of many online assessment options is automated reporting. Technically speaking, the narrative component of the report is derived by the system selecting from a library of phrases and paragraphs based on responses to the assessment.
The upside of this technology is the simplicity of reporting, which can often be the most costly part of an assessment. (Consider several hours to write a report at several hundreds of dollars per hour).
The downside is you have to live with the narrative report as it is written. In a recent executive assessment for an international transportation company, an assessment on the president resulted in a phrase in the report stating: “Unable to make logical decisions.” Even if the comment were true, the wording is not constructive. And, given that this is a self-assessment, the definitiveness of the wording is not appropriate. Unchecked or un-validated this type of comment can have serious ramifications.
Some online assessments provide an alternative. Hogan Assessments provides two options for the reports on many of its assessments, a more detailed report that is designed to hand over to the individual, or a less-costly and less narrative “coaching” report. A trained individual can use the results in the coaching report with data from other assessments, interviews and other sources to create a role and organization specific summary report.
Hiring better staff
Research shows that adding some form of assessment into the selection process improves the predictability of on-the-job performance. In fact, authors Hough, Oswald and Ployhart, in a 2002 article in the International Journal of Selection and Assessment, reported that adding personality, cognitive and job sample assessments increases predictability of success by 36 per cent. This improvement in selection benefits the organization by increasing confidence in data for selection purposes, reducing recruitment costs and increasing productivity by getting the right employee in the right place quicker. This evidence is important and worthy of further exploration.
Technology, such as the Internet, has made assessment an easy addition to the recruitment process — sometimes too easy, however. Due diligence is required on the part of hiring organizations to ensure assessments used are in fact finding the right people for the organization. There are all sorts of anecdotal and quantitative data out there, often overlooked, of organizations that have added an off-the-shelf assessment to the recruitment process only to find years later that the addition of the test has done nothing to improve the quality of recruiting.
The best selection processes are the ones that can link each step of the recruitment process to a well-defined profile. And don’t let the advent of technology override the old-fashioned approach of talking to the candidate. Getting to know them can be a useful way to validate the results from the assessment.
Assessing assessment tools
Look for technology that meets the following criteria:
•Reliability — selection procedures and decisions are applied consistently, with consistent results;
•Validity — selection decisions are justified based on a demonstrated relationship between selection procedures and job performance, and ensures legal compliance of the system;
•Relevance to business needs — the selection system needs to reflect the behaviours that differentiate the organization’s business from the competitors’ business;
•Data-based — evaluation of candidates is based on objective data from validated selection procedures, and is thoroughly documented; and
•Practicality — processes are time- and cost-efficient, and straightforward to implement.
To develop a credible assessment protocol that is relevant to an organization, reliable, valid and legally defensible, begin with some form of job analysis or profiling. The most comprehensive and reportedly most dependable include both skills and behaviours for the job, as well as ethics and cultural fit.
Mary Marcus is a senior managing consultant at the Toronto office of Right Management Consultants, a global organizational consulting and career transition firm. She can be contacted at (416) 926-1324, [email protected] or visit www.right.com/ca.
