What does the rise of Artificial Intelligence mean for Psychometric Assessment?

Artificial Intelligence (AI) has been a hot topic for the last decade and is advancing at a rapid pace, with the development of a variety of AI tools including the likes of ChatGPT, which is now used so widely it rivals Google as a search engine, and acts as an assistant performing a variety of tasks. Initially, AI predominantly focused on areas like natural language processing and basic pattern recognition, performing narrow, specific tasks and functions. But recent advancements have seen the development of powerful models which can generate coherent narratives and human like text, even participating in complex decision-making processes. This progression has significantly broadened AI's applicability across various industries, enhancing everything from customer service to creative content generation. As with any advancements however, there are risks and gaps that need addressing.

What are the risks of AI within the world of Psychometric Assessment?

Increasingly, candidates are using AI to assist in writing their CV’s, and cover letters. Indeed, a recent article published by Stuff NZ speaks to an emerging dynamic where candidates are increasingly leveraging AI to counter the AI in use by employers through the selection process. In some instances we are even seeing candidates leverage AI as an assistant when completing psychometric assessments, which poses risks; psychometric assessments are designed to support businesses to identify the candidates that are the best fit for a role. Where candidates use AI, this can distort the process as the use of AI:

1. Creates inconsistent testing conditions for candidates - where some candidates use AI and others do not, this creates inconsistent testing conditions. Ensuring all candidates are assessed under the same conditions is critical in order to be able to compare candidates directly and fairly.

2. Undermines assessment validity and reliability - whether or not AI is a reliable tool in the context of psychometric assessment, candidates may still choose to use AI in the completion of their assessments. This undermines assessment integrity as the results are usually inconsistent with a candidate’s true abilities, and across assessments under conditions without the use of AI.

What are we doing to manage the challenge of candidates using AI in Psychometric Assessment?

There are a number of approaches we have implemented to manage these risks effectively including:

1. Candidate communication: Perhaps the most important piece is ensuring our communication to candidates is clear that the use of AI when completing psychometric assessments is prohibited. As such, we have embedded this communication throughout the psychometric assessment process, most importantly when providing candidates with instructions and information regarding the assessments they have been requested to complete. Where candidates complete unsupervised assessments, we also communicate that they may be required to undertake the assessment under supervised conditions in order to validate their results. Beyond this, we emphasise to candidates that using AI is not necessarily advantageous in that they may perform more poorly on the assessments with AI assistance than without.

   ‍
2. Supervised assessment: When conducting unsupervised psychometric assessments, we can never guarantee that the candidate is completing the assessment unaided - whether this is through leveraging the assistance of family members or friends, search engines such as google, and now AI. We are currently exploring the use of proctoring software that has the ability to monitor the candidate and detect the use of AI, which is expected to be available in the coming months. However, this still does not preclude the support of family and friends, and highlights the importance of opting for supervised assessments wherever possible to ensure the reliability and validity of results.


   
   What do we mean by a supervised assessment? This involves one of our consultants setting a time to meet with the candidate online via Teams or Zoom, or in person in our Added Insight office based in Auckland. When supervised online, candidates are required to have their camera and microphone on and are also required to share their screen throughout the assessment. When conducted in person, the candidate completes the assessment in a designated testing room at the Added Insight office, with a specific testing computer which limits access to websites such as Google and ChatGPT. Beyond ensuring the candidate completes the assessment unaided, the supervision of assessments provides an additional opportunity for real-person contact through the selection process, at a time when use of AI technologies are creating a less personalised candidate experience.  We take our role as ambassadors of our clients’ employment brand seriously and prioritise a high level of candidate care to enhance their experience.
   ‍

1. Ongoing research: To determine whether utilising AI helps or hinders a candidate’s performance in psychometric assessments, we reviewed existing literature and engaged AI experts to use ChatGPT to complete the assessments. Those who tend to perform strongly in the assessments independently tended to score less favourably when using AI, even when they had extensive AI experience. While AI may assist some candidates to achieve stronger scores than they could alone, the results are still very inconsistent. There are a number of reasons for this:


   ‍
   Our reasoning assessments are timed: the process of sharing the information/questions with ChatGPT via screen sharing, screen snipping, and copy and pasting into ChatGPT takes time, as does the processing time for ChatGPT to determine what it believes to be the answer. Over and above this, the answers ChatGPT provided were often incorrect.

   
   AI does not have access to psychometric assessment answer keys: People may be able to copy the questions into an AI tool such as ChatGPT, but AI does not have access to the answer key. This means that AI is unable to learn how to answer the questions correctly consistently.

   ‍
   Results are inconsistent and unreliable: Indeed, the literature suggests that although AI tools are very advanced, they are inconsistent and unreliable when attempting Verbal and Numerical reasoning assessments (Hickman et al, 2024). Our own testing aligns strongly with the literature and indicates candidates outperform ChatGPT. Moreover, ChatGPT demonstrates even less success with Abstract Reasoning, performing particularly poorly.

   ‍
2. Encourage the use of Abstract Reasoning: While AI has demonstrated success in multiple areas, when it comes to psychometric assessment, the results are variable and inconsistent. This is even more evident with Abstract Reasoning where it performs particularly poorly. As such, we strongly recommend including Abstract Reasoning in the testing battery as current AI models find this the most difficult assessment to respond to accurately.
   ‍

In conclusion, AI is a rapidly evolving field and while there are benefits to the advancing technologies across a variety of industries, there are a number of risks this poses for psychometric assessment including creating inconsistent testing conditions and impacting the validity and integrity of the assessment results. That said, by communicating to candidates that using AI is prohibited, supervising assessments, and recommending Abstract reasoning, we can have more confidence that the results reflect an individual’s true capabilities. As AI technology evolves, and new literature emerges, so too will our practices within the field of psychometric assessment.