In a recent webinar, BTL Surpass welcomed representatives from prestigious accountancy awarding bodies including: Robbie Burns (Assessment & Quality Lead at ICAS), Ioana Raileanu (Senior Assessment and Quality Specialist at ICAS) and Andy Thomas (Director, PEP Examinations – Evaluation and International Assessment at CPA Canada). Unfortunately, due to an outbreak of the COVID-19 virus within her team, Janine Claassens Exam Director at the South African Institute of Chartered Accountants (SAICA) was unable to join the session. Nevertheless, the discussion was a fascinating one, revealing insights into the year in accountancy assessment, adapting to new challenges that the year posed, and the role of technology in modernising accounting assessments.
In any normal year, CPA Canada provide test centre-based assessments to candidates across the country – the largest of which is hosted near Toronto and involves the delivery of 1,200 exams at once. Usually the most prominent challenges on exam day are the length of the in-person verification process (ID checks, PC checks, and seating arrangements) or the possibility of an internet failure though of course, CPA Canada have disaster planning arrangements for such cases.
However soon the very definition of disaster planning was called into question as measures to counteract the outbreak of the COVID-19 pandemic were put in place. What follows has to be one of the most innovative adaptations in the face of adversity that you will hear.
Instead of the usual session-based approach to assessment and, as opposed to using exam halls across the country, CPA decided to rent individual hotel rooms for each candidate: a total of 10,000 of them across Canada! Over three days, this endeavour amounted to 10,000 hotel rooms across roughly 200 hotels all chosen to be as close as possible to the usual exam hall locations. Clearly, logistically and ‘administratively this was a heck of a lot of work’. From setting up 10,000 hotel rooms with their own devices to providing live proctors in the hotels themselves the effort and ingenuity CPA showed is astonishing.
In implementing this novel delivery model CPA had the proctors walk up and down the hotel hallways while the candidates left their room doors open in order to be monitored during the exam. CPA also provided their own devices in each hotel room and as such were able to add layers of security and control to the devices their students used as well as the environment that they took the test in. Perhaps as important and by all accounts the students loved the approach and were particularly pleased that, in spite of everything, their assessment schedule remained consistent.
ICAS, faced with the same issues, took an alternate approach. ICAS exams tend to be 25 or 40 questions in length, often taking the form of a combination of MCQs and long-form questions such as spreadsheet simulators, which allow them to test elements of the syllabus that would be difficult to cover if they were limited to MCQs alone. Another benefit of this approach is the availability of auto-grading, even on the long-form question types as these question types replicate accountancy tables which, in Surpass, can be designated for auto-marking.
In comparison to CPA Canada, ICAS students have more frequent exams with lower numbers for each one, with 13 exams every year and 4,000 candidates for each assessment. Rather than book every hotel room in Scotland, ICAS opted for an on-demand, remote testing model for a total of 45 weeks of the year with up to 1300 candidates sitting tests in one day. This approach provided more freedom for candidates as they were able to choose when was best for them to take their assessment, and many candidates did in fact praise the reduction in stress the remote model caused; reducing travel times, and the stress of travelling at all on the day of their exam.
In order to maintain the integrity of their item bank, as well as the overall security of the test content while testing remotely, ICAS utilised the LOFT functionality within Surpass. The Surpass LOFT simulator is particularly useful for this purpose as it takes item exposure into account when assembling test content. This approach minimises the exposure of items across the item bank and ensures that no items are used repeatedly – to the neglect of others. On the exam day, security concerns are further allayed by the LOFT algorithm as each candidate will be presented with a unique set of questions – preserving the integrity of the item bank.
ICAS’s use of LOFT has further benefits such as attaining fair representation and distribution across topic areas, as well as maintaining a consistent standard of difficulty throughout the assessment. Combining this with the use of the tagging in Surpass, ICAS are able to identify the questions that need updating and altering by performing analysis following a testing period.
Reflecting on the success of the implementation of the remote testing approach, Ioana said:
Our number one take away has been that we live in a world where we have to adapt and we are lucky that we have got the technology to be able to deliver exams in these circumstances.
Ioana Raileanu, ICAS
Not only have the delivery models of both ICAS and CPA Canada been adapted to changing times but so too has their use of item types. Following a demonstration of the new Task Based Simulation (TBS) item type in Surpass, Andy explained that CPA plan to roll out the item across their assessment programmes. This means the TBS is due to replace the use of MCQs in the entry level exams and replace case studies in the more advanced exams. Andy revealed this is due to five main benefits the item type provides:
- Consistency to be found in auto-marking
- Ability to replicate real-life scenarios and test real life skills
- Assesses the higher skills of Bloom’s Taxonomy, not just memory skills
- No human marking reduces cost and increases efficiency
- Analysis provides great metadata and binary data points for analysis
The ease of psychometrically evaluating a TBS item type was also discussed as a major benefit along with the relevance of an item that replicates the tools used in the workplace. As Andy himself put it:
Technology moving forward will allow us to test in more and more real-life situations.
Andy Thomas, CPA Canada
ICAS are also maneuvering into a position to take advantage of this advanced item type. In conversation with their customers, they have focused on adapting and improving their assessments to stay relevant in an ever-changing industry, something the TBS can assist. In Robbie’s words:
More accurately reflecting how a candidate works in the real world today gives the organisations a better indication of how work-ready and well-equipped a candidate is.
Robbie Burns, ICAS
To watch the full discussion between Jim, Andy, Ioana and Robbie, visit surpass.com/webinars.
For further information on how Surpass might benefit your finance certification programme, visit our dedicated finance and accountancy page surpass.com/finance.
