<\/p>\n\n\n\n
A Computer Adaptive Test (sometimes referred to as personalised assessment) is an online test which adapts to the candidate\u2019s ability in real-time by selecting different questions from the bank in order to provide a more accurate measure of their ability level on a common scale.<\/p>\n\n\n\n
A personalised assessment pulls questions from a large pool of items that have been carefully calibrated in order to determine their difficulty level (more on this in the next section).<\/p>\n\n\n\n
When a candidate begins their test, they are first presented with an item of medium difficulty deemed appropriate for their year group. If they get that question right, the next item they see will be slightly harder, if they get it wrong, they\u2019ll see a slightly easier item. The system is constantly calculating the candidate\u2019s estimated ability depending on what they get right and wrong, and presenting them with a personalised set of items until the level of confidence in the ability estimate has exceeded a pre-defined level (or the maximum number of questions have been presented) and the test ends. As every learner takes a different path through the test, with a different set of questions, they can potentially receive tests of a different length.<\/p>\n\n\n\n
In contrast to a linear test which in some scenarios only give useful results for learners of average ability, with a personalised assessment, all the items presented to the candidate are designed to be challenging; the number of easy items that are presented to high-ability candidates is reduced, as is the number of hard questions to low-ability candidates, as neither give a clear indication of the ability of those learners.<\/p>\n\n\n\n
As everything is scored in real-time, at the end of the test the candidate can receive immediate, personalised feedback in the form of ability statements as opposed to a raw score or grade, which provides factual information on their strengths and weaknesses based on the questions they answered.<\/p>\n\n\n\n
For a CAT to work, it needs reliable data and a comprehensive item bank with a good spread of content coverage and difficulty level. This means that the item bank must first be calibrated through pre-testing. This is one of the key barriers to CAT as a larger item bank and extensive work is required to get reliable data before any live tests can be delivered. The general rule is that an item must be exposed a minimum of 200 times before reliable data can be generated. Using this exposure data, Item Response Theory (IRT) is then used to calculate IRT parameters for each of the items in the bank. These IRT parameters include the difficulty of the item, and the discrimination of the item, i.e. the factor which determines how much an increase in ability of a candidate will have on the probability of them getting that item correct. In Surpass, these values are attached to the items as tags.<\/p>\n\n\n\n
A test \u2018blueprint\u2019 is generated which determines factors such as content coverage of the test. Many more parameters can also be specified, including, minimum and maximum number of items to present and stopping conditions. An item pool is created which contains all of the items that could<\/em> appear in the test.<\/p>\n\n\n\n Whereas with a linear test, the system knows which items will be delivered before the test begins, with an adaptive test, an algorithm selects the next item in real time, at the point the candidate clicks the \u2018next\u2019 button in the test driver. The algorithm works to the blueprint to ensure good coverage of all content areas and controls item exposure across the bank as a whole (so that some items aren\u2019t presented more frequently than others), meaning the entire item bank is most efficiently used. The algorithm is capable of supporting up to three IRT parameters \u2013 difficulty, discrimination, and guessing.<\/p>\n\n\n\n In Surpass, all of this clever logic happens in just 300 milliseconds<\/strong> of the learner selecting \u2018Next\u2019 to move to the next question, meaning there\u2019s never a delay to the candidate. The algorithm continues until the candidate\u2019s ability has been estimated to the required level of accuracy.<\/p>\n\n\n\n The Surpass team have worked hard to ensure the system can handle these large volumes of data without affecting performance. Microsoft Azure apps have been utilised which are automatically scalable depending on anticipated volumes, and throughput (number of requests per second) has been tested at volumes much higher than those currently being delivered.<\/p>\n\n\n\n One of the key benefits of the adaptive assessment delivered through Surpass is that not only can you make use of the standard reporting functionality, but bespoke reports can be defined and generated via the Surpass API, making use of all of the rich data that is produced from an adaptive test. Reports can show individual candidate journeys throughout the test, as well as reporting on a group or class or even national level as well.<\/p>\n\n\n\n There are numerous benefits to CAT over paper-based testing for formative assessment (providing the item bank has been properly calibrated) including:<\/p>\n\n\n\n Traditional linear tests, where all candidates receive the same set of items, only ever really challenge the middle third of learners. A CAT is designed to challenge learners of all ability levels, providing and accurate and useful picture of leaner ability for everyone.<\/p>\n\n\n\n Many school-level tests are still delivered on paper, which presents significant workload for teachers with the marking and administration of results. Immediate scoring and accurate learner-specific feedback gives teachers more time to focus on teaching, and implementing feedback to help their students to progress.<\/p>\n\n\n\n With an on-screen personalised assessment, there\u2019s no restriction to deliver within the paper test window, meaning they can be delivered for diagnostic purposes at any point throughout the year when the teacher feels it\u2019s suitable. As every learner receives a personalised test, there\u2019s no need for the cohort to all sit the test at exactly the same time.<\/p>\n\n\n\n More accurate feedback can be provided immediately after the test in the form of competency-based ability statements rather than a score. This indicates to the candidate areas they have done well on, and areas they may need to improve. This kind of feedback is more useful in formative assessment, demonstrating to learners that there are areas to progress to, or constructive guidance on where to improve. Teachers can also see the performance of a class as a whole, indicating areas they may need to focus their teaching on.<\/p>\n\n\n\n With questions that challenge learners of all ability, learner engagement throughout the test is better maintained. Low-achievers are encouraged, and high-achievers are challenged. Adaptive assessments can also take less time to complete than a traditional linear test, with an accurate ability measurement reached in a shorter time.<\/p>\n\n\n\n At the 2019 Surpass Conference, Gavin Busuttil-Reynaud from AlphaPlus updated the Surpass Community on the use of adaptive tests built in Surpass for a large-scale national testing programme of primary and secondary school children in Wales. Some of the key points are summarised here, or you can catch up on the presentation in full by watching this video.<\/p>\n\n\n\nWhat are the benefits of CAT over paper-based testing?<\/strong><\/h2>\n\n\n\n
Precise information for candidates of all ability<\/h3>\n\n\n\n
Decrease in teacher workload<\/h3>\n\n\n\n
Potential for on-demand<\/h3>\n\n\n\n
More accurate feedback that can be actioned immediately<\/h3>\n\n\n\n
Learner engagement<\/h3>\n\n\n\n
Using CAT for a large-scale national testing programme in the UK: A Case Study<\/strong><\/h2>\n\n\n\n