Tailored test development studies
Between 2012 and 2015, ACARA conducted a number of development and design studies designed to pave the way to NAPLAN Online. These studies were funded by the Australian Government, Department of Education and Training. These studies included examining and reporting on:
- the impacts of moving from paper-based to online assessment including in relation to time series data (to maintain trend lines) and analysis of test mode impact for students at different year levels
- the impact of the online delivery mode on the established measurement model for reporting of student performance
- using the digital environment to broaden and enhance assessment design and item types used in NAP programs
- engagement with/provision of the test delivery mode for students with disability and students with educational disadvantage and/or restricted access to computer technology.
The main goal of research in 2012 was to collect evidence on the performance of NAPLAN items, tests and scales when existing NAPLAN items developed for paper are administered as online tests. Approximately 125 schools across Australia participated in the 2012 study. The study investigated whether items perform differently in paper-based and computer-based tests. Researchers conducted cognitive interviews with students, intended to gauge levels of student engagement with the computer-based assessments.
In 2013, ACARA investigated the feasibility of the proposed tailored test design for NAPLAN Online. ACARA trialled online delivery of a multistage computer adaptive test, or 'tailored test' design. More than 250 schools across Australia volunteered to participate in the study. The study included remote schools as well as a trial with a small number of home-based and geographically remote students.
The research study also investigated the cognitive and behavioural engagement of students with the tailored test and the effectiveness of the proposed test design in better targeting students who struggled with the test.
Read the summary research report ( 955 kb).
In 2014, ACARA conducted a further trial of the tailored test design in more than 260 schools. Fully branching tailored tests in reading and numeracy were trialled as well as new options for language conventions, including the testing of spelling using audio files. Results of this study helped to finalise the targeting and measurement aspects of the tailored test design. The findings confirmed that the method for determining the branching cut scores represents a valid and robust solution for tailored NAPLAN online tests.