Research and development

A broad research and development program has informed the transition from paper-based to computer-based assessments. The purpose of these studies was to build a new, well-informed assessment model and provide evidence-based information to education ministers and the broader education community about delivering NAPLAN in a computer-based, or online, environment. 

Research studies

ACARA’s research program delivered a number of studies designed to ensure technical and content readiness for NAPLAN online.

On this page:

See also our Tailored tests page.

Tailored test development studies

Between 2012 and 2015, ACARA conducted a number of studies designed to pave the way to moving NAPLAN online. The studies were funded by the Australian Government, Department of Education and Training. These studies included examining and reporting on:

  • the impacts of moving from paper-based to online assessment including in relation to time series data (to maintain trend lines) and analysis of test mode impact for students at different year levels

  • the impact of the online delivery mode on the established measurement model for reporting of student performance

  • using the digital environment to broaden and enhance assessment design and item types used in NAP programs

  • engagement with/provision of the test delivery mode for students with disability and students with educational disadvantage and/or restricted access to computer technology.

The main goal of research in 2012 was to collect evidence on the performance of NAPLAN items, tests and scales when existing NAPLAN items developed for paper are administered as online tests. Approximately 125 schools across Australia participated in the 2012 study. The study investigated whether items perform differently in paper-based and computer-based tests. Researchers conducted cognitive interviews with students, intended to gauge levels of student engagement with the computer-based assessments.

In 2013, ACARA investigated the feasibility of the proposed tailored test design for online NAPLAN assessments. ACARA trialled online delivery of a multistage computer adaptive test, or 'tailored test' design. More than 250 schools across Australia volunteered to participate in the study. The study included remote schools as well as a trial with a small number of home-based and geographically remote students.

The research study also investigated the cognitive and behavioural engagement of students with the tailored test and the effectiveness of the proposed test design in better targeting students who struggled with the test.

Read the Tailored test design study report (PDF 955 KB).

In 2014, ACARA conducted a further trial of the tailored test design in more than 260 schools. Fully branching tailored tests in reading and numeracy were trialled as well as new options for conventions of language, including the testing of spelling using audio files. Results of this study helped to finalise the targeting and measurement aspects of the tailored test design. The findings confirmed that the method for determining the branching cut scores represents a valid and robust solution for tailored NAPLAN online tests.

top

Automated essay scoring

ACARA has undertaken research reviews and studies into automated essay scoring (AES) for marking NAPLAN online writing tasks. Initial research began in 2012 (released 2015) and An Evaluation of Automated Scoring of NAPLAN Persuasive Writing report (PDF 976 KB) summarises these research findings. Analyses and results of the evaluation of training and validation stages for each of the automated scoring solutions are detailed in the technical report (PDF 11 MB) companion piece.

Additional research designed to collect and evaluate empirical evidence on the feasibility and validity of AES in NAPLAN writing assessments, working with a range of sources and analyses, is detailed within the NAPLAN Online Automated Scoring Research Program: Research Report (PDF 1.1 MB) and was released in early 2018.

See our news item regarding NAPLAN online: automated essay scoring on the ACARA website. 

top

Device suitability

The device effect study was conducted to ensure that NAPLAN can be equitably administered across all devices that meet the minimum technical requirements (that is, laptops, tablets, paper). The report concluded that:

  • NAPLAN online is capable of being taken on a range of devices (laptops and tablets), with no overall device effect across content domains, item types and year levels

  • an external keyboard is not necessary for successful interaction with online items when students are responding to tests on tablets (although external keyboards for tablets can still be used if a school preferred this approach) and

  • student familiarity with the device of administration is an important determinant of online readiness.

Literature review and findings are available in the NAPLAN Online Device Effect Study (PDF 3.45 MB).

Results from the device effect study informed the minimum technical specifications.

top

Student engagement and motivation

Between 2012 and 2015, ACARA conducted a number of research trials into student engagement with aspects of online assessments beyond tailored test design. These additional studies have investigated student engagement with technology enhanced items, with challenging (difficult) items, and with the delivery of spelling using audio files. While these trials were limited in terms of sample size, the results from them was promising and formed part of the research program conducted by ACARA to support the move to NAPLAN online.

When working with the technology enhanced items, students were asked to interact with the technology in the context of NAPLAN test questions. This involved using drop-down menus, moving items across the screen, highlighting text and objects or playing an audio or video file. Students were observed across all year groups engaging with the technology and appeared motivated by the experience of working online.

Students participating in the trials recounted the test as more engaging than paper-based NAPLAN tests and said they felt more confident in their performance. The audio delivery of the spelling test via headphones or speakers meant that ACARA was able to test spelling in the way that it is taught and assessed in everyday school practice. Observations of students participating in the spelling test showed that they adapted rapidly to the dictated format, and found it easy and convenient to operate.

The trials also found that no students reported feeling penalised for typing their responses. Overall, students described typing as being easier for changing and correcting answers, and the audio delivery of the spelling words was widely perceived by students to be fair and valid.

Another activity investigated the cognitive and behavioural engagement of students with NAPLAN items delivered within the new tailored (multistage) test design, with a particular focus on the testlet which contains the most challenging and difficult items, many of which are more challenging than those in the paper tests. Students’ knowledge, thinking skills and strategy use in dealing with these challenging items were monitored. The research also investigated how the inclusion of these items related to the intended assessment outcomes.

The main goal was to investigate the extent to which these items are able to target high-level knowledge and skills that the most able students should have achieved. For numeracy, the research found that students who are mathematically capable of solving challenging numeracy items were able to access and engage with most of the questions presented in the testlets containing the most challenging set of questions. 

For reading, the research found that students at each year level drew on wide-ranging reading skills relating to each of the 4 reading development levels: literal comprehension, interpretation, critical reading and creative reading. Overall, the knowledge and strategies demonstrated by the students who participated in these studies indicated that the items in these most challenging testlets were fundamentally appropriate for students with higher levels of numeracy and reading achievement.

A study was also conducted into student motivation factors and effects. Of particular interest was the effect of the fixed and branched forms with regards to students’ performance, students’ motivation while doing the test, and students’ subjective experience of online NAPLAN numeracy tests. Findings shed further light on the positive motivational effects of adaptive online testing and extend current understanding beyond NAPLAN performance outcomes to the important area of student motivation.

See the research reports:

top

Readability and layout

The readability and layout study investigated how students perceive and react to a range of different online item configurations and displays. The purpose of the study was to evaluate different item display and navigation options relative to onscreen layouts.

Findings from the readability and layout study have been incorporated in interface, design and layout options in order to provide optimal readability for online NAPLAN assessment tests in 2017.

Literature review and findings are available in the NAPLAN Online Research and Development Readability and Layout Study (PDF 9.95 MB).

Accessibility options

Research questions in 2013 and 2014 focused on the research basis for current adjustments to pen and paper NAPLAN tests for students with disability and on what basis they may be duplicated or extended in the online delivery mode. The study comprised a literature review and interviews with experts to determine whether the use of certain accessibility options is supported through research and practice. Research findings from this study have helped build the case for NAPLAN online approved adjustments.

Accessibility adjustments need to be:

  • supported by a sufficient evidence base demonstrating its effect and its effectiveness for a specific identified student user group/s with a specific identified need/s

  • compatible with the NAPLAN test construct

  • operationally feasible

  • feasible to deliver on an equitable basis.

Read the report on Accessibility Options for Students with Disability (PDF 1.65 MB).

top