New research finds that performance on financial literacy tests varies depending on question placement — and that scores are worse when questions about key concepts such as inflation, interest rates and risk come at the end of a survey.
In a staff working paper, researchers from the Bank of Canada study the impact of questionnaire design on financial literacy scores, finding that “survey fatigue” results in lower marks.
Following a study that randomized the placement of financial literacy questions, the researchers found that “when financial literacy questions are placed at the end of a survey, respondents are more likely to answer ‘don’t know’” in response to these questions.
Respondents were less willing to put effort into answering financial literacy questions when they were already cognitively fatigued, the paper noted.
The research also found that the increase in ‘don’t know’ responses came “largely at the expense of correct responses” — while there was little impact on the number of incorrect answers.
As a result, they conclude that locating these kinds of questions toward the end of a study results in scores that are 5–15% lower than they would be if the questions had come earlier in the study.
Given that the number of incorrect responses was not affected by question placement, the paper argues that indexes of financial literacy “should be grounded primarily in incorrect responses rather than in correct answers, as relying on correct responses may introduce bias and distort true knowledge levels due to their variability.”
The paper added that understanding the importance of survey design can also lead to more accurate assessments of literacy levels, inform educational interventions and “ultimately contribute to improving financial decision-making capabilities in an increasingly complex economic environment.”
“Emphasizing incorrect answers as the basis for literacy indices ensures a more consistent and valid assessment of financial knowledge, ultimately improving the accuracy of identifying knowledge gaps and informing effective financial education and policy interventions,” it said.