TRAINING STUDENTS FOR ONLINE EXAMS REDUCES COGNITIVE OVERLOAD

Teaching to the test doesn’t work. But teaching students about the test is imperative. Not only that, exam performance IS a thing, and you can assist students to get better at that performance. It’s all about mitigating cognitive load.

GAME TIME – Any sports person will tell you that match fitness is everything. Regardless of how much you prepare, you never achieve the same level of fitness and game knowledge compared to actually playing. Why? Because when the real thing happens, not only do nerves and adrenaline consume vast amounts of energy, interfering with the ability you have coming to the surface, but lots of other unexpected occurrences happen, all leading to increased cognitive load, and leading to exhaustion quicker. The cognitive load can be so debilitating that the player has to rely on muscle memory to get them through. When a student sits an exam, adrenaline and anxiety will naturally surge through their veins. Helping them revise the content is a must, but importantly, helping them become more familiar with the game/exam context is climactical, and this can be achieved by training students to automaticity with exam technique.

ABOUT THE TEST

1. Exam layouts

 Show students, and get them used to, the layout of the online exam. The more they see the module and layout of the exam and understand what the expectations are of each section, the less pressure they’ll feel when they see the real thing.  

Of particular importance with students having to complete exams online is detailing the processes involved if they experience technical issues. Take them through the procedures so if it happens during the exam they don’t lose all confidence and panic. ALSO: Ensure students have read the academic integrity policy and that you discuss it repeatedly – the more you talk about academic integrity the more of it you’ll get.

MANAGEABLE Student
cognitive load
 Student A – no trainingStudent B – training
Before beginning exam20%20%
Exam layout5%0%

2. Question requirements

Ensure students know what each question is demanding of them.

How long is a piece of string?

What does a short answer look like? What gets you full marks? What does a long answer look like? What gets you full marks? How much working out is necessary? How much detail is required?

Don’t expect students to guess the answers to these questions. Students who have to worry about what constitutes a good answer expend lots of valuable cognitive load. Model the expectations by showing previous examples, past exams, etc.

Manageable student
cognitive load
 Student A – no trainingStudent B – training
Before beginning exam20%20%
Exam layout5%0%
Exam content 30%0%

IN THE EXAM

1. Time training

Training students with timings of questions in exams will significantly propitiate cognitive load. It’s one thing to know what the question demands of you, but another to actually do it in a stressed environment. If a student isn’t used to the pressure of time, the longer the exam goes on, the greater the likelihood of their cognitive load increasing and their performance reducing as they panic with the evaporation of time. So, get them to practice doing a mock of a section in the exam – let them experience what it’s like to type in the allocated time – do their fingers get tired? What’s it like to upload if necessary etc. The more practice they get the better, but if you are running out of lesson time to train students, at least give students the chance to practice once – just one section that requires an upload process for example.

This image has an empty alt attribute; its file name is image-2.png

The other aspect of time training is in helping students to set personal timers. Obviously, the online exam doesn’t have all the usual cues that an invigilated exam offers: a large clock, a warning by the invigilator of 5 minutes to go, and even the cues of students completing and organising their work on the next desk. But an advantage of online exams is that students can set their own alarms to negotiate each individual section of the exam, and not accidentally spend too much time on a certain section:

Manageable student
cognitive load
 Student A – no trainingStudent B – training
Before beginning exam20%20%
Exam layout5%0%
Exam content 30%0%
Exam timing training20%0%

2. Editing their work

Rereading responses is difficult for exhausted students to do at the end of a lengthy exam. It is usually at this point that they have a sense of relief, and the last thing they want to do is reread what they’ve done. Of course, it’s madness not to, to ensure there are no silly mistakes, particularly in multiple choice questions, or content mistakes. Even checking for structural, punctuation and/or spelling issues could benefit the overall grade. 

So, I have to build that practice into their normal way of working, so it becomes a part of the process, and not an add on. This can really only be achieved by repeatedly physically getting students to do it: at the end of each ‘mock’ assessment, stop the test and get students to spend 4 – 5 minutes in dedication to proof reading…and explain the rationale, repeatedly: I always tell my students they WILL lose more marks with errors (they can fix) than they are able to gain by writing more response in the last 5 minutes. But without it being a normal way of working, exhausted students won’t do it automatically.   

Manageable student
cognitive load
 Student A – no trainingStudent B – training
Before beginning exam20%20%
Exam layout5%0%
Exam content 30%0%
Exam timing training20%0%
Editing responses5%0%

3. Being professional

Not panicking in certain situations is crucial in reducing cognitive load. Taking students through possible scenarios will help to calm them if the situation presents in the exam, scenarios such as:  If you’re running out of time what should you focus on to get you the most marks? What to do if you can’t answer a question – do you panic and lose total focus for the rest? Should you move on and come back to questions? Are you aware that the brain will warm up and so coming back later may be easier than it is now? This last point is absolutely crucial to convey to students. As the exam progresses, lots of the exam content itself may trigger or cue retrieval of content that couldn’t previously be answered, so teaching students this metacognitive notion could make a significant difference to their overall performance.

Manageable student
cognitive load
 Student A – no trainingStudent B – training
Before beginning exam20%20%
Exam layout5%0%
Exam content 30%0%
Exam timing training20%0%
Editing responses5%0%
Being professional 10%5%

As you can see by the very much made up numbers, the cognitive load experienced by Student A is significantly greater than Student B, and would indubitably affect performance in the exam. The student’s knowledge would have to fight a great deal to break through the pressure. 

BEGIN NOW!

The more you do something the better at it you get, provided of course you’re doing it the right way. Students don’t really get that many opportunities to learn to negotiate the exam environment on their own, especially in the current context of moving to online non-invigilated exams, and so providing them with such training is critical. 

I’m Paul Moss. I’m a learning designer at the University of Adelaide. Follow me on Twitter @edmerger, and follow this blog for more thoughts on education in general.  

Is it even possible to set an online open book mathematics exam?

When trying to offer advice on how to modify exams for the coming semester exams, some subjects have presented with unique issues. Mathematics, for example, has the unenviable dilemma of not being able to set calculation type questions as students can simply type them into an online calculator and be presented not just with the solution, but the workings out too.

The remedy presented to other subjects that require numerical calculations, such as statistics and accounting, of randomising questions, both through the formula question type in Canvas as well as question banks, is not appropriate for mathematics.

The only hope of confidently reducing the amount of ‘Googling’ during the exam is to create more complex questions, questions that require deeper understanding or the application of knowledge, which also requires deeper understanding. Whilst this is of course the ultimate goal of any subject, if such application demands haven’t been taught, then the likelihood of students producing quality answers in exams is limited. If the amount of content that has been introduced determines that only superficial understanding is possible, a breadth rather than depth approach, then question types in the exam can’t change because it’s now open book – students simply wouldn’t have been prepared sufficiently, and thus the exam will not produce valid inferences.

In defense of mathematics, many of the calculation questions that an ordinary invigilated exam would test are designed as such to strengthen fundamental processes and skills that are required for further study in the discipline. The building of the schema is essential to be able to apply understanding in further contexts. But open book exams now pose a large threat to such a design of curriculum. It may be in the future that a depth rather than breadth approach is the only feasible option, so that the depth of understanding in less of the content can open opportunity to assess the application of the knowledge, and thus mitigate against cheating.

Baby with the bathwater?

However, there is something that mathematics’ exam designers should also be conscious of before eliminating all questions that a student could simply look up. The beginning of an exam should really be designed to ease students into the process, to provide a quick boost as they solve a question they find relatively easy. The anxiety, practically 100% concomitant with sitting university examination, is immediately partially assuaged, and thus reduces cognitive overload and allows a student to think more clearly. Exams that begin with very difficult problems can throw off students’ confidence significantly, even those who know enough to pass. It may be that you still set those initial questions as fundamental skill questions that could be looked up but knowing that for the majority, who won’t need to look them up, they will benefit from gaining some confidence in the initial stages of the exam that will facilitate better attempts at the more difficult questions later on.

In the end, it’s not about those who will cheat, it’s about those who won’t.

I’m Paul Moss. I’m a Learning Designer at the University of Adelaide. Follow me on Twitter @edmerger

KNOWLEDGE TRANSFER and designing EXAMS

Few would argue that a goal of education is for knowledge to be able to be transferred from one context to another. However, making it happen is not as easy as it seems, and this has implications for epistemological decisions needing to be made in designing curricula, exams, and indeed, deciding on an institutional ethos.

From research discussed below, knowledge transfer relies on two conditions:

  • transfer is usually only possible when a student possesses a relatively well-developed schema: the closer to expert the better
  • the transfer needs to happen within or close to the known and acquired domain of knowledge.

WHAT THE RESEARCH SAYS

What characterises an expert is their acquisition of schema. Experts tend to have lots of knowledge about a subject, but knowledge that is organised and elaborate in how it connects it all together. Particularly important, in terms of knowledge transfer, is the expert’s ability to see the underlying deep structure of problems, regardless of surface differences. It is this ability to make analogies with what they have previously encountered that not only improves the encoding of new content, but also its retrieval:

  • Experts are better than novices at encoding structure in examples and recalling examples on the basis of structural commonalities (Dunbar, 2001). For example, Novick (1988) found that students completing a second set of mathematics problems all recalled some earlier problems with similar surface features to the present problems, but students with high Mathematics SAT scores recalled more structurally similar problems and were also better at rejecting the surface features than were students with low scores.
  • The reason for this is that when experts think about problems, they draw on/retrieve their large reserves of schema that have evolved, through practice and deliberate exposure to worked examples, to contain the deeper structural features of question types. On the other hand, novices tend to do the reverse, only being able to identify the surface structural characteristics and thus using an inefficient means-end solving strategy (Sweller 1998). The issue with this is that it heavily taxes the working memory, and often results in cognition being overloaded. What’s worse, is that such a taxing ultimately denies the problem from becoming a part of the schema for future use – so there’s a double loss.

The implications of this for education are enormous. The need for schema is irrefutable, from Bartlett to Ausubel and even to Bruner: but for novice students to develop it efficiently, they need to engage in learning that builds knowledge over time and experience, through examples they can store and eventually make analogies with, and interestingly, as Sweller states above, not through problem solving.

So, here’s how transfer can be developed:

  • a student learns by an example, which with the right conditions (retrieval), is then stored in their long-term memory. At this point, only the surface structure of the problem is recognised.
  • The student then encounters another example that has a similar surface structure. Now the student has 2 models to draw from. At this point, only surface characteristics are likely to be seen.
  • The student then is provided another example but this time the surface structure is different but the deeper structure is analogous. The teacher at this point must direct student attention to the analogous deeper connections, as they usually won’t see them for themselves, as proven by Duncker’s tumour problem – see the study below.
  • Repeating this process eventually builds the student’s repertoire of problems they can draw from to make analogies with. The more they have, the greater the chance of them behaving like an expert, identifying the deeper structural components and working forward with the problem, thereby using less cognitive load, and inevitably adding another example to the schema.

How to deliver the analogous examples

Gentner, Lowenstein and Thompson (2003) conducted a study to ascertain what the most efficient delivery combination was. The study used 2 negotiation scenarios, one from shipping and one from travelling as a means of training students to be better negotiators. 4 contexts of delivery were investigated:

  • separate examples, where student were presented both examples on separate pages. Students were asked questions about each text
  • comparison examples, where students saw both examples on the same page and were directed to think about the similarities between the 2 stories
  • active comparison group, where students were presented with the first example on one page and the solutions to that example were carried to a second page that presented the second example with questions asked about the similarities between the two
  • a group that had no training

Clark and Mayer (2008) adapted the findings and presented them graphically:

The results showed that an active comparison was a far superior technique to train the students

Implications for exam design

There are 2 considerations in this regard:

  • When designing open book exams that rely on the application of knowledge (in the current climate primarily to mitigate cheating), it is important to consider the cognitive conditions for transfer to take place. If you have taught your students a range of examples that have facilitated analysis of deeper structural connections, then your question in your exam can test understanding of the deeper structural connection. If you haven’t taught your students in such a way, then your question choice will be limited to more surface level questions. If you ‘jump’ to deeper structural questions, in an attempt to make the questions harder to compensate for the openness and accessibility of the content, then the results of the exam may well be invalid, as you have tested for something that students weren’t capable of doing.
  • On the other hand, knowing that you can safely change the superficial structural elements of a question and test ‘real’ understanding because transfer is difficult if the concept isn’t truly understood, also mitigates against cheating as students can’t simply rely on their notes. If they can’t make the connections, an indicator of a novice learner, then they can’t benefit from the notes as an expert would – who ironically, probably wouldn’t need them anyway.

Duncker’s tumour problem

A problem that has been studied by several researchers is Duncker’s (1945) radiation problem. In this problem, a doctor has a patient with a malignant tumour. The patient cannot be operated upon, but the doctor can use a particular type of ray to destroy the tumour. However, the ray will also destroy healthy tissue. At a lower intensity the rays would not damage the healthy tissue but would also not destroy the tumour. What can be done to destroy the tumour?

Gick and Holyoak used this story to test the transference success of knowledge. Prior to the tumour problem, students are then given the story below, and another group a second story to accompany the current 2. Both additional stories have superficial differences to the tumour case, but similar structural or convergent features. They found that most students who tried to solve the tumour problem on their own had difficulty, those with the aid of one story still struggled, but those with the aid of 2 stories could see the convergent abstract similarities. In other words, they were able to see the deeper structural analogies.

A small country was ruled from a strong fortress by a dictator. The fortress was situated in the middle of the country, surrounded by farms and villages. Many roads led to the fortress through the countryside. A rebel general vowed to capture the fortress. The general knew that an attack by his entire army would capture the fortress. He gathered his army at the head of one of the roads, ready to launch a full-scale direct attack. However, the general then learned that the dictator had planted mines on each of the roads. The mines were set so that small bodies of men could pass over them safely, since the dictator needed to move his troops and workers to and from the fortress. However, any large force would detonate the mines. Not only would this blow up the road, but it would also destroy many neighbouring villages. It therefore seemed impossible to capture the fortress. However, the general devised a simple plan. He divided his army into small groups and dispatched each group to the head of a different road. When all was ready he gave the signal and each group marched down a different road. Each group continued down its road to the fortress so that the entire army arrived together at the fortress at the same time. In this way, the general captured the fortress and overthrew the dictator.

References

Clark, R., Mayer, R. (2008). e-learning and the Science of Instruction. Pfeiffer, San Francisco, CA.

Image sourced from here

I’m Paul Moss. I’m a learning designer. Follow me on Twitter @edmerger