Background and context
The Institute of Education (IOE) has a reputation as the UK’s premier institution for the education and training of teachers and for educational and social research. Each year over 1,000 graduates train as teachers at the IOE on PGCE courses and over 4,000 embark on higher degrees. Many students are working and study part-time and there is a growing body of international students. As of December 2014 the IOE merged with University College London. This case study looks at work undertaken in the period 2011 to 2014 as part of the Jisc assessment and feedback programme and particularly the Assessment Careers project.
EMA implementation
A key aim of the IOE Learning and Teaching Strategy (2011-14) was to encourage innovation and variety in assessment and to ‘Further develop the feedback we give students to assure a sound foundation for improvement’. The Institute did not however have a strong basis for technological innovation and it was clear that increased application of EMA would need to build on existing technologies rather than develop new ones.
The key focus of the activities was to better support learner longitudinal development and there was particular emphasis on the role that feedback plays in this. The Institute has however undertaken interventions in many parts of the assessment and feedback lifecycle.
- Specifying
The IOE embraced the idea of principle-led change as a key instrument in the move from assessment of learning to assessment for learning and developed a set of 6 principles which provide the pedagogic framework for assessment design. The 6 principles are:
- Feedback helps learners to self-evaluate their work.
- Feedback enables students to address performance goals in both the short-term and longer-term.
- Feedback includes dialogue (peer to peer and teacher-student).
- Learners have opportunities to apply previous feedback.
- Feedback is motivational for all students.
- Students have frequent formative assessment opportunities.
To ensure that the principles demanded action as well as agreement the IOE set institutional goals for the implementation of each principle and the dialogue ultimately resulted in the principles being enshrined in an institution wide feedback policy.
Implementation of the principles demanded curriculum change: fragmented curricula and the different degrees to which staff have an overarching knowledge of programme goals were identified as barriers. The IOE undertook a curriculum review in which programmes were clustered around common themes and modules. . Each programme now has a core module as well as options..
Students and staff need to understand programme aims and discussion of overall programme goals with students is recommended and gives students a sense of ownership of their ‘assessment career’. The process of discussing programme goals also helps ensure all staff, particularly part-time staff are aware of programme coherence too.
- Setting:
As well as the implications for the broad area of curriculum design, application of these principles has important implications for the setting of individual assignments for a specific instance of delivery. Assessments must be seen by learners not as a set of isolated activities but as linked up to form a learning pathway. The perspective is one of assessment as a learning journey over multiple-assessments that enables a learner to reflect on their previous feedback, act on that feedback and become aware of progress over long periods of time. A journey through a programme can be supported through ipsative feedback (feedback based on comparison with previous performance) and continued dialogue with assessors and/or peers.
The IOE looked for technology-enabled efficiencies for supporting the approach through linking feedback across modules to address programme level attributes, capturing and monitoring that feedback, ensuring learners can respond to feedback, avoiding repetitive feedback, and helping learners take a long-term cumulative view of learning.
- Supporting:
Pilot studies indicated that students exhibit a wide degree of assessment literacies and views on feedback: from those who think it is the assessor’s responsibility to tell them what to do, to those who realise that they must take responsibility themselves and be proactive. It was apparent that even at Masters level students cannot be relied on to arrive with experience of good practice in responding to feedback. While devices such as a reflective assignment Student Feedback Response Sheet may help students, it was realised that this is not sufficient alone to bring about change and discussion of the benefits of student-centric assessment and feedback needs to be routinely embedded into modules, especially early on.
A largely intangible benefit of the Assessment Careers project was that staff became very aware that they needed to do much more work in enabling many students to engage with feedback. Staff realised that they need to explain and discuss the benefits of engaging with feedback as part of the teaching of their programmes and include more peer assessment activity to help students develop assessment literacy. Enabling students to discuss assessment during teaching sessions and to become self-reliant with practice appears to be a better approach than adding an additional layer of dialogue to either staff or student workloads. One suggestion has been to help students to write feedback on sections of past assignments and compare their feedback with the tutor feedback and other peer feedback. This might give students a better understanding of why action on feedback is important as well as help with interpreting feedback. The longitudinal approach of spreading out dialogue over a programme as part of the normal feedback practice is also a way of ensuring that student workloads are manageable.
A recommendation is therefore that assessment literacy is not assumed and that all students are provided with advice and practice in developing the skills of self-review, systematic action planning in response to feedback and dialogue over feedback.
- Submitting:
Prior to the start of the Assessment Careers project e-submission had been discussed by the IOE teaching committee but there had been reluctance to make this compulsory for students because of concerns about printing out assignments, increased administrative loads and staff unwilling to mark electronic documents.
Once there was an agreed way forward as regards the application of pedagogic principles, it was apparent that achievement of these goals depended on e-submission and e-feedback. The IOE Senate therefore agreed early in 2013 that e-submission would be mandatory for all programmes from the start of 2013/4. E-submission was applied to all students and staff although programmes could continue with paper submission in parallel if they wished.
- Marking and production of feedback:
- Recording grades:
- Returning marks and feedback:
These three elements of the life-cycle are considered together as the IOE has piloted a solution to cover all of these areas.
A longitudinal and cumulative approach to feedback requires that past feedback is stored and is accessible to staff and students. At the start of the project the IOE stored grades in a central database but there was no central facility to store feedback across programmes. Feedback was stored locally by individual lecturers or by programme administrators and was not readily accessible to other staff. Tutors could only access assessment and feedback that they were directly responsible for. Access to a learner’s previous submissions and, more importantly in this context, to previous feedback, would however provide tutors with a more holistic overview of a learner’s development. Students needed to keep their own records. A resolution of the problem of feedback being ‘hidden’ was needed that would grant easy access to past feedback for both assessors and students.
The original project plan did not include technological development, so any new technological implementation needed to be realised within existing support arrangements as much as possible. A technical team explored the options for centrally supporting the process of storage and access to feedback across a programme. The technical team found a solution in a new Moodle Assessment reporting tool in collaboration with their technological partner ULCC and this is outlined in a Technology Report.
The Moodle Assessment Reporting tool was piloted in 2013/14, with an institution-wide implementation planned for 2014/15. The Moodle reporting system makes viewing student progress at the Programme level much quicker and easier. It also enables assessors to build on assessment and feedback from past modules rather than start giving feedback from a blank slate with each student. Concerns that staff may be influenced by seeing a student’s past records have not materialised because staff have often had access to past grades and are aware that these should not influence a future grade. The tool, which is available as a Moodle plug-in, is already generating interest from other institutions.
Although e-feedback is still not compulsory at IOE, the Moodle Assessment Reporting tool requires e-feedback and it is likely that increased interest in feedback records will encourage the further development of e-assessment policy such that it will become mandatory.
E-marking, on the other hand, seems likely to remain as a tutor’s preference for the immediate future. Online marking was explored in one of the pilot projects and Turnitin, which is generally available at the IOE via the VLE, was identified as the most obvious platform for online marking. However, due to concerns about its stability during key submission times, the incompatibility of its grading system with the IOE student information system, and the screen-based nature of online marking overall with associated support resource requirements, Turnitin is currently only recommended as an optional tool for educating students about good academic writing during the formative stages of the assessment process.
- Reflecting
In order to encourage students to reflect on feedback and request feedback, the IOE implemented a simple and low-tech tool known as the Student Feedback Response Sheet that was piloted across 5 programmes. Before the intervention students reported that they read feedback but less than half took any systematic action to apply feedback to future assignments. The pilot results showed evidence from all programmes (135 students) that students were able to use the feedback response sheet to draw upon previous feedback and think about how to address feedback for the future although this alone was not sufficient to address all of the issues relating to assessment literacy (see the section on supporting).
As well as tackling the issue of improving student reflection, the project also took some very important steps to encourage staff to reflect on their feedback practice. They set out to address what they described as ‘inconsistent and opaque feedback practice’. It was noted that, whilst grades were discussed between moderators and made available to staff at Exam Boards, feedback tended to take place in a ‘black box’ and was not generally discussed by programme teams resulting in considerable differences in attitudes and approach. The project produced a feedback audit tool and analysis of feedback across a number of programs revealed a predominantly ‘praise culture’ with feedback tending to be content focused and short-term rather than truly longitudinal. As a result of these analyses a number of programme teams have begun an important dialogue about feedback. The project report concluded: ‘Formative assessment needs to be brought out from under the radar as a key learning mechanism for students. Staff need to be willing to share and discuss feedback practice and be able to identify time to undertake this which might mean reducing some other aspects of teaching. This requires a significant cultural shift across the IOE.’
Link to feedback profiling tool and guidance on the project website.
Another finding was that the way that student evaluations are collected, typically at the end of teaching but before assessment has taken place, means that teaching staff do not normally get any evaluation of their assessment practice apart from External Examiner comments which are usually focussed on the marking process and equity issues. The standard module evaluation pro-forma only has a question to students about whether the assignment processes and criteria are clear and makes no mention of feedback. Students have opportunities to give extensive evaluation of teaching and make any concerns known through both a programme committee with student reps and through completing end of module evaluation forms. Neither process aligns with the timing of formative and summative assessment. This means that tensions over assessment and feedback usually remain hidden. Asking for student evaluations of previous assessments as part of the existing evaluation process might begin to address this issue.
Benefits
The IOE has seen many benefits from the use of EMA across the assessment and feedback lifecycle.
- Good practice is being sustained and embedded through the introduction of the institutional feedback policy.
- The definition of principles has supported a review of the curriculum such that it better support student longitudinal development.
- Evaluation evidence has shown that students making use of the student feedback response sheet are making better use of feedback in the longer term – engaging students more with feedback reduces the need for assessors to write lengthy and time-consuming feedback.
- Evaluation evidence has shown improvements in student attainment as a result of better feedback dialogue.
- Staff have found that the student feedback response sheet improves dialogue without increasing workload.
- The feedback analysis audit and the student feedback response sheet have both resulted in a greater degree of feed forward and tutors better targeting feedback to student needs – tailoring feedback to student requests is more likely to have an impact than feedback that the student is not looking for at the time thus targeting resources effectively
- Staff are now more aware of the need to develop student assessment literacies in the early stages of every programme.
- The IOE has delivered business process efficiencies through the creation of a tool to store feedback centrally and make it available to both staff and students.
- Greater transparency around feedback practice is delivering cultural change.
- The IOE has produced a range of resources that are useful to other institutions and it is likely that the Moodle plug-in tool will be adopted by others.
- External examiners’ reports reflect a positive change in feedback practice.
- Seeing students making progress as a result of feedback helps to ensure that marking and feedback are worthwhile and rewarding activities for assessors: there is no reason why assessment cannot be as enjoyable and inspiring as teaching.
Find out more:
- The Assessment Careers project has produced a wide range of resources and reports available from the Jisc Design Studio.
- Key Assessment Careers project resources are also available from the project website.