by Jay Dempster, JISC Evaluation Associate
Sound evaluation designs for developing digital literacies stem from projects achieving clarity in two aspects: first, having a strong sense of what they are trying to do, for whom (beneficiaries) and in what ways; and second, identifying relevant and valid ways of measuring outcomes related to those aims and activities.
Outcomes may be short term, tangible outputs and benefits within the project’s funding period, medium term indicators of impact during and beyond the project lifetime, or shared successes that make a difference over the long term to institutional strategies and practices.
In the context of the JISC Developing Digital Literacies programme, a review of project plans and discussion with project teams reveals aims and outcomes that span various levels and affect many different stakeholders, including:
- Understanding the strategic context
- Enhancing digital capabilities
- Sharing good practice in how these are developed
- Identifying issues/gaps/barriers/limiters
- Supporting digital heroes/the digitally excluded
- Building student hybrid roles/partnerships.
With such scope and complexity come inevitable challenges to evaluation. As with any change initiative, it’s been important for projects to avoid trying to ‘change the world’; to resist biting off more than they can chew within the funded time frame and resource. Focusing early activities on baselining has been one way of identifying the relevant scope and parameters.
Right now, the synthesis and evaluation support role has been about helping projects to clarify, ratify and stratify the framework they are using for developing digital literacies, as well as to identify the practicalities of baselining methods and tools. Evaluation is supported by baselining, but it’s doing a different job. Support and guidance has centred on not seeing evaluation necessarily as separate and distinct to core proejct activities.
For many projects, baselining has helped kick start this more integrated approach to evaluation, one that involves key stakeholders in continuous data gathering and reflection. Some of the audits and surveys created for baselining may be reused or repurposed at key stages across the project lifecycle. Project plans have evolved as teams find opportunities to carry out evaluation tasks as part of the project’s development and consultative activities.
Projects will also need to reflect regularly on the effectiveness of their work processes and collaborations to maximise their short and medium term outcomes and bring about their organisational change objectives in the longer term. Projects are encouraged to develop their plans iteratively and transparently on the programme wiki, sharing the various ways in which they have been capturing evidence that is both credible and relevant.
We’ll be running an ‘evaluation’ webinar next month to talk through how projects have approached some of the ideas and challenges emerging from their plans and baseline reports. As we resolve some common challenges of evaluation collaboratively, we’ll be back to blog some more.
Image CC BY-NC-SA ecstaticist