Three emerging insights from the Digital discovery pilot

Originally posted on Jisc Building Digital Capability Blog .

Co-authored by Clare Killen

Map showing locations of UK pilotsOver one hundred universities, colleges and other providers are piloting the Jisc Digital discovery tool in the UK and overseas. The design of this tool encourages individuals to reflect on and develop their digital capabilities. It provides a summary of their self-assessment in response to nuanced question prompts as well as suggestions for further development with links to relevant, interactive resources. Whilst it is very much a personal tool, additional features allow institutional leads tasked with supporting digital capabilities development to gain insights from anonymised data and translate them into the institutional context.

Jisc team members have visited several pilot institutions to support the implementation process. In doing so, and through our in-depth conversations, we have learned about what works, at a practical level, when it comes to providing opportunities to develop the digital capabilities of staff and students in the various organisations. Further insights have emerged from conferences, events and meetings featuring presentations from our pilots, for example, the Student Experience Experts meeting and the Digital capabilities session at Digifest18.

As the roll-out gathers pace, we are starting to gain some fascinating insights into how institutions are using the opportunities offered by this tool to bring about positive change in their organisations. There are some clear themes emerging around what organisations that are benefiting from the Digital discovery process typically have in place:

 1. Clear strategic vision

We are seeing that the organisations with a clear message about the importance of digital technologies, communicated and understood by everyone, provides a meaningful context for the use of the discovery tool.

“It is important to have a clear strategy and people need to know that digital is part of the strategy and part of what they do. You need to engage people in it, allow them to see how it affects them and why it is important to them. It needs to be exciting, so for example, we have run several big events that inspire and excite people around the idea of using technology to support teaching and learning and the college business.”
Penny Langford, head of e-learning, Milton Keynes College

2. Culture

Having a safe space in which teams can explore their thinking about their own priorities for development creates an environment in which individuals can thrive.

“The individual reports which each member of my team had, generated discussions and comparisons, with staff considering their different roles and how that has had an impact upon their individual percentage. More than that though, it made them consider how they might acquire skills where they didn’t score as highly. I have eLearning Technologists and Librarians in my team and each had different scores, particularly the Information Literacy category. Which prompted all manner of discussion around the fake news agenda and critically evaluating information sources.”
Sarah Crossland, academic services manager, Doncaster College and University Centre

3. Connections

Establishing the connections between individual self-identified aims, the overall picture for all staff and the resources available to support professional development to meet organisational strategic aims.

We wanted to identify gaps in staff confidence in their digital skills and use this information to target staff training and support. We looked at other products but there was nothing really out there to meet those requirements. We were looking for a standardised tool and wanted something to self-motivate staff. The approach taken by the Digital discovery tool supports that.
Joseph Pilgrim, digital learning co-ordinator, ACT Training

Digital capability community of practice

The next digital capability community of practice event is being hosted in partnership with the University of Leicester on 22 May 2018. This provides an opportunity to learn about related initiatives and hear more from the wider community including many members who are taking part in the pilot of Digital discovery tool.
While registration for this event has now closed, the keynote sessions will be live streamed. Follow the hashtag #digitalcapability on the day and presentations and any outputs will be available from the event page.

There is still time to engage staff

If you are part of the pilot, you still have time to engage staff, perhaps through end of term staff development events. Remember that feedback is required by the end of May but the Digital discovery tool will continue to be available until 13 July 2018.

How HR teams support staff digital capability

Originally posted on Jisc Building Digital Capability Blog .

At the end of 2017 we began a short review into how Human Resources (HR) departments support staff to develop their digital capability. We developed an online survey and interviewed some of the respondents to try to capture a snapshot of current practice.

Initial results

The results of these activities confirmed our initial expectation that many HR teams have been working across several areas of the digital capability framework, often in partnership with other teams within their institutions. However for both HE and FE respondents there were quite significant variations to the questions about HR team involvement in the 6 core digital capability areas. Whilst 90% of people said they were involved in supporting ICT proficiency of staff, only 50% said they were involved supporting staff with information, data and media literacy, digital communication, collaboration and participation, or digital learning and teaching. 84% said they were not involved in digital creation, problem solving and innovation and 58% said they were not involved in digital identity and wellbeing.

Later questions and in-depth interviews revealed that many HR teams are in universities or colleges which are just starting to take an institution-wide approach to staff and student digital capabilities. One of the challenges for HR teams is in identifying their roles and potential areas where they could input to institution-wide initiatives and the developments of strategies for developing digital capabilities. Whilst some HR teams were aware of the Jisc tools and resources to support this work, many had not seen them before or had not engaged with them. It became clear to us that there was a need for some practical materials to help HR teams map their various activities (often split into specialist sections) to the digital capabilities framework.

The original survey is still open so if you did not get a chance to respond earlier we would still welcome your input.

New materials for HR teams

HR teams cover a wide range of activities that require them to consider and/or support staff digital capabilities across their institutions. These include recruitment and selection, onboarding, appraisal/performance review, learning and development, relationship management and health and wellbeing. Data management and analytics, increasingly sophisticated institutional systems and the impact of social media mean that Human Resource teams themselves need a range of digital capabilities to effectively carry out their work.

We have produced two sets of powerpoint slides that could be used within HR teams and we are interested to find out if they are useful. Thanks are due to Abi Mawhirt, Head of People and Organisational Development at Dundee and Angus College who worked with us to refine these slides and to make sure we did not have any serious omissions. Abi will be using the slides within her own institution and we have some other HR teams who have said they might try them out.

HR teams could use the slides (or select the ones that they feel are most relevant to their context) to consider their activities, identify and build on strengths, as well as identify any gaps or areas where they could enhance their support of staff digital capabilities. It may highlight areas where HR teams could take the lead, for example in the area of Digital identity and wellbeing.


This set maps HR activities and roles to the Jisc digital capabilities framework. It highlights where HR teams can input to institution wide approaches to staff digital capabilities and offers some suggestions for activities where they could get involved. Some of these areas involve other teams and would encourage HR input to support teams leading on a particular area.


This set offers a view of HR activities through the Jisc digital capabilities framework. Each area of HR activities is mapped to the 6 key elements of the Digital capabilities framework and highlights where HR teams can impact on digital capabilities of staff (and to a lesser extent students).

We have also highlighted those activities that relate to digital capabilities of staff in HR teams.

Please pass these on to your own HR team and ask them to try them out. We have produced a brief pdf document which offers ideas for how they might be used.

Here are some of the suggestions:

  1. Use the slides to deliver a team presentation highlighting areas of most relevance to the team.
  2. Use the slides or a selection of slides in a presentation to focus on particular aspects – either a particular area of HR activities such as recruitment and selection or on a specific area of the digital capabilities framework such as Digital wellbeing.
  3. Use the slides as a pdf document to share within teams and follow up with workshops to consider them within your own context.
  4. Get different teams within HR to focus on specific slides (or pdf pages) and ask them to come up with an action plan following their discussions.
  5. Use the slides or some of the content to present to different teams within the organisation to highlight what you are doing in different areas of digital capability or what you would like to do.
  6. Use the materials to highlight areas for joint working or partnership approaches to other teams or departments within the institution.
  7. Link to other Jisc digital capabilities, guidance, tools or resources to highlight possible HR roles across the institution.

We would like to gather some feedback about these so that we can adapt or enhance them. Link to a brief survey.

Let us know what you think. Help us make them better.

Discovery tool: understanding the questions

Originally posted on Jisc Building Digital Capability Blog .

We have just been through an interim analysis of feedback from staff users of the Digital discovery tool. Thank you for directing so many staff to complete the feedback form – 225 general staff and 150 teaching staff have done so already, and it has been an invaluable resource.

Screen Shot 2018-05-14 at 15.20.38The feedback so far has been very positive, with some interesting perceptions that we will report in the next blog post. This post is about some of the changes we have made to the content of questions. It also seems like a good opportunity to explain a bit more of the thinking that goes into the three question types, and into the reasons for designing the discovery tool in the way we have. There is some general information at the top of the post, and more detail further down for those who are interested in the different question types.

Development rather than testing

At the start of the design process we had to make a significant decision. We could have written ‘testing’ questions, as in a typical assessment test, to find out what users really  understand about digital applications and approaches. But we decided to write ‘developmental’ questions instead. These are designed to develop understanding, for example by making clear what ‘better’ (deeper, better judged) performance looks like. Rather than hiding the ‘right’ answer, they make transparent what expert digital professionals do and ask users to reflect and report: ‘do I do that?’

We have gone down this road partly because we are not convinced that testing abstract understanding is the best indicator of actual practice, and partly because this approach is more acceptable to end users. Staff want to be treated as professionals, and to take responsibility for assessing and moving forward their own practice. Also, we are not designing in a platform that supports item-by-item matching of feedback to response. So it’s not possible for the feedback itself to be closely matched to users’ input – as it would be in an assessment system – and our questions themselves have to do a lot of the work.

This has important implications for the meaning of the scoring ‘bands’ that we use to assign feedback to users (more of this shortly).

Where do the question items come from?

Essentially, to design the questions we first developed a wide range of real-world activities that digital professionals do. We’ve tested those out with expert panels, and also against the relevant professional profile(s) – which have had professional body involvement.

Of course we could just have presented these activities in a random order, and this was an early design idea. But the digital capabilities framework already had good recognition in the sector, and we needed a navigational aid. So in the case of the generic assessments (for staff and students) we allocated activities to the different framework areas, e.g. ‘data literacy’. In the case of role-specialist assessments, we used specialist requirements from the relevant profile, such as ‘face-to-face teaching’ or ‘assessment and feedback’ in the case of the teaching assessments.

We then took one activity that was central to the area in question and framed it as a ‘confidence’ question (‘How confident do you feel about doing x?’). We developed another activity into a mini-scenario or example to create a ‘depth’ question, with four levels of response possible (‘Which of these best reflects your response?’). Six further activities became options in a ‘breadth’ question (‘Which of these can you do? Select any or all that apply to you’). This provides us with three questions, 8 activities, for each area of practice. There is more about the different question types below.

We have not statistically tested to discover whether responses to all three questions in one area  hang together to create a distinct and separate factor. There is the opportunity to do that with system data at this point, but our first aim was to create a navigable user experience – making sense and generating helpful feedback – rather than to validate a model.

Ideally the feedback we give to users would relate to their responses for each of the eight different activities. Without this option, we have used scoring bands to allocate roughly appropriate feedback to users, based on their responses to the three questions. It’s not exact, and some users have picked that up. However, most users rate the quality of feedback highly – it has the most positive comments of any feature – so we know we are getting it more or less right. We hope we have dealt with the lack of specificity by offering a range of ‘next steps’ that participants can choose from, according to their own interests and self-assessed development needs.

You’ll understand from this that scoring is an artefact of the system we are using and the design choices we have made within it, not an objective measure of any kind.

We were pleased when we analysed system data from the first two months of use to see that in all but three of the 45 generic staff questions, and in all the teaching staff questions, the scoring bands were evenly distributed. This means that the questions were doing a good job of discriminating among staff according to their (self-declared) expertise, and the full range of scoring bands and feedback was being used. Three questions had median scores outside of the normal range, and a couple of sections elicited comments that users did not feel their feedback reflected their actual capability (‘information literacy’ was one). Rather than changing the underlying scoring model for these questions, we decided it was more appropriate to work on the content to try to produce a more even distribution of responses around a central median point. So if users’ scores differ from the median, that should mean something – but we can’t say that it means anything about their objective performance.

Of course users who answer the questions after the changes were made on 5 May will not be scoring in the same way as users who answered the questions before. (It’s also possible that in making the changes suggested by user feedback, we have inadvertently shifted the scoring for some other questions – we will be checking this.) This will need to be communicated to any staff who are returning to use the discovery tool again. It will also need to be taken into account when looking at data returns, since data from before and after the changes can’t be treated as one data set. This is one reason we have cautioned against using scoring data to draw any firm conclusions, particularly during this pilot period when the content is still evolving.

We hope you will convey to all the staff who took the time to complete a feedback form that we have listened to their views – and that you and they will feel that the revised questions are an improvement. This is why this pilot process is so valuable.

How have the questions changed in response to feedback?

(Some changes to wording and options is based on findings from early user testing and not from the more general feedback we gained via the user feedback forms.)

We’ve slightly changed the lay-out of questions and added some more navigational text to clarify how to answer them.

We’ve removed or clarified some terms that were not well understood. Overall we know there is a need for a glossary – ideally with examples and links. That is something Lou will be working on for the future service. We’ve also changed a couple of examples we were using for illustration. There have been many discussions about the pros and cons of examples. Some people find generic terms difficult to understand without examples: but more people object when examples are used, because they favour some applications or approaches over others that are equally valid. Examples can confuse further: ‘if I don’t use that tool, I’m obviously not doing it (right)’. Overall we have gone light on examples, and we hope users’ understanding of terms will improve when we have a detailed glossary we can link to.

We have tried to focus more on activities users do at work, in an educational organisation (college or university). There were some negative comments about references to digital practices beyond this space. However, because of the need to cover a very wide range of roles – and because some roles don’t allow people to express digital capabilities they actually have – we can’t avoid offering some examples from beyond a narrowly-defined work role. For example, one of the activities under ‘digital identity’ is ‘manage social media for an organisation, group or team‘, and under ‘data literacy’ we have ‘judge the credibility of statistics used in public debate’. This is to allow users who don’t manage social media or evaluate statistics as part of their job to reflect on whether they have these capabilities anyway – perhaps gained in their personal life or another role. And indeed to consider whether these activities might be useful to them.

We’ve changed several references to social media, as a number of users objected to what they felt was an underlying assumption that social media would or should be used, and that this was a positive sign of capability. There are still several ways that users can show they are making wise judgements about the appropriateness of social media.

We’ve tried our best to use prompts that reflect capability (‘could do’, ‘would do’, ‘have ever done’) rather than current practice (‘do’, ‘do regularly’), which may be constrained by organisational issues or may reflect judgements not to use. However, we are also mindful that self-reported practice (‘I actually do this’) is usually more accurate than self-reported ability (‘I could do this if I wanted to’). Where we feel it is justified, we have continued to ask about actual use. So long as users understand that they are not being judged, it seems appropriate for the questions and feedback to indicate areas where they are not as capable as they might be if their organisation were more supportive of different practices, or their job role offered more digital opportunities.

There have been changes to the teaching questions, again to focus on pedagogical judgement rather than digital practice. There are now quite a number of caveats e.g. ‘if appropriate to my learners‘, which were suggested by more expert users. Of course we always listen to our experts (!) but as designers we’re aware that introducing caveats like this makes the questions longer and more complex, creating more cognitive load for users, and potential annoyance. We will monitor completion rates to see if this is a problem.

We have particularly reviewed the assessment questions and the online learning questions to be sure we are covering the very wide range of good practice in these areas.

There follows more detail on specific question types and the changes we have made to each of these.

‘Confidence’ questions

Why have we included questions that ask users ‘How confident do you feel about..?’ when we know that self-assessed confidence is generally unreliable? We do this at the start of each element to give users an orientation towards the questions that follow – ‘this is the area of practice we are looking at next’ – and a sense that they are in control. By trusting users to rate themselves, we are both reassuring them that they are not being ‘tested’, and asking them to be honest and searching in their responses. We have weighted the scoring for this question at a low level to reflect users tendency to answer inaccurately – though in fact when we came to compare confidence scores with scores on the other two question types in the same area of practice, there was a positive match.

In feedback, quite a number of users mentioned the tone of these questions positively. Screen Shot 2018-05-14 at 15.03.11However, some felt that they were too ‘subjective’, or ‘vague’. We have tried to deal with this in the update by focusing some questions more tightly on specific practices within the overall area we are looking at. So for example in the generic staff set, under ‘digital creativity’ we ask: ‘How confident are you creating digital content e.g. video, audio, animations, graphics, web pages?’ In the teaching set, under ‘learning resources’, we ask ‘How confident are you about using digital resources within the rules of copyright?‘ We have to find a practice that is generic enough to be available to staff in a wide variety of different roles, but specific enough for the response to feel rooted in a real-world activity.

We have had internal discussions about whether to move the confidence questions to the end of each set, or to remove them altogether. For now they stay where they are.


‘Depth’ questions

These questions are the most difficult to write and currently the most troublesome to end users. There are some ongoing issues with how they are presented on screen, and we are looking into whether any improvements are possible, but for now we have reworded the questions to make the steps to answer them as clear as we can.

These questions offer a short situation or example. Users select the one response that best matches what they would do or what expertise they have. The lay-out of the question reflects the progression logic: the first option reflects the lowest level of judgement or expertise, and the fourth option reflects the highest. There is no trickery here. We describe how progressively more expert practitioners think or act, and ask users to report where they sit on that scale. (At the moment, the visual cues do not make clear that it is a scale, or that higher levels of judgement encompass aScreen Shot 2018-05-14 at 14.45.16nd include the lower ones.)


Beyond the difficulties some users had in ‘reading’ the answer logic for these questions, it is clear that we have to get the progression logic right in each case. When people disagree with our judgement about what is ‘more expert’, they don’t like these questions. When they agree, they say they are ‘nuanced’, ‘thoughtful’, and ‘made me think‘. We know that our users expect us to reflect issues of judgement and discrimination (‘how well is
digital technology being used?’) at least as much as extent of use (‘how many different digital tools?’). So we know these questions have to be in there. They have to reflect important issues of digital thinking or mindset, and we have to get them right – in a very small number of words!

Our recent updates aim to clarify the focus on judgement and experience rather than extent of use. And we have added modifiers such as ‘when appropriate’ or ‘if appropriate for your learners’ (teaching staff) to emphasise that we don’t believe technology is always the answer – but good judgement about technology is. This creates more words on the screen, which will put off some users, but we want our champions to feel that our words represent thoughtful
practice and not a shallow checklist of skills.

‘Breadth’ questions

Screen Shot 2018-05-14 at 14.48.55These are in many ways the most unproblematic. They offer a range of digital activities that staff may do already, may want to do, or may not even have thought about. As before, we try to clarify that we don’t think digital practices are always the best, but we do want people to extend their repertoire so they have more experience of what does (and doesn’t) work. We try to use wording that values skills users have, even if they can’t use them currently due to their role or organisational context. We have tried to avoid very role-specific activities, but not to preclude the possibility that people might develop some professionally-relevant skills in their personal lives, or take on tasks from ‘other’ roles that they enjoy. We include fairly basic activities that many users will be able to select, and quite advanced activities that offer something to aspire to. The ‘nudge’ information is obvious: think about doing some of these things if you don’t or can’t already.


What next?

We are always interested in your views on the questions and other content. The user feedback forms will remain live until the end of the pilot project and we expect to make some further updates to content at that point. If you are an institutional lead, you will shortly have an opportunity to give us feedback via your own detailed evaluation survey.

14th UK Learning Analytics Network meeting, University of Gloucestershire, Cheltenham 12 June 2018

Originally posted on Effective Learning Analytics.

Jisc’s next learning analytics network meeting is in Cheltenham at the University of Gloucestershire on 12th June 2018. These popular events comprise a range of presentations and discussion sessions – and an opportunity to network with colleagues involved in learning analytics projects at other institutions.


The theme for this meeting is “Beyond Retention”, we will see some examples of new data sources around student feedback, explore ideas for supporting student success and improving teaching and get some suppliers views on interventions. Consider the following and come prepared to share your thoughts

How can we enhance students’ performance through analytics? What are the key indicators of successful students. What information should we gather? What parameters can we use and what is ethical? How can we capture the uncapturable, the unmeasurable learning that goes on in the café, the corridor and in halls? How can we reward good behaviour, can we gamify these systems? As humans we respond well to tasks, can we adapt our systems to incorporate these tasks?

There is a growing debate about interventions and how these can be applied within the Learning Analytics context. How far should interventions go and how measurable are they? Should interventions be automated or manual? Can recording interventions be effective and what format should they take. Students are barraged with notifications, alerts and messages from a host of apps and social media. Will interventions and alerts just contribute to that noise?

The draft agenda is below. Please register early to ensure a place at the event.


10:00 – 16:00, Tuesday 12 June 2018
Hosted by the University of Gloucestershire

University of Gloucestershire, The Park, Cheltenham, GL50 2RH

Registration form   Transport & parking     Campus Maps

09:30 – 10:15 Arrival and coffee
10:15 – 10:45 Arrangements for the day & welcome to the University of Gloucestershire
10:45 – 11:15 Update on Jisc effective learning analytics project  Michael Webb, Rob Wyn Jones, Jisc
11:15 – 12:00 Unitu – engaging with realtime student feedback Anish Bagga, CEO and Founder of Unitu
12:00 – 12:15 Coffee
12:15 – 13:00 Using student questionnaires in analytics – HEFCE catalyst A project
Christine Couper, University of Greenwich
13:00 – 14:00 Lunch
14:00 – 15:00  Interventions how, when and why (Panel Session)   Invited experts, chaired by Steve Hoole, Jisc
15:00 – 15:15 Coffee
15:15 – 15:55 Beyond Retention Workshop Sarah Davies, Jisc
15:55 – 16:00 Farewell


CC: BYE, an update to the last post

Originally posted on lawrie : converged.

In my last post I wrote

…I want my work to be seen, used and adapted, but I want it done in a way that ethically aligns with my values. I do not mind if it is used commercially, but I do want to hold people to an ethical standard. Do I want an ethics rider for my Creative Commons license?

This was done in the wake of  Cambridge Analytica and debates about algorithms and analytics.

Until today my work was licenced CC:BY but over the month since I wrote it I have been reading around the debate, most recently Audrey’s Watters’ brilliant “Invisible Labor and Digital Utopias

I want to come clean at the outset, I am a white middle-aged male, and I recognise that I am privileged in my online interactions in ways that the majority of people are not. But I also have constraints on me because of the culture of organisations I have worked in over 20 years.

Audrey wrote of trolls and bigots and that is a big part of being a woman online, or a person of colour, or a minority, or just being anything other than a “straight white man”. But one passage stood out for me.

“As a woman who writes online about technology, I have grown far too tired of “permission-less-ness.” Because “open” doesn’t just mean using my work for free without asking.”

I like to think that I work in a collaborative and open way, I aspire to open practice. But more and more in this culture (the rise of the neo-liberal education agenda?) I see people’s work being taken and used, people like Audrey. I have seen Audrey’s, Dave Cormier’s and others work used in workshops, and keynotes with a minimal 14pt “thanks to…” I have sat in an auditorium and seen my argument, and slide appear in a keynote from a person that was in my keynote a few weeks earlier.

If you quote me – put the citation on the damn slide, I don’t care if you are arguing with the premise, or agreeing with the premise, if I said it cite it.

“Permission-less-ness” seems to be the norm for the way some people work in education, and perhaps more so EdTech.

So today I said CC: BYE.

At the moment I have decided that I am happy to try attribution, no-derivatives, non-commercial. (CC:BY:NC:ND)

So by definition I am now “categorised” as Least Open

So I guess I am now wondering where this leaves me with my open practice? I am not an artist, nor am I employed as an academic but my ideas and my content are very much my livelihood, my identity. This is why it matters, why the licence matters. Generosity has limits, both in how you use my work, and how you cite my work. Part of my identity is now “least open”.

Time to analyse and present your tracker data

Originally posted on Jisc Digital Student.

 Screen Shot 2018-05-02 at 22.02.05

UK trackers are now closed, and we can confirm we’ve collected over 39,000 responses from around 90 UK institutions. That’s almost double last year’s response rate and number of institutions engaged.

After all the hard work of engaging students, we know that data analysis can seem daunting. That’s why this year we want to make it really easy to analyse and present your data.


We’ve provided a handy summary of your analysis options via this blog post, which links to detailed guides. Don’t forget that you will be able to benchmark your data with the rest of your sector very shortly.  So you can download clear, visual reports of your responses – and the associated benchmarking results – directly from the Tracker platform.


This is still quite a lot of information to deal with. So this year, thanks to some brilliant creative work by Tabetha, we are pleased to offer you a PowerPoint template and associated Excel template for organising and presenting your key findings. Here are the links you need to get started.image001

Based on last year’s feedback, we have selected those questions we think will have the greatest relevance and impact at organisational level. The excel file identifies the relevant question numbers, allowing you to drop in your data and quickly generate charts and tables. The charts and tables can then be exported straight into the powerpoint slides, which include explanatory text and colour coding for the different sections of the Tracker.


The summary slide presents six key metrics about the institutional and course-based digital experience. This is followed by more detailed slides, one from each of the four sections.

image003So now you can present your key findings with clarity and credibility, and a minimum of effort. Of course you can always cut out slides and bullet points you don’t want to use – and you can add in more detail in the same style. But we think these two documents give you the main narrative, and we really hope you find them useful.

Coming up

Next week will see the arrival of benchmarking data in your trackers – look out for emails from Tabetha and Mike to let you know it is ready.

Also in the coming weeks we hope to share with you:

  • A poster template you can customise and use to share your findings with students and staff
  • A new digital student benchmarking tool – now mapped to the Tracker questions – so Student Unions and Guilds can drive the agenda for improvement
  • In discussion, a ‘digital toolkit’ template, allowing you to draw on your tracker data to create a toolkit for incoming students.

Please let us know what you think of these new resources. And remember, we are here to make your data do the work, so you don’t have to ?

Engaging users with the Digital discovery tool

Originally posted on Jisc Building Digital Capability Blog .

There are only a few weeks to go before we wrap up this pilot phase of the the Digital discovery tool, but still time to get new users involved. Some pilot sites have finished engaging users and are now evaluating how things have gone, but others are still looking for staff and students to give the discovery tool a try.

There are five new promotional posters from the Jisc team that can help. These can be adapted with an institutional logo and the details of any live workshops or support materials.

Screen Shot 2018-04-19 at 23.17.53

Download your posters here:

There are other ideas for engaging users on our Guidance page: Engaging users.

Thinking ahead, lead contacts at all the pilot sites will be sent a survey about their experience on 5 May. The survey is quite comprehensive, as this is our best source of information about how the Digital discovery tool is being used in practice. There are 15 questions, covering user engagement, support and follow-up for the discovery tool, and whether there have been any individual or organisational benefits. We ask for this to be completed by 30 May.

Before completing the form, we suggest that leads run a focus group or consultation event with users. This will allow evidence to be gathered that can help to answer the evaluation questions. There are materials for running consultation events on our Guidance page: evaluating with users, but this doesn’t have to be complicated. It could be as simple as getting some users together and exploring a couple of the questions on the evaluation form.

Just now, we are using all the valuable feedback from users to make some refinements. You may notice these in the questions and feedback for staff. There will be more significant updates once the pilot has finished. It’s really helpful if you can point your users to these feedback forms, which are found on their dashboards. We can only make things better with their help – and yours!


#CAN18 resources

Originally posted on Change Agents' Network.

IMG_1282The Change Agents’ Network conference at the University of Winchester on 19 and 20 April was an opportunity for staff and students to share their experiences in organised sessions as well as many informal ways; the breaks and meals in the warm sunshine encouraged us to sit and talk.  The conference was excellently organised and supported by the Winchester team who we thank for the considerable effort involved in putting it all together.

In the coming week, we will be adding resources from the sessions and photos.  While you’re waiting, here is a little sample of of images from the event.

IMG_6400 IMG_1279 IMG_6392 ATCV2827 image001


Analytics lab: teaching quality benchmarks

Originally posted on Effective Learning Analytics.

Exciting new Jisc Learning Analytics opportunity

At our most recent network event in Edinburgh, Michael Webb and I introduced ‘Learning Analytics Labs’. An experiment to see whether we can use selected learning analytics data to improve mandatory data returns and performance metrics. This is your opportunity to get involved in an initial exploration to improve teaching quality metrics.

We want to explore whether teaching quality benchmarks can be automated and enhanced from learning analytics data. We’ll determine whether the data can serve up some rich and reliable insights to help institutions to better address regulatory requirements (such as TEF2 and subsequent iterations) as well as contribute insights to related institutional uses.

This will only happen if we undertake some experimentation and with your help we can!

Jisc will provide a safe and secure data processing environment, developers and a data sharing agreement to undertake the work. But we also need your help.

We’re seeking

  1. Champions (institutional lead learning analytics contacts) to join a team and advise us on direction
    Effort expectation 2 hours per week (remote via Skype / Skype for Business) and 3 days face to face, over 12 weeks
  2. Data
    To be decided by the team, but likely to be pseudonymous (where identifiers have been replaced) historic data as used by the Jisc Learning Analytics Service, including student (UDD) data and activity data

You’ll bring expertise and knowledge of the data, the sector, the challenges and the potential for benchmarks which might be created from the data. You will join a small team of peers and be supported in identifying the most promising benchmarking areas for exploration and the data detail required.

In return you’ll get the opportunity to steer future mandatory collections / returns, helping to make these less burdensome and more useful. We’ll also invite you to meet 10 other teams exploring different areas of exploration and you’ll see a range of data visualisations for a variety of purposes.

Our target date for launching this initiative is 16 May at a face-to-face event in Manchester (Jisc will cover your travel costs for both F2F meetings).

At this stage we’re mocking up proofs of concept only – no institutional data will be published for benchmarking. We’re hoping that once we show what is possible, it will spark more interest in the potential ultimately benefiting the whole sector.

If you’re interested in taking part, email  by 30 April outlining in 200 words or less how your experience would contribute to the task. We’ll shortlist and invite successful applicants to the 16 May event and you’ll join a planning F2F meeting with your team the week after.

You can read a little more in Niall’s blog post at

or the morning video of our presentation is amongst those at

and the slides I used are at

Thanks for reading!

Lee Baylis

Senior Analytics Innovator


Organisational data available

Originally posted on Jisc Building Digital Capability Blog .

If you are part of our organisational pilot of the Digital discovery tool, you will now have access to your data dashboard with visual results from your staff users. Guidance for accessing and reading your data visualisations can be found here.

There is also a collaborate webinar on Tuesday 17 April at 13:00 which will walk you through the process and help you to make use of your data. You can access the webinar live here, or after the event you can access the recording here.

The rest of this post is about how you might make use of the data in your organisation. Please remember that the data provided as part of the pilot is still in development. We are in the process of finding out what data is useful. You should not rely on these data visualisations as a definitive source of information about staff training needs.

Making use of your data

You may want to use the number of staff completions – possibly broken down by department – to compare the number of staff who have fully engaged with the number of staff you hoped to reach at the start of the project. Who has and who has not engaged? Do you have feedback from your engagement sessions or a follow-up process (e.g. focus group) to explain any differences? How might you encourage engagement from other groups of staff?

You could also compare the number of staff who have completed the general (‘all staff’) assessment with the number completing the specialist teaching assessment(s). How would you explain any differences? Again consult with your users: were teaching staff more motivated and satisfied by the role-specific assessment?

The ‘in progress’ data allows you to see if there is a significant drop-off as staff are going through an assessment. This is a figure Jisc is looking at closely, as the user experience needs to be easy and supportive – that is our responsibility. But if you find differences in the drop-off rate across different staff groups, could this be because of differences in the support you make available to them?

Scoring band data should be interpreted with great caution. Jisc is using this data to ensure that the questions we ask produce a reasonably even spread of medians across the different areas of digital capability. But this is a broad aspiration: it is inevitable that some areas will prove more challenging to users than others. Also, some areas are essential for all staff (such as digital wellbeing), while others such as information, media or data literacy are more important in different roles.

This is why all our feedback to individual users asks them to reflect on their role and its demands before deciding how to prioritise their next steps. It is also why you should not compare scoring bands across completely different areas of digital capability and conclude that your staff have a ‘deficit’ in one area as compared with another. If you want to make comparisons, look at overall sector scoring bands and compare with the relevant banding in your organisation. But even this should be done with great care, particularly if you have a low number of users overall or in one departmental group, as this will skew the results.

Scores are all self-assigned, and their purpose is to ensure that users get appropriate feedback. If staff believe that their scores are being used for another purpose, they may not answer questions honestly, and the value of the Digital discovery tool will be severely limited.

Jisc encourages you to use the Digital discovery tool to support a dialogue with staff about the training and development they need. The spread of scoring bands across different departments may encourage you to target training in specific areas towards specific groups of staff. Because of the caveats above, you should not do this without consulting with the staff involved. Where staff score lower than others in their sector, this is definitely a cue for you to investigate whether they would appreciate more training and support, but it is not a performance measure and should never be used as such.

Following up and closing the feedback loop

The information you gather from the Digital discovery tool can be used to start conversations:

  • with HR and staff development about overall staff training and development needs;
  • with teaching staff about their confidence with digital teaching, learning and assessment, and their further development needs;
  • with IT and e-learning teams about support for specific systems and practices;
  • with budget-holders about investing in staff development resources and in online services.

You should report back to your staff users about how you are using this data, and what you are doing to support them more effectively in the future.