Assessment through multimedia creation

From Case Studies Wiki

The case study explores our experience of assessing students through their creating short multimedia presentations, with the minimum of technical training (The MAIK project).

Contents 1 General Description 2 Context 3 Resources 4 Benefits 5 Evaluation 6 Further Details

General Description

The Department of Information Studies already employs quite diverse assessment methods. The MAIK project (funded by HEA-ICS) explored further diversifying our assessment repertoire through assessing students’ creation of multimedia, in a non-technical module (INF201 Information Management In the Learning Organisation). The popularity of sites like YouTube and of podcasting suggested that the time might be right to test students through these means.

Students were required to produce a three minute presentation using Microsoft PhotoStory, Moviemaker or PowerPoint, integrating sound, text and images to discuss a standard essay type question. Other elements of the assessment were storyboards and plans for the multimedia, a more traditional essay and a reflective piece. Students participated in various activities to develop valid assessment criteria.

Students produced some very high quality work. The case study explores the benefits, costs and issues with this relatively novel form of assessment.

Context

The approach was developed with thirty level 2 students, studying Information Management and Management and Information Management. Although it might be considered that these would be technically “savvy” students, in fact, when we surveyed them, few had experience of using multimedia creation tools as such.

Resources

One of the team is a learning technologist and another has some experience of producing multimedia. In the end, we opted for free software, much of which was already installed on university computers on the standard desktop.

Benefits

In outline the following might be considered to be the potential benefits of this form of assessment:

• Creating multimedia requires significant synthesis and reflection
• The translation of book knowledge into a more visual form involves deeper understanding than simply writing an essay
• It tests forms of skill not normally assessed, e.g. collaborative skills, project management, problem solving, creativity
• The success of YouTube and podcasting suggest that the technology is becoming easier to use. People at student ages seem to be the most active in taking up the technologies.
• The necessary discussion about assessment criteria opens up discussion about assessment in general and reduces the hidden curriculum
• As multimedia may be increasingly used in organisations, e.g. to enhance oral presentations, it is an authentic form of assessment that could increase employability
• Reusable teaching content is generated

Potential issues might be:

• There is a risk of focusing on technical proficiency and presentation not content
• Its easy to assume that young people are all active users of such technologies, certainly there may be inequalities in familiarity (e.g. based on gender)
• Any change in assessment methods produces an element of stress and uncertainty and unexpected impacts on learning processes
• Diversifying assessment methods can produce a lack of coherence across the programme
• Changes to assessment practice have implications for all aspects of the module such as teaching material, learning activities – this is time consuming

Overall, at the end of the project we felt multimedia creation worked well and was fair as a form of assessment. Because the multimedia were only three minutes long a lot of intensive thought and reflection went into the work, making it a deep learning experience. Students enjoyed the novel and authentic challenge. The character of group working was subtly reconfigured, compared to work on a group presentation.

There were some less obvious issues such as that the whole module needed to be redesigned. Despite our best endeavours absolute clarity of assessment criteria was not achieved. It will be a lot easier next year, now that we have examples of the quite diverse approaches to the assignment students took. We thought technical issues might be about using the programs like Photostory, but in fact these are easy to learn. What is less obvious are the media skills to produce well paced, visually powerful multimedia. There are quite complex copyright issues to be considered about how the final multimedia can be shared and reused.

Evaluation

In some senses, the quality of the work students produced speaks for itself. As well as standard departmental module evaluation, we did ask students to complete an evaluation form specifically of the multimedia creation exercise. Overall these were positive and encouraging for a rather novel assessment method. Informal feedback also suggested that students felt the assessment had been fair, though they did pick out a number of issues that pointed to a lack of total clarity of assessment criteria.

Further Details

We are currently working on a paper for publication and we would be happy to share a draft of this with you. If you wish to view the students’ work please do also contact us.

Supporting evidence (photos, multimedia files, relevant documents, links):

1. Questionnaire about students' prior knowledge/attitudes to multimedia and assessment in general
2. Module Assessment guidelines - showing how we constructed a package of assessment elements. Discussed in detail in the draft paper below.
3. IM the movie – handout as part of an exercise to get students thinking in the right frame of mind
4. Guides to using software: PhotoStory. For moviemaker we relied on existing LETS guides: Using Widnows movie maker The film makers toolkit and also Sue Horne's Guide to assembling a digital story. Cics Creating presentations with Microsoft powerpoint 2003 is a good basic introduction.
5. Final Multimedia marking Rubric - developed with student input. University of Wisconsin have some good model rubrics
6. Module evaluation form. Results from this available on request.
7. Staff reflective form – we used this as a way to more systematically capture thoughts about the fairness of the assessment
8. Draft conference paper – this is to be presented at the 9th Annual HEA-ICS conference and reflects on the set up of the project
9. A full paper being prepared for publication is available on request

Useful references:

Ivers K. & Barron E., Multimedia projects in education: designing, producing and assessing. 3rd edition. London: Libraries unlimited (2006). A useful practical guide to planning multimedia assessment.

McDowell L. and Sambell K., The experience of innovative assessment: student perspectives. In: Brown, S. & Glasner, A. (Eds) Assessment matters in Higher Education. Milton Keynes: The Society for Research into Higher Education & the Open University, pp. 71-82. (1999). Useful evidence based guidance on the issues with making any sort of innovation to assessment.

Useful resources:

• Marking rubrics - we found the most useful pre-existing rubrics were available from University of Wisconsin, especially the powerpoint rubric and the video rubric. Many other rubrics can be found through Rubistar.
• Image sources: Flickr, creative commons images - The well known photosharing web site allows users to define various reuse licences to their photos.
• Sound sources: Freesound - offers many reusable sound effects for download. Registration is required, but free. For music try ccmixter jamendo or musopen
• There are innumberable story boarding templates online. We advised students to use Bernajean Porter’s Story Boarding Template
• Web Presentations And Online Slideshow Creation Tools: A Mini-Guide is a useful introduction of web based slideshow creation tools.

To discuss this Case Study/Contact Details:

Andrew Cox/A.M. Cox (http://www.shef.ac.uk/is/staff/cox.html a.m.cox@sheffield.ac.uk),

Ana Cristina Vasconcelos (http://www.shef.ac.uk/is/staff/vasconcelos.htmla.c.vasconcelos@sheffield.ac.uk),

Peter Holdridge (http://www.shef.ac.uk/is/staff/holdridge.html p.g.holdridge@sheffield.ac.uk).