It’s time to ask - is it all worth it?

The Education Endowment Foundation toolkit – recommended to be used in conjunction with Pupil Premium – suggests that digital technologies provide “Moderate impact for high cost, based on extensive evidence.” I have to say without reservation that this is a ludicrously sweeping generalisation for such an influential toolkit.

You can read their assertions in more detail on their site but let me respond emphatically with my view:

Different technologies have different levels of impact when used for different reasons by different teachers and students.

Having got frustrated at the EEF headline the content starts to be a little more reasonable as they explain that actually:

“there is considerable variation in impact. Evidence suggests that technology should be used to supplement other teaching, rather than replace more traditional approaches. It is unlikely that particular technologies bring about changes in learning directly, but different technology has the potential to enable changes in teaching and learning interactions, such as by providing more effective feedback for example, or enabling more helpful representations to be used or simply by motivating students to practice more.”

It is precisely this truism which is why it is essential for any ICT Manager within schools to review any expenditure and ensure that the school has got value from it. Let me suggest a strategic way of conducting an “Impact of new ICTs audit”.  This post is part of the Redbridge Management Calendar (ICT-RMC).

Conducting an Impact of new ICT's audit.

1. List all the significant* technologies that you have implemented in your school over the last 12 months. Put these in the first column of a spreadsheet and remember to include online services, software and physical devices in your list.

*”Signficant” means the ones you expended effort or money to implement. Email services within the LGfL community may be free to use, but might still involve training and configuration work to implement.

 

2. Estimate as a percentage (using observable data where possible) what proportion of a typical school day that technology is actively engaged in supporting teaching, learning or administrative functions within your school. Add a second column to your spreadsheet with this data.


3. Estimate as a percentage (using observable data where possible) what proportion of the school community actively use this technology. Add a third column with this percentage…

4. Survey a subset of the user population to establish subjectively** whether the system is important to teaching and learning or administration. Mark this out of 5. 5 out of 5 would mean it is “essential to my every day “and 1 out of 5 would mean “it is rarely useful”. Allow 0 out of 5 to mean that “it has not been of any value at all”. Translate each of these marks to a percentage and then average the results of your mini-survey for each technology. This is your fourth column.

**Subjectivity isn’t bad. It is a quick way of garnering professionals’ qualified opinion or children’s reaction to a technology.

5. Estimate the actual cost to the school of implementing that technology. This is probably best represented as the next two columns – one for financial cost (how many thousands) and another for time taken to implement (how many staff hours were directly involved in implementing the project – both in planning and training activities). Don’t allow zero values – but if something really did was free put 0.1 in this field.

6. Multiply columns 2, 3 and 4 together to get answer 1. Multiply separately columns 5a and 5b to get answer 2. Divide the first answer by the second. (Does this sound like a child’s maths-magic trick?) You now have the answers! Do not expect the resulting metric to mean anything directly! What you have calculated is a comparative ‘impact’ rating. It’s not a perfect measure but it is an indicative measure.

7. You can sort your whole list by the rating.  You can even perform rank calculation to list the order using natural numbers. At one end of your list will be the digital technologies that created highest impact for the least outlay. At the other end will be the high cost – low impact.

What to do now you have completed the audit

Start from the high cost – low impact end of the list and do something about this potential waste of valuable funds. Do you need to provide better training to the users, configure it differently, give the technology to a completely different set of users who will use it better, or simply stop using that technology? Only you will know, but you can’t leave it being a potential waste in school resources.

Then look at the low cost – high impact end. Congratulate yourself on these projects but ask why these systems worked so well. The users found them intuitive? They were free online resources? The teachers/learners could see the educational benefit quickly? …

In the example shown, clearly implementing LGFL's Staffmail service has been high impact for low cost - well worth the time invested.  At the other end of the scale the Netbooks given to year 3 and 4 have cost a lot without confidence that they are having impact.  Can you do anything to increase any of the impact measures - have them being used amongst a larger proportion of the school and therefore active for more of the school day.  Can you provide training so that teachers see them as more valuable?  If not should you move them to another part of the school or just make the decision to not buy any in future?  These are all useful follow on questions that you will start asking once you begin doing analysis like this.

Feed these results and questions back to your ICT Steering Committee (which should include your finance manager, a member of the senior management team, the ICT coordinator and the technical manager) and decide how to act on this. Aim to do more low cost - high impact ICT and prove the research wrong.

Digital technologies can be powerful, inspirational, transformational tools in the classroom – but it is up to us to make that so.

Alex Rees, @alxr1

(This spreadsheet template is available for download from the Redbridge ICT & Computing Fronter room.)