testing the effectiveness of digital storytelling Peter Spyns – Flemish dept. of Economy, Science and Innovation

Flemish government | Department of Economy, Science and Innovation Agenda 2 initial problem on introducing digital storytelling for funding agencies organising an experiment on digital story telling in a call for project funding aim policy context set-up outcomes comments discussion outlook conclusions

Flemish government | Department of Economy, Science and Innovation initial situation 3 con’s of digital storytelling  potential additional threshold  time  technical set up  adequate content  good scenario, style  favours already “strong”/experienced submitters pro’s  one image is worth a 1000 words ( Frederick R. Barnard in 1921 )  not all submitters are excellent writers  for some themes or topics, words fall short problem: how to convince external funding/facilitating agencies/organisations that digital storytelling has an added value for them and their stakeholders and proposal submitters ?

Flemish government | Department of Economy, Science and Innovation aim of the experiment 4 organise a “real life” context to apply digital storytelling find out how popular digital storytelling already is investigate in a serious way the effect of digital storytelling collect evidence-based arguments to convince funding organisations

Flemish government | Department of Economy, Science and Innovation 5 EWI call for proposals on social innovation what: definition of social innovation SI aims in a structural way at meeting a societal need, such as tackling multidimensional societal problems of the most vulnerable groups in society; SI is about a new or significantly improved product, service, process, marketing method or organisational model.  adequate topic how: call document excerpt (translated): OPTIONAL: Submitters are allowed to add, as an extra support to their proposal, a (digital) animation, video or self referencing presentation. By doing so, submitters can visualise better some aspects of their proposal (e.g., innovative features, way of putting to use the results aimed at, etc.). Submitters who want to make advantage of this possibility are invited to contact to arrange to file transfert that has to be done before the deadline. context of the experiment

Flemish government | Department of Economy, Science and Innovation 1)collecting the proposals and – if any – the associated digital story 2)distributing the proposal amongst the reviewers without the digital story 3)regular reviewing process (without the digital story) 4)uploading digital stories to separate (and protected) webpage 5)after the regular review, each expert with a proposal supported by a digital story is invited to also have a look at the digital story and send an with comments about 1)whether or not the digital story adds new information 2)whether or not this new information makes him/her change his/her opinion on one or more of the evaluation criteria 3)whether on not this leads to a change in score for one or more of the evaluation criteria 4)if so, please give a short explanation why 6)a statistical test can determine whether or not a change in expert behaviour is significant or not 6 practical set-up of the experiment

Flemish government | Department of Economy, Science and Innovation 1)on 105 eligible proposals, 11 added a digital story (10,5%) 2)leading theoretically to 23 reviews by 19 different reviewers 3)3 proposals with a digital story were accepted (vs. 17 without) 4)in practice, 4 reviewers responded with 6 reviews  example: Blue Assist movie (a best practice of FP7 INNOSERV project) 7 outcomes of the experiment

Flemish government | Department of Economy, Science and Innovation 1)presentation made more clear what I guessed but did not privde real answer and hence, did not lead to a change in score or opinion 2)movie confirmed my initial negative opinion, but did not lead to a change in score or opinion 3)the URL for the movie was in the proposal so I already had a look at it: it contributed positively to my scores (2x) 4)proposal was well written and contained photos so the movie did not really add but rather confirmed my opinion – hence, no change in score or opinion 5)movie merely confirmed my initial opinion, hence no change in score 8 comments by the participating reviewers

Flemish government | Department of Economy, Science and Innovation 9 discussion on the experiment 1)too few responses for a statistical analysis mostly due to 1)lack of time (regular review was delayed) 2)bad timing (too close to holidays) 3)digital story already taken into account during original review 2)only 10% of the submitters added a digital story 1)could depend on the nature and topic of the proposal 2)for many submitters it was their first proposal 3)in most cases, the digital story merely confirmed the initial impression of the reviewer 4)no change of mind and scores 5)when looked at during regular reviewing, it was considered as a positive source of additional information

Flemish government | Department of Economy, Science and Innovation outlook and subsequent steps participation of “the Flemish Social Innovation Factory” has to stimulate social entrepreneurship and social innovation in Flanders by a.o.  capturing ideas for project proposals  supporting submitters in elaborating their ideas into project proposals  enriching the project proposal to enhance its acceptance potential  guiding social entrepreneurs to setting up a “social entreprise” acts as a thematic front-end for the Flemish Innovation Agency (IWT) that formally approves and funds the submitted proposals have discovered “themselves” digital storytelling are interested in becoming involved in InnoFun and trying out digital story telling as part of their regular way of working  in the long term, that could imply that the IWT accepts a digital story as (additional) part of a proposal we hope to integrate them in the RIP 10

Flemish government | Department of Economy, Science and Innovation 11 conclusions 1)scientifically inconclusive experiment 2)more experiments are needed – maybe in a different setting 3)low participation (submitters and reviewers) 4)effectiveness of digital story telling is questionable 5)it is no source/driver of change in opinion or score 6)at best, it is an extra source to confirm an opinion in the making 7)the “social innovation factory” could be a promising route for adopting digital storytelling as a regular part of a project proposal preparation procedure.

Thank you Department of Economy, Science and Innovation (EWI) Koning Albert II-laan 35 box 10, 1030 Brussels |