The Value of Information John Medhurst, Larrainzar Consulting Solutions Ltd Maj Ian Stanton, Dstl Ian Mitchell, Dstl
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016 Value of Information Seeks to answer the question – “What is the value of information in NBC defence” Work for this year –Seek to understand one element of NBC decision making – the response to biological detector alarms –Use of an experiment to find out how military personnel use information to decide whether a BW attack has occurred
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016 Experimental Method Simple scenario and abstract, high-level map provided, with sensor locations and rough location of own units Other information presented on a series of cards Key advantages of cards –Allows us to present info in ‘bite sized’ chunks –Allows us to ensure that information is received and understood Players required to decide when to declare ‘Precautionary Alert’ and when to declare ‘Probable Attack’. Corresponds to issuing of NBC 3 and decision to recommend initiation of medical countermeasures. Eight Serials with Latin Square design (8 x 8), order of serials assigned randomly from Latin Square Total of approx 550 data points if all sequences of cards accounted for – all serials carried through to end, even if decision made
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016 What the Cards Look Like
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016 Quantitative Results Analysed using multivariate Probit analysis Simple MV Probit Model –Consists of An intercept for precautionary alert An intercept for probable attack A factor for each of the six card types –Gives a value which can be looked up on the normal distribution to give a probability –Intercepts can be interpreted in terms of Bayesian ‘prior belief’
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016 Discussion Although the simple model ignores elements such as pair-pair variability, these elements are what the probabilistic element of a probit model is intended to represent The simple model accounts for 78% of the variation in P(precautionary alert) and 63% of that in P(probable attack) Key question is how good the model is at predicting behaviour To look at the accuracy of the simple model, we have applied the simple formula on a serial by serial basis A sample of the results (Serials 1-4 for Precautionary Alert) are shown on the next slide
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016 Effectiveness of the Simple Model To assist in using the game as a training aid a ‘DS Solution’ has been developed for the serials Accuracy of the simple model in predicting the ‘DS Solution’ is 75% if p>0.5 is used as the trigger Accuracy of median human pair is slightly less There is potential for a similar algorithm to be incorporated in the NBC BISA as a ‘wake-up’ system Could be calibrated by use of the game as a training aid – would allow a continually updated knowledge base to support decision- making in the field
© Dstl 2003 Dstl is part of the Ministry of Defence 14 February 2016 Conclusions The initial aim of this phase of the study has been achieved – we have established the relative value of information in responding to a possible BW attack There is potential for use of the experimental method as a training game, for which there is a demonstrable need We have a simple model of human response to information, and an experimental method for estimating the effect of individual pieces of information on a decision We also have a simple model of the response to biological alarms that has potential to be used as a decision support tool in the NBC BISA