1. Discussant: Patrick Joseph Sadornas, Bangko Sentral ng Pilipinas (BSP)
The Hidden Content of Absent Transactions An Algorithmic Approach for Identifying Participants’ Operational Outages Marc Glowka, Payments and Settlement Systems, Deutsche Bundesbank
Discussant: Patrick Joseph Sadornas, Bangko Sentral ng Pilipinas (BSP)
DISCLAIMER: The views and opinions expressed in this presentation are solely those of the discussant and do not represent the official position or policy of the BSP.

2. The Problem
“… enlighten the dark area of participant operational outages in FMIs”
Operational outages of FMI participants are relevant to overseers, operators, and participants
We cannot manage what we cannot measure – tackling the measurement problem is an essential first step in risk management
“… until now a comprehensive data set on participants’ outages does not exists”
Recognized limitations in reporting incentives, particularly in cases where reporting is not mandatory
According to the presenter/authors, this data set would allow investigation of relevant questions:
How often do these outages occur?
How long do these outages last?
When do these outages occur?
“… try to find evidence of absence where no such evidence was hitherto available”
Mining of payments transaction data can help provide answers to relevant research questions

3. Methodology
Algorithmic approach to identify participants’ operational outages based on transaction data
Broadly analogous to the approach of Furfine in 1999 to measure the US fed funds market
Indirect measurement of a phenomena due to lack of direct data/reports – e.g. interbank loans and operational outages
Applying an algorithm to payment systems data to generate a data set for further analysis
On the filtering of data: “… those sent by ancillary systems, are excluded as they might be triggered by the ancillary system rather than by the participant”
Transactions initiated by ancillary systems (e.g. PvP/DvP systems) may also be triggered by participants
Presenter/authors identified CLS as being initiated by the participant (an approximation)
Understanding the detailed functioning and interfaces of individual ancillary systems may serve to enrich the data (i.e. provide additional data points for mining)

4. Methodology (cont’d)
On the interval and threshold used: “… a potential outage is identified when the volume of transactions of a participant during a ten-minute interval on a business day lies in the first percentile of the participant over a year”
Question: Why a 10-minute interval? Why use the 1st percentile as threshold for outage?
A sensitivity analysis on the interval used and/or the outage threshold may provide insight into the robustness of the methodology
NOTE: Elizabeth Klee used 15-minute intervals (in the 2007 version of the paper cited)

5. Results and Conclusion
Algorithm and resulting data set is a welcome addition to research work on FMI operational risk
General applicability to other FMIs
Can serve as basis for quantifying operational risk in an FMI arising from participants
However, since generalizations are made about participant outages using the resulting data set, the reliability of the algorithm must be established first
False positives and false negatives may affect the inferences that we make from the data set output by the algorithm
Question: Has your team considered ways to measure the quality of the results? How do you intend to do this?
Criticism of Furfine algorithm by Armantier & Copeland (2015) may be broadly applicable
Subject was different (interbank loans) but also involves use of an algorithm to produce a data set
In their conclusion: “xxx need to validate formally, prior to any analysis, that indirect inferences produced by an algorithm are accurate.”
Armantier, O. & Copeland, A. (October 2015). Challenges in Identifying Interbank Loans. FRBNY Economic Policy Review.
Formally tested the accuracy of the algorithm and concluded that algorithm may be ill-suited to the task of identifying individual loans between two specific banks
For the period , an average of 81% of all pairs of payments identified by the algorithm are not fed funds transactions conducted by the two banks (false positive)
An average of 23% of the banks’ actual fed funds transactions are overlooked by the algorithm (false negative)

6. Some Recommendations
Consider measuring effectiveness of the algorithm using public info. and microprudential data
Obtaining some “ground truths” – also relevant for supervised machine learning (in future plans)
Consider comparing with reports from critical participants (if no legal restrictions on data)
These may provide input to the calibration of the intervals and outage thresholds used
Consider sensitivity analysis of the parameters of the algorithm to assess robustness
Aggregation period (10-minute interval)
Threshold for considering an outage (1st percentile threshold)
Exploring possible data enrichment that can be provided by ancillary systems
Transactions in ancillary systems triggered by participants (i.e. initiation manifests itself in the data) may provide additional clues to operational outages of participants
Transaction must be attributable to participants (usually possible using separate field or reference text)

7. Thank you.