Measuring Effectiveness John Pyne Consumer Information Department 15 December 2005

Measuring effectiveness Why should we measure? What should we measure? How should we go about it?

Food For Thought? “it is difficult to come to any other conclusion except that the achievement of the objectives which have been set for the FSA are non- operational in the sense that no measurement of success can be achieved” Professor Charles Goodhart, London School of Economics.

Measuring effectivness – WHY? Indicates progress against objectives Tells you what is working, what is not Informs strategic planning and resource allocation Focuses on value for money with limited resources Meets stakeholder expectations, as eventually some or all of the following will demand proof of effectiveness: –Consumers –Industry –Government –Media

Effectivness – WHAT TO MEASURE? Measure Inputs – are we doing things right? Information resources Publications Website Communication channels Consumer Helpline Information centre Website Advertising and media Measure Outputs – are we doing the right things? What is a ‘better informed consumer? Are Irish consumers better informed now than before? Do they know the costs, risks and benefits of the products available? Is the market safer and fairer than it was before?

Measuring effectivness – HOW ? Inputs Examine the main activities you undertake Decide the key performance indicators Measure achievement against these Outputs More complex and difficult to measure long term goals Do we influence consumer behaviour? How do we separate our influence from other ‘influencers’ - TV, press and other media, government, consumer groups, lobbyists.

Measuring effectiveness - our activities What we measured Awareness levels Satisfaction with helpline Website evaluations Publications evaluations

Measuring Awareness of the Financial Regulator -overall increase in the year to July 05 is up from 10% to 30%

Quantitative assessment of the Lo-Call centre’s performance Satisfaction with the Lo-call line is very high Callers are disproportionatly aged years ¾ of queries are claimed as resolved on first call 9/10 will use again and/or recommended to a friend “it wasn’t full of jargon, they didn’t use terminology that is only used in the financial services business” “it was actually a joy because so many people now have such horrible telephone manners and you're talking to machines and its press 1 for option this and then you end up getting cut off so it was a great relief to be connected to a human straight off”

Measuring effectivness of communication channels – website Qualitative research – focus groups & one-to-one accompanied surfs. Task-based assignments to assess the website in terms of: Design Content Functionality Exploration of comparative websites – Expectations vs. reality

Website Evaluation - Findings  “It is just bland. It just looks dull and boring.”  “It is just a real black and white website. There are no pictures, nothing visual to hold your attention.” Initial Impact Very little ‘excitement’ about the website Criticised for looking a bit ‘dull’ A number claimed it is a bit ‘boring looking’ Recognition that this is a ‘serious’ subject Some argue serious doesn’t have to mean boring Calls for more pictures and images on the site Most want a bit more ‘life’ Other Websites The site ‘competes’ with other websites for ‘entertainment value’. It is easy to leave the site because ‘first look’ does not engage the reader.

Publications evaluation - findings INSTITUTIONAL ‘feel’ Little sense of ‘independence’ Likely to be confused with bank brochure Old fashioned – ‘looks like school text book’ “I thought they were trying to sell me a mortgage. I could imagine picking that up in the bank and it would really be just promoting their products” “PICK ME UP” VALUE Nothing new at a first glance Dull use of colours, looks boring Few claimed they would be tempted to pick up

Measuring effectiveness - our long term goals What are our goals? Better informed consumers Aware of costs, risks, benefits Safe and fair market What do we expect to see if our goals are met?

Better informed consumers What attitudes, skills and buyer behaviour might indicate consumers are ‘better informed?’ they seek appropriate information and shop around before choosing a product or service; they know what products they need, at appropriate times in their lives they budget for things and plan ahead they are not usually persuaded to buy, but proactively choose products to suit their circumstances they know how to seek redress if things go wrong; Some possibilities?

Awareness of costs, risks and benefits What attitudes, skills and buyer behaviour might indicate consumers know more about the costs, risks and benefits? they know the relationship between risk and return they know the status of financial advisors and the implications of ‘tied’ vs. ‘independent’ they know how to compare similar products they recognise when they need advice they ask questions, and don’t rely completely on the advice of any party Some possibilities?

A safe and fair market What indicators can we use to assess how safe or fair the market is? Product transparency – degree to which warnings, penalties etc. are signalled ‘up-front’ clearly and simply Degree of firms’ compliance with Codes of Conduct Sanctions imposed on non-compliant firms Competition – number of firms, product competition, price variation, degree of product differentiation, comparative advertising by firms. Some possibilities

Measuring effectiveness - HOW Longitudinal surveys – perhaps every five years plus Analysis of industry data Have to measure actual buyer behaviour as well as knowledge, attitudes and skills Must allow for fact that there is no ‘ideal’ of the informed consumer. Must allow for fact that consumers are not a homogenous group. Methodology? Issues?

Consumer Protection Irish Financial Regulator in in the process of finalising a unified Code regulating the Conduct of Business of: Banks, Insurance Companies, Financial Intermediaries, Investment Managers, and Money-Lenders. The Code will also be extended to cover Stockbrokers and Credit Unions. Question: How can we ensure that effectiveness is a key criterion in the Code development process?

Ways of Considering Effectiveness of Codes Consultation with Industry, Regulatory Impact Analysis, International Best Practice.

Consultation with Industry Benefits: Knowledge of markets, Interest in reputation of the Industry, Knowledge of Impacts, particularly Costs. Risks: Regulatory Capture.

Regulatory Impact Analysis (RIA) Irish Government Requires All new Regulatory initiatives to be subjected to RIA. The Financial Regulator has just published its RIA in relation to the Code of Conduct : Key Elements:  Costs and Benefits  Impact on Competition  Impact on Smaller Firms

Returning to Professor Goodhart: His conclusion was that one of the most valuable ways of considering the effectiveness of financial regulators was peer review. This supports the view that the Financial Consumer Education and Protection Forum is a very worthwhile initiative, and one that we should seek to exploit to its maximum potential.

Thank you