The process of data collection Teodora Brandmüller

Did things happen according to plan? Content of my talk Plan What to do? How to do? Act How to improve? Do Do what was planned Check Did things happen according to plan? PDCA (aka the Deming Cycle, Shewhart cycle, or Deming Wheel) is an iterative four-step quality control strategy. The Shewhart Cycle PLAN  establish the objectives and processes necessary to deliver results in accordance with the specifications. DO  implement the processes. CHECK  monitor and evaluate the processes and results against objectives and Specifications and report the outcome. ACT  apply actions to the outcome for necessary improvement. This means reviewing all steps (Plan, Do, Check, Act) and modifying the process to improve it before its next implementation. Made popular by Dr. W. Edwards Deming, Father of Modern Quality Control, but always referred to by him as the "Shewhart cycle." Later in Deming's career, he modified PDCA to "Plan, Do, Study, Act" (PDSA) so as to better describe his recommendations. PDCA should be repeatedly implemented, as quickly as possible, in upward spirals that converge on the ultimate goal, each cycle closer than the previous. This approach is based on the understanding that our knowledge and skills are always limited, but improving as we go. Often, key information is unknown, or unknowable. Rather than enter "analysis paralysis" to get it perfect the first time, it is better to be approximately right than exactly wrong. Over time and with better knowledge and skills, PDCA will help define the ideal goal, as well as help get us there. Velocity of change is a key competitive factor in today's world. PDCA allows for quantum breakthroughs (typical Western approach), as well as Kaizen (typical Eastern approach with continuous improvement); thereby providing the best of both worlds. In this way, PDCA helps ensure the fastest rate of improvement; often a critical success factor. The power of Deming's concept is in its simplicity. While easy to understand, it is often difficult to accomplish on a on-going basis due to complacency, distractions, loss of focus, lack of commitment, re-assigned priorities, lack of resources, etc. While most claim full knowledge and on-going application, few have in-depth understanding, and even fewer practice PDCA on a consistent basis.

The quality check

Process of quality controll validate variable values Urban Audit Data Quality Control Update variables and indicators in the UA database control indicator values Checked database compute indicators using corrected values control variable values validate variable values

Variable controls Multivariate variable controls Male Resident Population + Female Resident Population = Total Resident Population Univariate variable control sum of the total residents at the sub-city district levels = total residents number at the city level Multi-variate controls involve more than 1 variable. The data collected enables to control the provided statistics for a specified variable by summing up values of other, related variables. By way of example the values of DE1002V Male Resident Population and DE1003V Female Resident Population were summed up and checked against the provided values of DE1001V Total Resident Population at the city level. The same control was applied at values provided at different spatial levels, i.e. LUZ level, SCD1 & SCD2 and national level. Some 111 multi-variate variable controls were defined and agreed by Eurostat. The detailed description of the controls and the algorithms applied is presented in annex A5. Uni-variate controls concern only one variable. Similar to the multi-variate controls, provided variable values were compared with calculated sums, this time not at the same spatial level but across, i.e. values at one spatial level were compared with sums of values provided for the same variable at another spatial level. This is possible due to the hierarchy of the spatial levels in the Urban Audit. For example, uni-variate control of DE1001V Total Resident Population consists in checking the value provided at City level with the sum of the values for the same variable DE1001V provided at SCD1/2 level for the same location. Some 183 uni-variate variable controls were defined and agreed by Eurostat (see annex A6).

Indicator controls Fixed ranges of accepted values according to common sense Nationals born abroad as a percentage of total population – Range 0%-20% Acceptable ranges of indicator values calculated using descriptive statistics The individual indicator values at each spatial level were checked against fixed control ranges defined by GHK International in the framework of the Urban Audit 2 Lot 1 project. The table defining the fixed ranges was provided by Eurostat (see annex A7). The individual indicator values at each spatial level were also checked against dynamic control ranges, i.e. the control limits depend on the data itself. Based on Average and Standard Deviation of indicator values over a specified population, i.e. spatial level (cities, LUZ, SCD1, SCD2 or national level) and groups of countries (EU12 – New Member States and Candidate Countries, EU15 – Old Member States, EU27 – All Countries), the control range was determined by interval +- * . The control consists in checking if the individual indicator value is out of the control range. In this case, the indicator value is not accepted by the control and need to be validated. To avoid very large control ranges (and possibly negative minimum values) a maximum for the coefficient , i.e. the upper and lower control limit, has been set to the ratio average and standard deviation / . For example, the control limits of SA1026I “Proportion of non-conventional dwellings” for 1999 based on the City values of the group EU15 are set to 0 – 2.73: the average is 1.37 and the standard deviation 1.76, so  =0.78 that is /.  Exceptions of these setting are the indicators DE1061I “Total population change over 1 year”, DE1062I “Total population change over 5 years” and EC1201I “Annual average change in employment over 5 years” because negative values are possible. To ensure meaningful descriptive statistics, a minimum number of values in the population is required per reference period to compute average and standard deviation. It has been decided by Eurostat to require a minimum number of 6 values. The control limits are set depending on the number of data points as shown in the table. Indicator controls are not applied if the concerned number of data points is less than 6.

Variable controls Based on Average and Standard Deviation of indicator values over a specified population, i.e. spatial level (cities, LUZ, SCD1, SCD2 or national level) and groups of countries (EU12 – New Member States and Candidate Countries, EU15 – Old Member States, EU27 – All Countries), the control range was determined. The chart presents data for the indicator DE2004I Proportion of nationals born abroad. Data for the 15 EU Memnber States in 2001 is displayed. The acceptable range marked with orange was calculated on the basis on the average and standard deviation. Average +- z * standard deviation where z = 2 but limited by Average/ standard deviation Average 6.2 standard deviation 4.2

Variables to validate Results of the variable controls Results of the indicator controls Indicators not falling within the “accepted” control range were considered potentially erroneous . All the variables required to compute the indicators were listed to be validated.

Assessing lesson learned

Conclusions and actions (1) Variable list needs review on-going in the “Urban Audit Think Tank” There are still national or even regional/local definitions applied The usage of flags and footnotes need to be clarified Eurostat prepared a proposal which gives clearer instructions

Conclusions and actions (2) Better and clearer definition and labelling The update of the Methodological Handbook is foreseen A reference guide containing more detailed meta data is needed The preparation of a Reference Guide is foreseen

Preparatory work

Series of Think Tank meetings Major achievements The list of indicators for the 2006 data collection was reviewed Special focus on environmental indicators Integrate the UN Indicators? Improvement of spatial delimitations (especially LUZ) was discussed The possibility of adding more cities to the Urban Audit was examined

Preparatory work at the Commission and National Statistical Institutes Updating the list of indicators and variables the glossary of variables Bilateral discussions on Additional cities Larger Urban Zones Kernels

Planned timetable Action When Start grant agreements with 25 Member States March 2006 Start the data collection May 2006 Identification of variables June 2006 Finalise data collection January 2007 Quality check of the 2006 results March 2007 Publication of results April 2007

Basic data flow of the Urban Audit provider National Urban Audit Co-ordinator Data is transmitted according to the pre-defined format Production database of Eurostat

Your turn!

Thanks for listening ! Any Questions ? Any questions?