1. Polifonia Conference Sessions 3:
Internal Quality Assurance as Preparation for the Evaluation of a Degree Programme
Christopher Caine – Trinity College of Music Accreditation Working Group

2. Aims of session
Introductory comments on background and purpose of internal quality assurance
Small group discussion to address specific quality assurance related question which relate to the work of sessions 1 and 2
Summary discussion
To begin this session I’ll be making a few observations about internal quality assurance: looking in detail at what is internal quality assurance, why we engage in it and how we might carry it out. I hope to be able to draw parallels between quality assurance, the way in which individuals learn and also the way musicians learn.
There will then be small group discussion and this will be followed by summary observations. By the end of the session I hope that we will be able to have understanding of the practices that take place in the various nations represented here and that groups will have formulated concrete suggestions in relation to the questions that we’ll be posing.

3. Questions for group discussion
How (with what methods) do we assure ourselves that learning outcomes of a programme are, and remain, relevant to the training of the professional musician?
How (using what methods) do we assure ourselves that our assessments are consistent, reliable and valid (relevant to the learning outcomes)?
How (with what methods) do we assure ourselves that our teaching is appropriate for the curriculum?
These are the questions that we’ll be posing. The first one concerns programme design – what gives a programme its currency and validity as a mechanism for professional training.
Assessment – very thorny because we have to make judgements. Students more likely to challenge than in the past as the notion of fees and “the customer” become more common.
The third question deals with delivery of the programme and examines the relevance of the teaching to the curriculum and assessment. This has direct links with the “triangular circle” that Jeremy Cox presented to us during the last session.
So the questions relate to the programme –that is to say what we offer our students; the award – that is to say their reward for taking the programme; and the delivery of the programme. Hopefully the relationship between internal quality assurance and the design of programmes discussed in the previous sessions will be clear.
I thought I would draw briefly on two examples of internal quality assurance from areas outside higher education to begin the process of examining what internal QA is.

4. Toyota – statement on quality assurance
“There are a number of hurdles that this globalization of production has to overcome. Among these the most important is quality assurance, which requires that no matter where Toyota vehicles are made, they have the same quality.”
e-by-toyota-aiming-for-global-quality-assurance
Just to underline a part of this statement “Among these the most important is quality assurance, which requires that no matter where Toyota vehicles are made, they have the same quality”. The statement goes on to say “to put it another way, we don’t put a label on our vehicles which says “Made in such and such a country;” we put the same label on all vehicles which reads “Made by Toyota.”
Keywords – quality, globalisation, reputation
Globalisation – internationalisation – “world class conservatoire”

5. International Union of Pure and Applied Chemistry
Internal quality control
Set of procedures undertaken by laboratory staff for the continuous monitoring of the results of measurements in order to decide whether results are reliable enough to be released.
Emphasis on reliability of results. This implies caution about releasing results before they are tested so as to protect reputation.

6. ENQA (European Association for Quality Assurance in Higher Education)
establishment of a self-evaluation culture
the future quality assurance system should maintain a strong focus on quality improvement.
These quotes come from a report by an ENQA review panel of quality assurance of higher education in Portugal published in November
As one might expect in the lofty environment of higher education there is a slightly different tone. No more plain talk of quality and reliability – instead high ideals and emphasis on culture, reflection evaluation, improvement, striving for something better.

7. “L’assurance qualité interne, qui a son objectif braqué sur l’amélioration constante de la qualité, devient désormais un processus dynamique dans la gestion des hautes écoles.”
(Internal quality assurance, aimed at continual improvement in quality will in the future become a dynamic process in the management of higher education schools.)
“Les résultats d’évaluations internes doivent être utilisés afin d’améliorer continuellement la qualité de l’enseignement et de la recherche.”
(Results of internal evaluations must be used to improve continually the quality of teaching and research.)
(Organe d´accréditation et d´assurance qualité des hautes écoles suisses (OAQ)) o
And this was taken from the Swiss accreditation agency OAQ. Again there are suggestions of high values through the key words, constant improvement of quality; dynamic process, evaluation.
I wouldn’t wish to suggest that on the basis of the small amount of looking around that I have carried out that there is a big difference in the way that more commercial bodies describe quality assurance; I’d need to do much more research to make that claim. However, it’s interesting to see in the very small sample presented here, the difference between higher education language – evaluation, culture, continual improvement, reflection and the examples of Toyota and the chemistry body – quality, reliability, globalisation.

8. What should internal quality assurance consider?
Brisbane Communique
ENQA (European Association for Quality Assurance in Higher Education)
Quality Assurance Agency (UK)
I’m now going to move on to look at what internal quality assurance should consider and you’ll see from the screen the sources that I have consulted.

9. Brisbane Communique Principles:
A quality assurance culture is created, defined, supported, and promulgated.
Quality assurance aligns with and is embedded within the institution’s unique goals and objectives.
Periodic approval, monitoring and review of programs and awards.
A strategy for the continuous enhancement of quality is developed and implemented.
Quality assurance of academic staff is maintained
Appropriate and current information about the institution, its programs, awards and achievements is made publicly available.
(Higher Education Quality Assurance Principles for the Asia Pacific Region: The need for quality assurance principles for the Asia-Pacific Region)
Looking at the first bullet point you have periodic approval monitoring and review of programmes and awards. That is to say approval for the creation of the award; monitoring implies an on-going process of examination; and review implies a re-consideration of the award. The second bullet point suggests that there should be a strategy for improvement and that it should be organised systematically – in other words, this is not a random process. The third bullet point talks of the quality of staff which may encompass teaching methods – in other words, this is a focus on delivery. Then we come to public information – is what it says on the packet, what it actually is? This emphasises the importance of making courses explicit to future students – a nod towards the consumer culture. When I was a student…

10. Policies and procedures for quality assurance
ENQA - European standards and guidelines for internal quality assurance within higher education institutions
Policies and procedures for quality assurance
Approval, monitoring and periodic review of programmes awards
Assessment of students
Quality assurance of teaching staff
Learning resources and student support
Information systems (evidence)
Public information
There are major similarities in the ENQA standards and guidelines – points on systematic methods of quality assurance, approval and monitoring of awards.
The third bullet point concerns assessment – are assessment results fair and reliable – how do we know for instance that 90% in performance is a fair mark. I remember as a student…
Information systems – this is an interesting requirement. How and from where do we gather evidence to support our internal quality assurance processes? How is it analysed and how do we know that this evidence is relevant and un-bias?
Public information – is what we say about our institution fair.

11. Quality Assurance Agency UK
Additionally asks institutions to consider:
Collaborative provision
Placements/work-based learning
Disabilities
Careers
Appeals/Complaints
Admissions
Alongside the focussed issues of curriculum, programme, teaching and assessment, the QA asks UK institutions to consider the displayed items as part of internal quality assurance monitoring. I’m sure that the list of criteria for internal quality assurance could be endless and as Sam Hope said yesterday, we have to know when to stop…

12. External Review/accreditation
Funding
Legal requirement
External Review/accreditation
Reputation/profile (national and international)
Here are some of the main operational imperatives for internal quality assurance. You may be asking yourself why I have put reputation as rather a negative imperative. There is a British expression called “keeping up with the Jones” which describes a state of doing something to keep up with your next door neighbour – in other words, the motivation is competitive rather that ulturistic.

13. Quality We want our courses to be good Student experience
And these are perhaps good reasons for engaging in internal quality assurance. Surely the student experience is of first importance and surely we want our courses to be as good as they can possibly be. I’ve given this slide a musical background and hopefully you’ll see why this is later in the presentation.

14. AEC 2007 – Handbook: Internal Quality Assurance in Higher Education
The PDCA – circle
Plan Do
Check
Adapt
Before group discussion I want to focus on some of the processes involved in quality assurance. P 10 of the AEC handbook talks of the PDCA circle. I quote: “This lies at the very heart of any quality assurance system: Plan what you are going to do; then Do it; afterwards Check if you have done what you meant to do and if it delivered good results; reflect on the results of your checking and prepare Adaptations for future actions; then start again; Plan – Do - Check – Adapt. “Of course the idea behind this is not simply that you execute this circle endlessly. The idea is that, because of structural checking and adapting, your results will become better and better, and you will find yourself not on the treadmill of a circle, but on a staircase in the form of a spiral, aspiring to ever better quality. Or, in the theoretical case that your quality goals have been reached, maintaining your quality on the same high level.
Give module development as an example. Perhaps what is missing here is evidence and information – how do we know whether our ideas for new courses have validity and we could reference the previous slides of the Brisbane Communique and ENQA with regard to information systems.

15. Kolb (1984) The Lewinium Experiential Learning Model Experiential Learning: experience as the source of learning and development New Jersey: Prentice-Hall ( ).
The AEC model is backed up by this theory of individual learning. Kolb (1984) outlines two models of experiential learning, both of which provide a sound theoretical basis for using evaluative methods of learning. The Lewinium experiential learning model describes a continuing process of concrete experience followed by observation and reflection. These inform the formulation of abstract or new concepts which are tested in new situations. These tests form concrete experience and thus, the process continues in a cycle. Let’s look at this again – concrete experience, observations and reflections, formation of abstract concepts and generalisations, testing of concepts in new situations and on it goes. This is not far removed from the AEC method of Plan Do, Check and Adapt. Surely instrumental practice is almost an identical process?

16. Kolb (1984) Dewey’s model of experiential learning
Here’s another theory created by Dewey which is similar, but perhaps makes the continuum of the cycle more explicit. Dewey refers to a process of impulse or action followed by observation. Observation is transformed into knowledge forming the basis of judgement. This judgement is the foundation for another impulse thus commencing another cycle but at a higher level. What can be more akin to learning music than impulse or actions, followed by observations and reflections, supplemented by knowledge which is transformed to judgment which is the basis of another action. Is this not again the process of practising an instrument?
I would like to suggest that these personal learning models can be relevant and applicable to an institution which is learning about itself. Isn’t quality assurance institutional learning? – in other words, internal quality assurance.

17. Questions for group discussion
How (with what methods) do we assure ourselves that learning outcomes of a programme are, and remain, relevant to the training of the professional musician?
How (using what methods) do we assure ourselves that our assessments are consistent, reliable and valid (relevant to the learning outcomes)?
How (with what methods) do we assure ourselves that our teaching is appropriate for the curriculum?
So we now return to the three questions that we will be asking you to consider.
The first question is about programme design – what gives it currency? We all have good ideas for courses – how do we know that they are really that they are really appropriate for professional training. Where will be gather evidence that supports the programmes learner outcomes in the context of training a professional musician. How do we monitor the programme? At what point do we review the programme?
The second question concerns assessment. How do we know that our assessments actually assess the learning outcomes and are valid. I borrow and example from Jeremy Cox with regard to final recitals. Is a solo recital of 40 minutes a relevant professional requirement for the majority of bassoonists and percussionists. How do we know that assessments methods give consistent results? Is 60% given by one panel the same as 60% by another? Is 60% the same in 2010 as it was in 1995? What is the definition of 60% in the context of a performance examination - what does it mean? Is 60% in your institution the same as 60% in another.
The third question is concerned with the relevance of the teaching to the curriculum. This refers to Jeremy Cox’s circular triangle. What is the point in having excellent teaching if it is irrelevant to assessment methods and the curriculum? How do we assure ourselves that teaching quality is sufficient to deliver the learning outcomes?

18. Examine question in the following context
Discussions may relate to the programme and assessment discussions that took place in sessions 1 and 2
Each individual in the group will have experience of his/her own national and/or institutional internal quality assurance requirements and procedures. The aim is to learn from these and to try and suggest efficient practical measures in relation to the question(s)
Remember that whatever is suggested, the quality of the evidence used is paramount and you may also wish to consider the dynamics of quality verses quantity
You may wish to keep in mind the learner outcomes and assessments discussed in the earlier sessions or you may wish to think more broadly. We are hoping that you will share your own ideas and experiences of practical methods relevant to the three questions with each other and propose concrete suggestions for the bigger group. It may be that you will decide that all your ideas are relevant but equally you may decide, if, for instance there is duplication, to drop some of the ideas. We hope however that your group will, having discussed at least one of the questions, formulate efficient and practical methods of internal QA.
Remember that practical methods of internal QA need evidence. As an example, if your are tackling the first question, what evidence will you use to justify the setting up of a new programme? Where will this evidence come from? Feedback forms from students can produce a mass of information but is this information good quality? Will you gather information from just one source?
Each room should divide into two groups and debate at least one question, preferably different questions. Your group facilitators will stop the discussions before the end of the session and will try and collate the concrete suggestions through a whole group summary discussion. We will then try and report back the results of these in the final session.

19. The student experience
We want our courses to be as good as possible
That’s what musicians do

20. Thank you for listening!