1. Appraisal Assistant workshop Evidence collection (ISO15504-5 assessments)

2. Overview Evidence collection overviewEvidence collection overview Evidence related screensEvidence related screens Getting evidences into the databaseGetting evidences into the database –Importing a list of documents –Using spreadsheet to collect initial evidences –Using evidence registry to collect evidences Manage evidences of process outcomes/attribute achievementsManage evidences of process outcomes/attribute achievements Transfer evidence data between assessment projectsTransfer evidence data between assessment projects Workshop exercise setup: project : ISO15504, with ACQ.1, ACQ.2 on target CL5, with instantiation: Org.Process

3. Evidence collection- Overview Evidence collection is guided by the assessment indicatorsEvidence collection is guided by the assessment indicators Map the evidence to the indicator whenMap the evidence to the indicator when –(Part of) the evidence exhibits the same characteristic as a work product indicator –(Part of ) the evidence is considered the input/output of implementing a practice –(Part of) the evidence implies the non/existence of the required infrastructure/resource. Evidence indicates the strength/weakness of the implementationEvidence indicates the strength/weakness of the implementation Do NOT need to find evidences for all assessment indicatorsDo NOT need to find evidences for all assessment indicators –Ratings are based on the satisfaction of the process outcomes / process attribute achievements, not the fulfilment of indicators Evidences are automatic mapped to process outcomes / attribute achievementsEvidences are automatic mapped to process outcomes / attribute achievements Evidences can NOT be mapped to outcomes / achievements without first being mapped to related assessment indicatorsEvidences can NOT be mapped to outcomes / achievements without first being mapped to related assessment indicators

4. Evidence related screens – Evidence registry panel Map evidence to assessment indicatorsMap evidence to assessment indicators Evidences is automatic mapped to generic work product when it is mapped to specific work productEvidences is automatic mapped to generic work product when it is mapped to specific work product Comment how/why evidence is mapped to the indicatorComment how/why evidence is mapped to the indicator Evidence detail informationEvidence detail information Evidence coverage of process outcomes / attribute achievementsEvidence coverage of process outcomes / attribute achievements

5. Evidence related screen – Process Attribute rating tab Form the bases for rating process attributesForm the bases for rating process attributes Evidences mapped to process outcomes / achievementsEvidences mapped to process outcomes / achievements Can NOT add new evidencesCan NOT add new evidences Can remove irrelevant evidence mappingsCan remove irrelevant evidence mappings Characterize the nature of the evidence-outcome mappingCharacterize the nature of the evidence-outcome mapping Comment on how/why the evidence indicates the implementation of process outcomes / attribute achievementsComment on how/why the evidence indicates the implementation of process outcomes / attribute achievements

6. Importing a list of documents - 1 Documents are NOT mapped to assessment indicators (unlike the PII spreadsheets)Documents are NOT mapped to assessment indicators (unlike the PII spreadsheets) Can be extracted from the OU’s document databaseCan be extracted from the OU’s document database Instantiation name must be the same as defined in assessment scopeInstantiation name must be the same as defined in assessment scope –The row will be ignored if the Appraisal Assistant can not find a matched instantiation in the assessment scope Use the included DocumentList.XLS to ensure correct file formatUse the included DocumentList.XLS to ensure correct file format Save the Evidence list tab as TAB delimited Text before importingSave the Evidence list tab as TAB delimited Text before importing

7. Importing a list of documents - 2 3 documents3 documents Belongs to 2 instantiationsBelongs to 2 instantiations Not mapped to any practiceNot mapped to any practice Check error log for rows that are ignoredCheck error log for rows that are ignored List of Document Evidences InstantiationEvidence NameDocument LinkSource/AffirmationDescription org. processesevidence bah!www.gu.edu.aumoomoo's site org. processesEvd 1must import Test project 1EVD 2import once

8. Importing a list of documents - 3 Result of importing the previous document list

9. Using spreadsheet to collect initial evidences - 1 One row per evidence, insert rows for more evidencesOne row per evidence, insert rows for more evidences Leave the row EMPTY if there is NO evidenceLeave the row EMPTY if there is NO evidence Each evidence can be mapped to multiple assessment indicatorsEach evidence can be mapped to multiple assessment indicators Each evidence can NOT be mapped to the same assessment indicator more than onceEach evidence can NOT be mapped to the same assessment indicator more than once Do NOT need to find evidence for ALL indicatorsDo NOT need to find evidence for ALL indicators Consists of all indicators of the entire assessment scopeConsists of all indicators of the entire assessment scope User ensures data accuracy and consistencyUser ensures data accuracy and consistency XML spreadsheets can be opened by Microsoft Excel 2002 or newerXML spreadsheets can be opened by Microsoft Excel 2002 or newer Save as ‘Tab Delimited Text’ file before being importedSave as ‘Tab Delimited Text’ file before being imported Program merges evidences only if they have the same name AND the same document link stringsProgram merges evidences only if they have the same name AND the same document link strings

10. Using spreadsheet to collect initial evidences -2 2 evidences here, mapped to 4 indicators2 evidences here, mapped to 4 indicators Process Instantiation name in the top most cellProcess Instantiation name in the top most cell Detail descriptions of process / process attribute / assessment indicator in cell commentsDetail descriptions of process / process attribute / assessment indicator in cell comments ACQ.1/18-08 has 2 evidencesACQ.1/18-08 has 2 evidences ACQ.1/BP1 has no evidenceACQ.1/BP1 has no evidence Org. Processes Process Attribute Indic. AbbrIndicator Name Evidence NameDocument LinkComment ACQ. 1PA1.118-08Supplier selection criteriaTest evidence 1www.cit.gu.edu.auevidence exhibits the same characteristics Test evidence 2www.sqi.gu.edu.aubah! ACQ. 1PA1.1ACQ.1.BP1Establish the need. ACQ. 1PA1.1ACQ.1.BP2Define the requirements.Test evidence 1www.cit.gu.edu.auevidence defines the criteria of the supplier selection ACQ. 1PA1.1ACQ.1.BP3Review requirements.Test evidence 2www.sqi.gu.edu.audoh!

11. Using spreadsheet to collect initial evidences - Imported 2 evidences imported from the previous spreadsheet2 evidences imported from the previous spreadsheet All 2 evidences belong to the same instantiationAll 2 evidences belong to the same instantiation Evidences mapped to specific work products are automatic mapped to the generic products of the same processEvidences mapped to specific work products are automatic mapped to the generic products of the same process

12. Using spreadsheet to collect initial evidences - 4 4 evidences mapped to 4 specific work products4 evidences mapped to 4 specific work products –The 2 ‘Test evidence A’ have different document links –The 2 ‘Test evidence B’ do not have document link Org. Processes Process Attribute Indic. AbbrIndicator NameEvidence Name Document LinkComment ACQ. 1PA1.108-02Acquisition planTest evidence Awww.sqi.gu.edu.autest evidence A, linked to SQI ACQ. 1PA1.112-01Request for proposal Test evidence Awww.gu.edu.auanother evidence named 'test evidence A', linked to Griffith main page ACQ. 1PA1.113-19Review recordTest evidence B ACQ. 1PA1.115-01Analysis reportTest evidence B another evidence named 'Test evidence B'

13. Using spreadsheet to collect initial evidences - 5 4 evidences are imported from the previous spreadsheet4 evidences are imported from the previous spreadsheet Evidences with duplicated name strings will cause confusions when validating dataEvidences with duplicated name strings will cause confusions when validating data –Use unique name string for evidence –Import routine can merge evidences if they have the same name AND the same document link

14. Using spreadsheet to collect initial evidences – Exercise

15. Using evidence registry – Collect evidence - 1 Organization provides evidences but unsure how they are related to the assessment modelOrganization provides evidences but unsure how they are related to the assessment model Assessors familiar with the ISO15504-5 modelAssessors familiar with the ISO15504-5 model Deleting an evidence record will also remove it from related process outcomes, achievements, and indicatorsDeleting an evidence record will also remove it from related process outcomes, achievements, and indicators

16. Using evidence registry – Collect evidence - 2 Click “Add” button to add a new evidence recordClick “Add” button to add a new evidence record Enter the Name of the evidenceEnter the Name of the evidence Specify the instantiation (optional)Specify the instantiation (optional) Map the evidence to assessment indicator(s)Map the evidence to assessment indicator(s) –If the evidence is mapped to a specific work product, the program will also map the evidence to its generic work product Comment on why/how the evidence is mapped to the assessment indicator(s)Comment on why/how the evidence is mapped to the assessment indicator(s)

17. Using evidence registry – Collect evidence exercise

18. Using evidence registry - Collect evidence by assessment indicators - 1 Organization provides evidences but unsure how they are related to the assessment modelOrganization provides evidences but unsure how they are related to the assessment model Assessors unfamiliar with the ISO15504-5 modelAssessors unfamiliar with the ISO15504-5 model –Use indicator description in the tool to help identify what evidences to look for No need to find evidence for ALL indicatorsNo need to find evidence for ALL indicators Deleting an evidence record will also remove it from related process outcomes, achievements, and indicatorsDeleting an evidence record will also remove it from related process outcomes, achievements, and indicators

19. Using evidence registry – Collect evidence by assessment indicators - 2 Select the assessment indicator of interest, use the indicator description as a guide, look for evidenceSelect the assessment indicator of interest, use the indicator description as a guide, look for evidence Click “Add” button to add a new evidenceClick “Add” button to add a new evidence –The evidence is automatic mapped to the selected indicator Name the new evidenceName the new evidence Specify the instantiation of the evidence (optional)Specify the instantiation of the evidence (optional) Comment on why/how the evidence is mapped to the indicatorComment on why/how the evidence is mapped to the indicator

20. Using evidence registry – Collect evidence by indicators - Exercise

21. Using evidence registry – Evidence detail information Only evidence name is mandatoryOnly evidence name is mandatory Evidence status, document link can also be changed through context menu of the evidence listEvidence status, document link can also be changed through context menu of the evidence list Evidence name and instantiation can be changed on the evidence listEvidence name and instantiation can be changed on the evidence list Label: a short string assigned by the assessment team to identify the documents.Label: a short string assigned by the assessment team to identify the documents. Source/Affirmation: the source/owner of the evidenceSource/Affirmation: the source/owner of the evidence

22. Using evidence registry – Evidence coverage report Show evidence count for process outcomes /attribute achievementsShow evidence count for process outcomes /attribute achievements Red cells highlight the model elements with 0 evidenceRed cells highlight the model elements with 0 evidence User needs to ensure the data gathered is sufficient to cover the assessment scope (ISO15504-2, 4.2.2 b/c)User needs to ensure the data gathered is sufficient to cover the assessment scope (ISO15504-2, 4.2.2 b/c) Select the process outcome /attribute achievement to display all related indicatorsSelect the process outcome /attribute achievement to display all related indicators

23. Process attribute tab – Remove irrelevant evidences mappings - 1 Evidences are automatic mapped to process outcomes/attribute achievementsEvidences are automatic mapped to process outcomes/attribute achievements –The mapping is done by looking up the mapping table defined in ISO15504-5 –No guaranty all mapped evidences are meaningful for the process outcome/attribute achievement Same evidence can be mapped to the same process outcome more than once through different indicatorsSame evidence can be mapped to the same process outcome more than once through different indicators

24. Process attribute tab – Remove irrelevant evidences mappings - 2 Clear the check box to remove an evidence mapping.Clear the check box to remove an evidence mapping. Click ‘Delete Unchecked’ will deleted it from the listClick ‘Delete Unchecked’ will deleted it from the list Click ‘Reset All evidence’ to reset the list back to the initial stateClick ‘Reset All evidence’ to reset the list back to the initial state

25. Process attribute tab – characterize and comment evidences mappings An evidence can be characterized as strength or weaknessAn evidence can be characterized as strength or weakness Comment on how the evidence contribute to the rating of process outcome/attribute achievementComment on how the evidence contribute to the rating of process outcome/attribute achievement –Not the same as the evidence-indicator comment –Evidence-indicator comments are shown at the bottom as read only

26. Transfer evidences between assessment projects Evidences that are out of destination scope are NOT importedEvidences that are out of destination scope are NOT imported Import routine does NOT check for duplication with the existing evidencesImport routine does NOT check for duplication with the existing evidences –Import the database once and ONLY once. –Import the database before generating any mini teams

27. Workshop revision history VerAuthorDateComment 0.1 Fred Liang 19/July/05 Initial version 0.2 Fred Liang 7/Dec/05 Add import list of documents