1. Audit RFP Evaluation Process
City of Vancouver
Esther Lee
COV Audit RFP two years ago in fall of 2002 (for 2003 financial statements).
Same auditors for a number of years and COV had gone thru significant changes – SAP implementation, new senior staff and PSAB implementation.
As well Council had requested RFP.

2. Evaluation Committee GM of Corporate Services/Director of Finance
Director of Financial Services
Manager of Accounting Services
Director of Financial Planning
Director of Business Support
Director of Corporate Services, VPL
(Director of Corporate Services, VPB)
(Manager of Finance, VPD)
COV has a fairly decentralized structure and Corporate Finance wanted to have input and involvement from the finance staff from various business units. Original committee was to be 8 members but ended up finally with 6 – VPD and VPB could not commit to time required at least 40 hours over a 2.5 month period.
Members of committee brought different perspectives to the evaluation. TC and EL had ultimate responsibility for financial statements. Members were given equal rank.
Objective technical support from Purchasing as well as Legal Services.

3. Timeline Week 1 Aug 26 Issue RFP Week 2 Sept 4
Week Oct 8
Week Oct 21-23
Week Nov 22
Week 16 Dec 10
Issue RFP
Written questions submission
Pre-Submission meeting
RFP Closing
Evaluation
Interview
Final Selection
Council Report
Basically over 6 months from start to finish. Draft of RFP started in early July and then handed over to Purchasing and Legal Services. Ended up with a 50 page RFP –Issued late in August over the Bid BC Website.
All correspondence to be directed thru Purchasing Contract Specialist.
Target was Council report to appoint auditors prior to the 2003 Fiscal year.

4. Evaluation Sections Proponent Qualifications 26%
Technical Requirements 46%
Value Added Services %
Quality of Proposal %
Pricing %
After issuance of RFP – Purchasing headed a number of meetings for the committee to develop an evaluation model based on the requirements stated in the RFP. This took at least 2-3 meetings to get agreement and understanding on not only the sections, weighting scoring, but also the overall process
We ended up with 5 sections
The largest focus was on technical requirements (just under 50% weighting, with similar secondary focus on proponent qualifications and financial offering at around 25%. This reflected that our prime concern was not to get the cheapest audit. We were interested in quality of service and approach to effective and value added audit.

5. Scoring
Scale of 0 to 5
0 Unsatisfactory – failed to meet essential criteria
1 Poor – barely satisfies criteria, significant risk
2 Average – satisfies criteria but further evidence required
3 Good – demonstrated criteria satisfied, understand the requirements
4 Very Good – demonstrated criteria clearly satisfied, substantiated by experience
5 Excellent – clearly satisfy or exceed criteria by proposing superior solution.
So before we were asked to review and evaluate, Purchasing helped set the scoring scale to ensure that there was a level of consistency.

6. Proponent Qualifications
Company profile
Experience
Key personnel
References
This section focused on experience of the firm and in particular the experience of key personnel assigned to the engagement.
Issue here were small number of firms in Vancouver in the municipal audit market. Did not want to limit it just to local expertise – open to national firms with breadth of exposure to large municipal audits.
Try to balance with BC experience.

7. Technical Requirements
Workplan
Proposed segmentation of audit
Level of staff and estimated audit hours
Extent of statistical sampling and use of CAATs
Review of systems and internal controls
Approaches to communication, process reviews, dealing with City’s decentralized structure
Information Technology Auditing

8. Technical Requirements
Development & Support
Coordination of Internal Audit
Staff Education & Succession
Conflict of Interest
Anticipated Audit Issues
City Assistance
Risk Assessment Additional Services
Transition Plan
Vancouver Public Housing Corporation
Separate Board Audits
Technical requirements comprised a major section of the evaluation with main focus on workplan.

9. Other Requirements Value Added Services Quality of Proposal
Compliance with submission instructions
Completeness, detailed solutions explanations, organization, readability and general appearance.

10. Scoring & Evaluation Weighting assigned prior to evaluation
Consensus scoring
Scores averaged
FOI issues
Members evaluated written submissions using evaluation forms – assigned score for each component with scale of 0 to 5 as well as comments.
Submitted to objective Purchasing Contract specialist who compiled all the scores and generated average scores.
Decision was to have a group score – concensus??
To address FOI issues – ie not having individual scores
Note-more time to spend on reviewing final average scores and highlight any anomolies – ie highs or lows and have members explain scoring.

11. Scoring & Evaluation - Pricing
Completed after evaluation of non-financial components (25%)
Analysis of differentials – hours and mix of staff complement – partner vs. field staff, IT specialists
Pricing – scoring was discussed as a separate item – in fact pricing information was removed from the evaluation binders until all the non-financial scores were tabulated – did not want pricing to influence the other factors.
Analysis was done by contract specialist – on differences due to number and mix of audit hours – reflecting different approaches.

12. Scoring & Evaluation - Pricing
Lowest Bid X Total Marks for Price
Bid
If Lowest Bid was 100 and
Total marks for Price was 25.
A bid of 140 would score
(100 X 25)/ 140 = 17.85
The low bid would score 25
Formula COV used - although lowest bid scores the highest, the pricing is allocated a weighting of 25% - therefore it is possible that the low bid may not be the highest scoring proponent – which in fact was what happened in this case.
Want to avoid low ball bids, followed by nickel and diming billing or surprise extra billings.

13. Interview Process Format and time limit (2 hrs)
Maximum delegation of 4 – including engagement partner/ principal and manager
Not a presentation
Specific questions from proposal – prepared questions and spontaneous discussion
After initial scoring – in our case only 3 proponents all fairly closely rated –
Interviews to provide an opportunity to answer more specific questions from the proposal-
To meet the engagement partner and manager – some flew in from other cities.
Forwarded in advance 12 detailed questions based on review of proposals to be answered in writing.
Unprepared questions as well.

14. References Current and past clients List of specific questions
Probing areas of concern or further clarification
Conducted by a subcommittee of the evaluation team.
Important to probe.

15. RFP Results
Five year contract with option to renew for additional year (to coincide with Council terms) commencing in 2003
Firm price indexed to CPI – clearer parameters with respect to extra billings.
Better understanding of service expectations – service level agreement.

16. Important Considerations
Clear definition of criteria – tied to RFP
Evaluation model and easy to use forms
Structured interview process
Time commitment and knowledge base of evaluation team members