1. Digital Services Contracting Professional Development MVP Program Release 4 Classroom Session – Day 3
January 2017

2. Day 3 Agenda LUNCH Morning Course Assessment Program Feedback
DHS FLASH Team Panel
LUNCH
Afternoon
USDS Guest Speakers
GRADUATION!
Happy USDS (Jackson Place)
Duration: 5 minutes
Timing: 8:00-8:05 am
Presenter: ICF/Peter Bonner
Facilitator Notes: Review the agenda shown on the slide.

3. Course Assessment Duration: 1.5 hours Timing: 8:05 – 9:30
Presented by: ICF/Peter Bonner, ASI Will Randolph

4. Release Assessment Results
Cohort scores on the assessments improved with each Release
Highest post- score overall was achieved for Release 4
Note that this only included Release 4A
Overall, highest scores were received on the following objectives:
Objective 3.3: Identify the effective characteristics of a change agent and strategies to apply at different phases of the change lifecycle as you return to your agency.
Objective 4.2: Implement evaluation methods and criteria to evaluate vendor maturity and ability to deliver a product that solves a given need.
Release assessments were administered both before (pre-) and after (post-) learners interacted with the content – classroom sessions and assigned activities.
Data on the slide focuses on post- scores, so those levels attained after learners experienced the content. (Pre-scores were designed to be informational both for learners and content team.)
Best/Worst Performance Objectives (Cohort-wide) for reference:
Release 1
Participants performed best on Performance Objective 1.4: Identify “who’s who” in the digital services arena
Participants performed poorest on Performance Objective 1.7: Identify the high-level principles of agile development and Performance Objective 1.8: Describe what sets agile methods apart
Release 2
Participants performed best on Performance Objective 2.5 – Identify why communicating openly and responsibly with potential vendors is critical to digital services acquisition success and how to do it, followed by Performance Objective 2.2 – Assess your agency’s readiness for change and innovation.
Participants performed poorest on Performance Objective 2.7 – Conduct effective market research for digital services, followed by Performance Objective 1 – Analyze stakeholders in your sphere of influence that will impact digital services acquisition.
Release 3
Participants performed best on Performance Objective 3.3 – Identify the effective characteristics of a change agent and strategies to apply at different phases of the change lifecycle as you return to your agency.
Participants performed poorest on Performance Objective 3.7 – Define evaluation criteria, given evaluation strategy discussed in your acquisition strategy.
Release 4
Participants performed best on Performance Objective 4.9 – Determine how to execute an exit strategy and course correct.
Participants performed poorest on Performance Objective 4.7 – Identify software engineering practices for high-quality digital services like version control, continuous integration, and continuous delivery.

5. Capstone Skills Test Results
Overall, partipcants did well on the Capstone Skills Test
Average scores for 21 of the 29 performance objectives was 70% or higher
Performance was highest for Release 2 content and lowest for Release 1
Most participants showed increased scores for Releases 2, 3, and 4 from the Pre-Assessment
Scores on the Capstone Skills Test ranged from 53% to 93%, with an average score of 79%.
Score ranges for each of the Releases were as follows:
Release 1: 29% to 100%
Release 2: 40% to 97%
Release 3: 41% to 100%
Release 4: 38% to 100%
Release 1 is the only Release for which a large number of partipcants saw a decrease in their scores from the Pre-Assessment to the Capstone Skills test. There could be multiple different reasons for this. For example, having a different number of items to assess objectives from the Pre-Assessment to the Capstone Skills Test can have a large impact because there are few items to measure each objective. Alternatively, it is possible that because Release 1 was the furthest removed timewise from the Capstone Test, that participants did not recall information as well or did not retain as much information because they were still learning about how the course was structured in the early days.
Release 1
Release 2
Release 3
Release 4
Score Increase
35%
81%
65%
77%
Score Decrease
19%
23%

6. Live Digital Assignment (LDA) Cohort Results
Teams That Earned Buy-In from All “Investors”
Fragile Development
Scrums-n-Roses
WebExers
CONGRATS!
Great Job on the LDA! The average score was 88%. The score is an equally weighted combination of ratings:
Engaged Contribution
Collaborative Action
Investor Buy-in
The Pitch
Engaged Contribution
22 people received all “High” marks from team members
Collaborative Action
23 people received all “High” marks from team members
The
Pitch
24 people were rated by at least 1 investor as having a “great” pitch
Overall scores on the LDA ranged from 63% to 100%. Over half of the cohort partipcants received a score of over 90% on the LDA; 11% received below a 70%.
The highest scores were obtained for the Engaged Contribution and Collaborative Action dimensions (self-ratings and peer ratings). Of the ratings performed by observers, scores for The Pitch Itself were slightly higher than scores for the Investment Buy-In.

7. Bronze Badges
Congratulations to each of our bronze badge winners!

8. Silver Badges
Congratulations to both of our silver badge winners!

9. Gold Badges
Congratulations to each of our gold badge winners!

10. Applied Skills Badge Leaderboard
Applied Skills Badges
Applied Skills Badge Leaderboard
Name
Applied Skill Badge
Developing Your Network (Level III)
Study, Learning, and Exploration (Level II)
Getting Your Name on the Map (x2)
Study, Learning, and Exploration
Getting Your Name on the Map
Study, Learning and Exploration
Congratulations to everyone who has earned an applied skills badge! (name removed) is our overall leader– after conducting 12 Stakeholder Interviews for her Stakeholder analysis, she has reached the Level III Developing Your Network badge!

11. Wrap Up & Classroom Session Feedback
Duration: 1 hour, 10 minutes
Timing: 8:50– 10:00
Presented by: ICF/Peter Bonner, ASI Will Randolph
Facilitator Notes:
Now that you’ve had time to see all of the presentations, we wanted to come back together to debrief on the past three days and gather your feedback on your classroom experience. This will be similar to the orientation session, we’ll be closing with the “liked, learned, and longed for” post it activity.

12. The Live Digital Assignment
Liked – things you really liked Learned – things you have learned Lacked – things you have seen the team doing, but consider that could be done better. Longed for – something you desired or wished for
Duration: 20 minutes
Timing: 8:50– 9:10
Presented by: ICF/Peter Bonner, ASI Will Randolph
Facilitator Notes:
Conduct this as a large group discussion to generally gather participants’ feedback on the Live Digital Assignment. Walk through each of the categories on the slide and ask participants how helpful the LDA has been in them applying their new skills to various activities. Record participant responses on the whiteboard. (We will also have a note taker in the room to capture notes.)
Ask them:
Liked – things you really liked
Learned – things you have learned
Lacked – things you have seen the team doing, but consider that could be done better.
Longed for – something you desired or wished for

13. Release 4/Capstone Classroom Session
Liked – things you really liked
Learned – things you have learned
Lacked – things you have seen the team doing, but consider that could be done better.
Longed for – something you desired or wished for
Duration: 15 minutes
Timing: 9:10– 9:25
Presented by: ICF/Peter Bonner, ASI Will Randolph
Facilitator Notes:
Prior to this activity, create a space on the wall for participants to place their sticky notes within each category for the Release 4/Capstone Classroom Session.
We’re going to take a little time now and think back through everything we did in this classroom session. As we do so, we’d like you to use the sticky notes in front of you to write down:
Things that you really liked about an activity
What activity might have taught you something really meaningful and what that is
If there were any activities that could have been done better
If there was some activity that you feel should have been done.
You do not have to write a sticky note for each activity in this classroom session, but the ones that stood out. Take 5 minutes to write down your feedback, then place your sticky notes on the wall in the appropriate categories, as we’ve done in past classroom training sessions.
Your feedback will be helpful for the next classroom session and future trainings, as well as a reminder to yourself of what you have learned and experienced this week.
After 5 minutes have passed, bring the participants back together and ask for volunteers to highlight feedback they added to each category. Spend 10 minutes discussing this feedback before moving on.

14. A Retrospective of Each Release & the Program
Release 1 – Digital Services in 21st Century Government
Understanding what digital services are, how digital services are designed and delivered, and the current role of a government digital services professional
A scan of the digital services market – potential sources of supply and how to communicate with them pre-solicitation
Release 2 – Understanding What You’re Buying
Exploring the need or outcome you’re trying to achieve with what you’re buying
Identifying and partnering with key stakeholders who will impact your acquisition
Conducting market research and preparing to develop your acquisition strategy
Release 3 – How Do You Buy?
Utilizing techniques such as the lean canvas for acquisition planning
Analyzing evaluation criteria and the various elements of a procurement package
Identifying your organization’s readiness for agile and the role that plays in your acquisition strategy
Duration: 35 minutes
Timing: 9:25– 10:00
Presented by: ICF/Peter Bonner, ASI Will Randolph
Facilitator Notes:
Now we’re going to zoom out even further and reflect on the Digital Acquisition Program as a whole. You’ve provided feedback during the program at each classroom session and in the assessments; thank you for that. Now is your opportunity to provide any overall and final feedback to help us better document how the program should be refined for future cohorts. We’ve listed the releases on the slide for your reference.
This is going to be the jar/hat game. Have two jars, one labeled “My Favorite Part of the Program” and one labeled “What I Did not like”
Pass the jars around and have them jot done their responses on slips of paper and place in the jar. This is non attribution, so no names.
Take 10 minutes to document your feedback & then we’ll come back together to debrief as a class.
Release 4 – Awarding & Administering Digital Service Contracts
Awarding digital service contracts by assessing the readiness of the technical evaluation team and implementing evaluation methods and criteria to evaluate vendor maturity
Managing digital services delivery
Identifying when failure occurs and what to do when it does

15. Morning Break Duration: 15 minutes Timing: 10:00-10:15
Presented by: ICF
Facilitator Notes:
Instruct participants to take a 15 minute break.
Direct to restrooms and water cooler.

16. Guest Speaker – DHS FLASH Team Panel
Duration: 1 hour 45 minutes
Timing: 10:15-12:00
Presented by: ICF/Peter Bonner & FLASH Team Panel
Facilitator Notes:
Introduce the DHS FLASH team panel. Explain that the DHS team recently awarded spots on its agency-wide agile software acquisition vehicle, which is called the Flexible Agile Support for the Homeland (or FLASH) vehicle. They’re here to talk more about the innovative procurement methods they used on this acquisition and some of their lessons learned along the way.

17. Lunch
The lunch break will run from 12:00-1:00pm.

18. Guest Speakers - USDS Duration: 1 hour Timing: 1:00-2:00
Presented by: Haley/Matt Cutts/Todd “big picture” Discussion
Facilitator Notes:

19. Graduation! Duration: 2.5 hours
Timing: 2:30-3:00 (travel); 3:00-5:00 (ceremony + reception)
Presented by: ICF/Peter Bonner
Facilitator Notes:
Tell participants that they’ll have from 2:30-3:00 pm to travel over to the Indian Treaty Room in the East Wing of the Eisenhower Executive Office Building. We’re building in extra time to get through security. We’ll start the graduation ceremony at 3:00 pm. See everyone there!
Winning team gives pitch
Read names – provides names to Traci of graduating folks as well as badging winners – slide that lists out these folks – e.g., top 5
Provide data from pre-program and post-program survey results – include on a slide (similar to what we did for pilot graduation ceremony)
Reception