Lessons Learned: Two-Step Verification HUIT Senior Leadership Team | January 5, 2017 Smith Campus Center

Agenda What did we do? What were the goals? How did it go? Ground rules What did we learn about the project? What would we do next time? Other thoughts/suggestions

What did we do? We invited all of HUIT and anyone in the Schools who was part of the two-step verification team to get together to talk about the lessons we could learn from the project The session was held December 20, 2016 for 80 minutes at Piper Auditorium at Gund Hall About 150 people responded affirmatively to the calendar invite 52 people participated In addition to notes, we tried capturing video for those unable to attend Christian Hamer (Project Sponsor) moderated the session and Dennis Ravenelle (PM) took notes

What were the goals? Learn what we could from anyone with any kind of input about how we can make projects like this better Be inclusive and involve anyone who was interested in either participating or just watching Model that it’s OK (even good!) to make mistakes if we learn from them Have a forum where it really was OK to say anything, as long as it was offered in a constructive spirit

How did it go? The reaction we’ve heard so far was positive We got a lot of good feedback on the project and heard some perspectives we hadn’t heard or considered To try to organize the conversation, we broke it into seven sections: Planning Internal communications External communications Rollout Support Project management/coordination Coordination with the Schools And tried to focus on three central questions: What worked well? What could we have done better? What would we change if we had it to do all over again?

Ground rules Logistics: No one takes anything that is said personally We explained that we were experimenting with video (so that people would know it was being recorded) and what that meant in terms of microphone and repeating questions The space: because we planned for the 150 responses, the space was not ideal, but we reinforced the point that we wanted this to be interactive and collaborative No one takes anything that is said personally The project went very well, but we can always learn – so people who worked on the project should not take any feedback critically NOTHING is off limits as long as it is offered in the spirit of learning

What did we learn about the project? Did well Could do better Tool/technology is easy to use Early testing and feedback, especially on the user interface Strong support from University leadership and strong community acceptance Good project coordination, information sharing and reporting, aided by Office 365 Good training for help desk and field support, with feedback from early adopters to inform FAQs Focus on first-call resolution Great documentation and website Adapted rollout to meet audience needs “Ear to the ground” from field support Personal appearances Better support for edge but important cases (international phone numbers, replaced phone) Address UI/usability issues sooner More clear communications (internally and externally) about what resources would be impacted Provide less information in reports to Schools Better internal communications when unforeseen issues cause schedule changes Ensure that support teams have screenshots of all possible paths for all user types Make sure key processes (list generation) are documented and not reliant on individuals Create a central/shared glossary sooner Consistent message from the start on hardware tokens Tell the community “why” from the start Earlier focus on how to use Duo while traveling

What would we do next time? Different space: We didn’t need a space that large and could have used Lamont, MWD or other room that could hold 50-60 in a setup more suited to collaboration While we didn’t want to turn people away, the space made it more challenging Repeat the feedback: We did this as a practical matter because we were recording and wanted as few mics as possible (one) – the feedback was that it really helped the audience feel like we understood what was being said Don’t justify or respond to constructive criticism: That was the most difficult thing for me to (not) do, but really important Have some ”seed” feedback: I had some notes of my own that I thought it was important to share (mainly feedback on things I would do differently) and I also had a nice list from someone who was unable to attend but sent feedback in advance This seemed to help “prime the pump” for the audience and, because not all of it was mine, did so in an authentic way Make it clear from the start that ANYTHING (constructive) is welcomed, and deliver on that by not justifying when feedback is offered

What would we do next time? Video? We did it, and we will publish it for those who couldn’t make it, but it meant that we had to use a mic (no speakers though), which made it seem “big” (or, perhaps more clearly, the opposite of “intimate”) and may or may not have limited feedback Who moderates? I thought it was really important for me to do this – but there may be a way to achieve the goals we set out using a moderator who was not at all involved with the project, but trained in conducting these sessions

Other thoughts/suggestions Consider sending a survey in advance to help identify themes Consider using clickers to help understand how strongly a theme is echoed/felt Consider asking the same set of questions across projects so that we can do some evaluation across projects over time THE REAL TEST IS GOING TO BE DOING THIS, AS GRACEFULLY AS WE CAN, FOR A PROJECT THAT FAILED!