1. 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services York, UK. Aug. 22 nd – 26th, 2011 Leifang HeLeifang He, Novartis Knowledge Center, New Jersey, USA An Outcome Study on the Learning Programs at the Novartis Knowledge Center

2. 2 Novartis is a global company with more than 100,000 employees in 140 countries. Headquarter is in Basel, Switzerland Novartis offers a wide range of health products through Pharmaceutical, Vaccines and Diagnostics, Sandoz and Consumer Health divisions Introduction (1) - Novartis, AG

3. 3 NKC is the only library & information service hub in the global Novartis community. NKC has 5 physical locations- Basel in Switzerland, New Jersey, Massachusetts, California in USA, and Horsham in United Kingdom Novartis associates can access e-library services 24 x 7 from anywhere in the world. NKC collection consists of 3000+ electronic journals, 4000+ e-books and nearly 200 databases Introduction (2) - Novartis Knowledge Center (NKC)

4. The goal of NKC learning programs enable users to better utilize the e-library for their job performance as learning has such an important role in Knowledge Economy [1] the definition, http://www.businessdictionary.com/definition/knowledge-economy.html [2] Amy C Edmondson, The Competitive Imperative of Learning, Harvard business Review, July-August 2008 4 Business growth in Knowledge Economy Actionable Knowledge for execution Essential competency of best knowledge obtaining Learning contributes to building the competency

5. 5 NKC Orientation for new hires This provides an overview of NKC information resources for different job functions and disciplines. NKC database (DT) targeted learning This is customized for specific customer groups for their specific information needs Two types of NKC learning programs

6. Methodology (1) - The Kirkpatrick's model for evaluation effectiveness Level 1 -- Reactions, which answers to the course perceptions e.g. reactions to the improvement of a learning program Level 2 – Learning, which measures advancing skills, knowledge or attitude Level 3 – Transfer, which measures learners behavior change due to a learning program, the newly acquired knowledge and skills (or attitude) are being used Level 4 – Results, which measures outcomes/business impacts http://coe.sdsu.edu/eet/Articles/k4levels/start.htm 6

7. Methodology (2) – Hypothesis and Key Performance Indicators (KPIs) Hypothesis on after engaging in the NKC learning programs Users come back more often to utilize NKC resources and to search them more efficiently The number of users, who rely on information for decision making, correlates positively with an improvement in their job performance KPIs used to measure against the hypothesis Information Searching efficiency (Level 2.skills, knowledge) Job /business impact (level 4, outcomes) Behavior change (level 3, attitude) Improvement in learning programs (level 1, reactions) 7

8. Methodology (3) - NKC survey questionnaire design Sample core questions reflecting Attitude – Before this session, on average, I used this database After this session, on average, I am likely to use this database Behavior – After this session, on average, I am likely to use this database Outcome – In my job, I rely on information for decision making This session improved my searching skills in this database After this session, on average, I am likely to use this database One to many relations one question with multi answers My overall experience of the session was positive Strongly agree Agree Neutral Disagree Strongly disagree 8

9. 9 507 of 671 associates, who participated the NKC learning programs in 2009, were surveyed via a web based survey Response rate is 70.87% (360/507) The collected data were mapped to the participants business functions, divisions and geographical locations for result analysis Methodology (4) – Survey data

10. 10 40.2% NKC learning programs were delivered via distance learning. It indicates that distance learning is the most utilized way of delivering NKC global learning programs Key findings and discussions (1) - delivery types

11. 11 The learner are from more than 30 locations across 20 countries. Research has the most learners (39%) and followed by Development (17%), Marketing (11%) and Generic (4%) Key findings and discussions (2)- Learner Division distribution

12. 12 From NKC Orientation group, 93% (152/163) responded to Agree (or Strongly Agree) to My overall experience of the session was positive and 6% (10/163) claimed neutral From DT group, 82% (145/176) responded to Agree (or Strongly Agree), and 16% (28/176) claimed Neutral Key findings and discussions (3) - Overall positive experience

13. 13 NKC learning programs improve the participants information seeking efficiency and effectiveness. 90%(147/164)of NKC Orientation participants learned how to find useful information for their job needs. They claimed that they would spend 66%(3.14/4.76) of their information gathering time in using NKV information resources (see the table below) Key findings and discussions (4) - Improve information seeking efficiency and effectiveness 88% (124/146) of DT group indicated that they have improved their information resource searching skills

14. 14 NKC learning programs improve generate business impact and improve job performance 93% (290/311) of the participants claimed that they need information for their job performance. It means that NKC learning programs have an excellent potential base to drive business impact 90% (148/164) of the Orientation participants responded that NKC resources will be useful for their job. And 85% (124/146) of DT participants responded that the NKC recourses presented to them were relevant to their jobs. These imply a strong alignment between NKC information sources to companys objectives DT participants responded that they would be using NKC resources more significantly (P<0.01) after the learning programs (See table below) Key findings and discussions (5) - Generate business impact and job performance

15. 15 On average, DT participants spent significantly more time per week, in the past 12 month on information gathering to the Orientation participants (P<0.001, see table below) This implies that NKC should engage more with DT groups as potentially, they have more information needs Key findings and discussions (6.1) – higher business impact (results) on DT group compared to the Orientation group

16. 16 There is a significantly higher potential business impact from DT learning programs compared to NKC Orientation (p<0.05) as DT group rely on more information for their decision making (see Table below) Key findings and discussions (6.1) – higher business impact (results) on DT group compared to the Orientation group Q. I rely on information for decision making

17. 17 The amount of DT group time spending on NKC resources correlates positively to their job performance (R=0.90). The more NKC provide the learning programs, the more business impact it produces ( See Figure below with Spearman Non-parametric test on Co-relation) Key findings and discussions (6.3) - higher business impact on DT group compared to the Orientation group

18. 18 The study results disclosed that the goal of NKC learning program was achieved and the hypotheses were confirmed by the changes in behavior, attitude and results: NKC learning programs improve the participants information seeking efficiency and effectiveness NKC learning programs generate business impact and improve job performance NKC drive higher business impact in Targeted learning programs as they utilize more NKC resources Conclusions

19. 19 Invest more resources in developing distance learning programs as participants come from more than 30 sites and 20 countries. Develop internal talent to get better understanding of different business functions to design and deliver superior quality DT learning programs. Seek new seek new learning market potential in Emerging Markets to keep pace with Novartis business ventures. Reduce Neutral or Disagree from all categories by reviewing the Comments field and Integrating the users voice into the learning program design Next steps

20. 20 Questions?