1. MENU MALAYSIA MOOCs PILOT – AN EVALUATION Pengenalan kepada PutraMOOC
Kesenian Melayu:
Seni Pertukangan –
Keris Melayu 1 2 3
MALAYSIA MOOCs PILOT –
AN EVALUATION
Green Screen
Technology
Assoc. Prof. Dr. Alyani Ismail
Deputy Director (Innovation in T&L)
Centre For Academic Development, UPM
TITAS Malaysia
MOOCs 1 2 3
Flipped Learning
Malaysia MOOCs –
Walktrough

2. CONTENTS OF PRESENTATION
PART 1 - Malaysia MOOCs and PutraMOOC
PART 2 - Research findings: Evaluation of Malaysia Pilot MOOC (A research project sponsored by MOHE)

3. PART 1

4. Overview of MOOC Development and Implementation
Global Community
Open Education Initiatives
Step by step, systematic & holistic
Why MOOC?

5. SHIFT #9: Globalized Online Learning
of Malaysian Education Blueprint:
Higher Education
Focus
Enhancing the quality of teaching
and learning; bringing niche area to
Global arena; visibility, positioning
and branding; as well as lifelong
learning.

6. MALAYSIA MOOCs Framework
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

7. PutraMOOC An Initiative Video Lectures Interactive Chats Forums
e-assessments
Quizzes
Other learning
activities

8. PutraMOOC The Objectives Main Objectives of
To enrich learning experiences of on-campus students
To showcase UPM’s expertise and niche areas
Main Objectives of
To Drive Innovative Teaching Methods

9. PutraMOOC How it works? leveraging on the use of
blended learning through
flipped classroom for
on-campus students
innovative expansion of
knowledge reflecting
Rhizomatic model of
learning

10. PutraMOOC The Setup Video Production Video Storage Chroma key
embeded
Chroma key
In studio
upload
Integrated
with various
of media
Why?
Prevent problem of
network congestion
Video of
lectures

11. 46,151 students PutraMOOC & Malaysia MOOC OpenLearning
- a platform chosen for Malaysia MOOC
Soft launch:
18 Sept 2014
4 pioneer courses: TITAS, Hubungan Etnik,
Entrepreneurship, IT Literacy. UPM is chosen to develop TITAS
Students of these 4 compulsory courses for all 20 public
universities must register for the MOOC courses.
Massiveness of enrolment
Current enrolment
of TITAS (developed
by PutraMOOC team,
UPM)
46,151 students
As of 14 April 2016

12. SNAPSHOT OF MALAYSIA MOOCS PAGE IN www.openlearning.com

13. SNAPSHOT OF PutraMOOC PAGE IN www.openlearning.com/putramooc

14. SNAPSHOT OF TITAS LEARNING MATERIALS IN www.openlearning.com

15. SNAPSHOT OF TITAS’S MOOC COURSEWARE

16. SHARING OF LEARNING MATERIALS
DELIVERING THE MOOC FOR TITAS – STUDENTS RESPONSE
SHARING OF LEARNING MATERIALS

17. DISCUSSIONS ON REAL CASE SCENARIO & CURRENT ISSUES
DELIVERING THE MOOC FOR TITAS – STUDENTS RESPONSE
DISCUSSIONS ON REAL CASE SCENARIO & CURRENT ISSUES

18. COMMENTS BEYOND CONTENTS
DELIVERING THE MOOC FOR TITAS – STUDENTS RESPONSE
COMMENTS BEYOND CONTENTS

19. SHARING OF ACTIVITIES ON PROJECTS
DELIVERING THE MOOC FOR TITAS – STUDENTS RESPONSE
SHARING OF ACTIVITIES ON PROJECTS

20. OpenLearning Analytics: TITAS (Sept 2014 – Sept 2015)
Hong Kong 50
United States 46
United Kingdom 28
Singapore 19
Japan 15
Anonymous Proxy 8
Netherlands 5
Australia 4
Indonesia
China 3
Thailand
Turkey 2
Korea
Saudi Arabia
Jordan
Qatar 1
Bangladesh
Canada
Brunei Darussalam
Germany
Morocco
Sweden
Pakistan
Egypt
United Arab Emirates
Kazakhstan
India
Azerbaijan
INTERNATIONAL STUDENTS
STUDENTS FROM MALAYSIA
201
(0.7%)
28 Countries
25,872 (87.21%)
UNKNOWN
Even though TITAS is delivered fully in Bahasa Malaysia as a compulsory subject for UAs, it still attracts a number of international students
3587 (12.09%)

21. OpenLearning Analytics: TITAS (Sept 2015 – Apr 2016)
Bil
Nama Negara
Jumlah Pelajar Mendaftar 1
United Kingdom 68 2
United States 37 3
Japan 12 4
Netherlands 7 5
China 6
Australia
Pakistan 8
Italy 9
Indonesia 10
Germany 11
Cambodia
Afrika Selatan 13
India 14
Arab Saudi 15
Taiwan
Pelajar dari Malaysia = 17,864
Pelajar Antarabangsa = 252

22. OpenLearning Analytics: AGRICULTURE AND MAN (Sept 2015 – Apr 2016)
Bil
Nama Negara
Jumlah Pelajar Mendaftar 1
Iceland 2
United States 3
Peru 4
Argentina 5
Afrika Selatan 6
Uganda 7
Tunisia 8
Pakistan 9
India 10
Filipina 11
China 12
Unknown
Pelajar dari Malaysia = 173
Pelajar Antarabangsa = 13

23. OpenLearning Analytics: KESENIAN MELAYU (Sept 2015 – Apr 2016)
Bil
Nama Negara
Jumlah Pelajar Mendaftar 1
Jerman 2
United States 3
Pakistan 4
Unknown 6
Pelajar dari Malaysia = 240
Pelajar Antarabangsa = 10

24. OpenLearning Analytics: Malaysian Food Heritage (Sept 2015 – Apr 2016)
Bil
Nama Negara
Jumlah Pelajar Mendaftar 1
Perancis 2
United States 3
Denmark 4
Pakistan 5
Greece 6
Australia 7
Unknown
Pelajar dari Malaysia = 82
Pelajar Antarabangsa = 9

25. PART 2
RESEARCH FINDINGS: EVALUATION OF MALAYSIA PILOT MOOC A RESEARCH FUNDED BY MOHE

26. Introduction
Started in November 2013
TITAS, Hubungan Etnik, Introduction to Entrepreneurship and ICT Competency courses in September 2014
Aims - to evaluate the various aspects of the four pilot courses which hosted in OpenLearning platform.
The aspects covered are infrastructure and info structure, curriculum, pedagogy, content, professional development, enculturation and enhancement in teaching and learning.
Evaluation of students and lecturers (survey) and MOOCs developers (interview).

27. THE TEAM Dr. Habibah Ab. Jalil (UPM) – Lead Researcher
Prof. Madya Dr. Alyani Ismail (UPM)
Prof. Dr. Sidek Ab Aziz (UPM)
Prof. Dr. Hanafi Atan (USM)
Prof Dr Abd. Karim Alias (USM)
Prof. Dr. Mohamed Noor Hasan (UTM)
Tn. Hj. Mohd Nazri Md. Saad (UPSI)
Dr. Hjh. Norasiken Bakar (UTeM)
Prof. Dr. Norazah Nordin (UKM)
Prof. Dr. Nor Aziah Alias (UiTM)
Prof. Dr. Hong Kian Sam (UNIMAS)
En. Mohd Hafriz Nural Azhan (UMT)
Puan Salfarah Abdullah (UMS)
En. Zaid Ali Alsagoff (IMU)
NABIHAH BT MOHD ZIN (UNIMAS)

28. Population
First year students from Malaysia HEI that registered for four pilot courses
Lecturers that teach the four pilot courses
MOOC admins/developers
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

29. Findings (Part 1)
More than 70% respondents registered TITAS and Hubungan Etnik.
Only a few universities offered ICT Competency and Introduction to Entrepreneurship.
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

30. © FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

31. More than 90% of the respondents experienced learning through MOOC for the first time.
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

32. About 50% of respondents accessed at least once a week
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

33. Almost 50% of the respondents accessing MOOC in the evening
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

34. Almost 80% of the respondents used Wi-Fi campus to access MOOC
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

35. More than 75% of the respondents accessing MOOC from their hostels
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

36. Infra & Info Items Mean 1. I can access MOOC during class time. 3.39
2. I can access MOOC at other times within the campus.
3.97
3. Internet speed during class time is bearable.
3.34
4. Internet speed at other times is bearable.
3.43
5. Streaming video can be accessed during class time.
2.96
6. Streaming video can be accessed at other times within the
campus.
3.55
7. Video are downloadable during class time.
2.90
8. Video are downloadable at other times within the campus.
3.48
Overall mean
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

37. Curriculum Overall mean 3.83 Items Mean
The content of the course meet the requirement of the syllabus.
3.84
The balance between the learning activity and content is sufficient to help my learning.
3.81
3. The learning schedule (course plan/lesson plan) is easy to follow.
3.80
The learning activities help me to better understand the content.
3.88
Overall mean
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

38. Introduction to Entrepreneurship
Learning Design (1)
Item
Course
Mean
1. The course is well designed
TITAS
3.97
Hubungan Etnik
4.00
Introduction to Entrepreneurship
3.92
ICT Competency
3.89
Learning through MOOC meets my
learning needs.
3.80
3.84
3.77
3.81
The sequence of learning activities help my understanding of the subject matter.
3.82
3.86
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

39. Introduction to Entrepreneurship
Learning Design (2)
4. The learning schedule (course plan/lesson
plan) suits my learning pace.
TITAS
3.83
Hubungan Etnik
3.86
Introduction to Entrepreneurship
3.76
ICT Competency
3.79
5. Opportunity to interact with large number of
students is beneficial to my learning. The
quizzes enhanced my understanding of the
course
3.84
3.88
3.82
The quizzes enhanced my understanding f
the topics covered.
3.94
3.95
3.85
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

40. Introduction to Entrepreneurship
Learning Design (3)
Additional resources
reinforced my understanding.
TITAS
4.02
Hubungan Etnik
4.01
Introduction to Entrepreneurship
3.93
ICT Competency
3.91
Assignments given helped me
to achieve the learning
objectives.
3.96
3.97
3.90
I am able to accomplish the
activities on my own.
3.82
3.86
3.74
3.78
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

41. Introduction to Entrepreneurship
Learning Design (4)
I am able to follow the course
at my own pace.
TITAS
3.86
Hubungan Etnik
3.87
Introduction to Entrepreneurship
3.80
ICT Competency
3.79
Overall mean
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

42. Introduction to Entrepreneurship
Content (1)
Item
Course
Mean
1. The content covers the
essential aspects of the course.
TITAS
3.90
Hubungan Etnik
3.91
Introduction to Entrepreneurship
3.85
ICT Competency
3.86
2. The content is clear and
understandable
3.96
3.97
3.89
3.87
The content is well organized.
3.93
3.88
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

43. Introduction to Entrepreneurship
Content (2)
The use of multimedia in the course is interesting.
TITAS
3.97
Hubungan Etnik
3.99
Introduction to Entrepreneurship
3.88
ICT Competency
3.95
The course activities are
useful for my learning.
3.93
3.96
3.91
The content met the learning
outcomes.
3.92
3.87
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

44. Content (3) Overall mean 3.95 TITAS 3.96
Overall, the quality of content is satisfactory.
TITAS
3.96
Hubungan Etnik
3.98
Introduction to Entrepreneurship
3.90
ICT Competency
The following activities are
useful for my learning:
Watching Video
4.05
Forum
3.67
Assignment
3.89
Quiz
Group Activity
3.91
Interactive Presentation
Self-learning
3.99
9. The additional resources are helpful (eg: link to websites,
readings and social media outside OpenLearning).
4.08
Overall mean
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

45. Enhancement in T & L (1) Item Mean
1. MOOC enhances my learning experiences.
3.86
2. Learn more effectively using MOOC.
3.73
3. Learning via MOOC is enjoyable.
3.78
4. The Learning via MOOC help me
to:
Remember (facts)
3.75
Understand (concepts, principles, processes)
3.89
Apply (what I have learned)
3.76
Analyze (situation)
Evaluate (issues)
Create (ideas, solutions, innovation, problem solving)
3.79
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

46. Enhancement in T & L (2)
Learning via MOOC enhances the following skills:
ICT
3.88
Problem Solving
3.81
Critical Thinking
3.82
Communication
3.76
Entrepreneurship
3.66
Writing
3.72
Collaboration (working together)
3.79
Leadership
3.65
6. Learning via MOOC enhances the following values:
Respectfulness
3.87
Caring
3.83
Honesty
3.84
Generosity
3.86
Helpfulness
3.95
Ethical
3.92
Overall mean
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM

47. Findings (Part 2) Population & Sample Analytical Approach Instrument
Population: 1532 lecturers that teach the four pilot courses
A 63-items online survey questionnaire using Survey Monkey for lecturers
Descriptive analysis
(mean and standard deviation)
Sample: 164 lecturers participated in online survey
Interview session for MOOC developers
Data transcription
MOOC developers (UPM, UKM, UiTM, UNIMAS)

48. Findings (Quantitative Part)
Objective i: Identifying lecturers MOOC usage profile
50% are male and 50% are female
More than 80% are Malay respondents

49. Population, sample and % of lecturers respond by university
Respond by lecturers
% by university
(Sample) UM 16 2
12.5
UKM 26 9
34.6
UPM 48
33.3
UTM 12 11
91.6
USM 6
66.7
UiTM
1178 20
1.7
UPSI 22
50.0
UTeM 5
40.0
UMT 4
36.4
UNIMAS 39 15
38.5
UNISZA 8
62.5
UMP 23
34.8
UNIMAP 7
71.4
UUM 37
59.5
USIM 10
60.0
UIA 1
100.0
UMK 58
UPNM
57.1
UMS 19
36.8
UTHM 17 3
17.6
TOTAL
1507
161

50. 121 of respondents are MOOCs learning support.
66 of respondents are teaching Hubungan Etnik course.

51. More than 40% of the respondents are competent in e-Content development, online learning task and interactive presentation

52. More than 70% of the respondents experienced teaching through MOOC
for the first time.
More than 60% of respondents said ‘Yes’ for support provided in MOOCs.

53. More than 70% of the respondents had support for delivery, 70% for content while more than 50% for pedagogy and design.
More than 80% of respondents expect support for pedagogy, delivery and content.

54. More than 80% of the respondents get support from university while more than 60% get support from MOOC team and OpenLearning.

55. Objective ii: Identifying lecturers’ perception about infrastructure and info structure quality provided for MOOC usage.
Items SD
(%) D
NDNA A SA
Mean
I can access the OpenLearning anywhere in the campus.
2.4
9.1
11.6
57.3
19.5
3.82
0.93
I can access the OpenLearning anytime.
0.6
12.2
15.2
55.5
16.5
3.75
0.90
I have sufficient equipment to use MOOCs for teaching.
3.7
14.0
21.3
53.7
7.3
3.47
0.95
I can access OpenLearning from any device.
14.6
29.3
47.0
8.5
3.48
0.87
I found OpenLearning very difficult to use.
1.2
13.4
31.7
45.7
7.9
3.46
I definitely utilize OpenLearning for other courses.
37.2
41.5
5.5
3.35
0.84
Overall mean
3.56

56. I feel demotivated due to extra workload because of MOOC.
6.1
27.4
36.6
26.8
3.0
2.93
0.95
MOOC gives me opportunity for knowledge sharing. .6
21.3
62.2
12.8
3.84
0.70
I believe my teaching has improved due to MOOC.
1.8
28.7
56.1
7.3
3.61
0.79
I believe my students’ learning has improved due to MOOC.
2.4
5.5
29.3
6.7
3.59
0.80
Overall mean
3.57

57. Course Completion Rate

58. INTRODUCTION TO ENTREPRENEURSHIP
Completion Summary
10% Completion
50% Completion
75% Completion
100% Completion
TITAS
2.20%
48.12%
36.17%
24.39%
HUBUNGAN ETNIK
4.97%
42.83%
30.70%
17.22%
INTRODUCTION TO ENTREPRENEURSHIP
3.54%
46.21%
38.62%
9.53%
ICT COMPETENCY
3.74%
48.91%
38.95%
14.38%

59. Completion Rate by University (TITAS)

60. Completion Rate by University (Hubungan Etnik)

61. Completion Rate by University (Introduction to Entrepreneurship)

62. Completion Rate by University (ICT Competency)

63. Comparing Two Groups Total 55596 272728 4.906 19953 119183 5.973
Course
Semester
Students
Comments
Comments Per Student
Kesepaduan & Hubungan Etnik
Semester 1
17689
83324
4.71
Semester 2
8808
40677
4.618
ICT Competency
5634
45724
8.116
1609
9412
5.85
TITAS
22017
110710
5.028
5757
60843
10.569
Introduction to Entrepreneurship
10256
32970
3.215
3779
8251
2.183
Total
55596
272728
4.906
19953
119183
5.973

64. Time Spent per Hour on The Course Over The Week
TITAS
Hubungan Etnik
ICT Competency
Introduction to Entrepreneurship

65. Conclusion
Our principal aim in this paper has been to present some preliminary findings on the Pilot Malaysia MOOC initiative which recently deployed.
These findings show that selection and utilization of MOOC courses among universities students would be much due to the fact that the courses offered are compulsory rather than it is taken for lifelong learning.
It is good to know that about half of them will access MOOC every week, which indicates a good return of investment of MOOC deployment.
Every university need to put their best effort on the infrastructure within campus area especially on their network connection as students were heavily depending on the universities’ facilities.  
The overall means for input and output variables indicate that the MOOC initiative is implemented as planned. Students’ positive insights on MOOC curriculum, design, content quality and the impact on the teaching and their learning shows that this initiative is adequate and should be possible to become a new mechanism of teaching and learning in these era for the higher education context.
There should be mechanism to reward lecturers effort in developing and using MOOC in their T&L
© FEB 2015 HABIBAH AB JALIL, SIDEK AB AZIZ & ALYANI ISMAIL UPM