1. NMLS Testing & Education
SAFE Mortgage Loan Originator Test Webinar
September 10th, 2009
NMLS Testing & Education

2. Your Hosts Anne Altman – Manager, Mortgage Testing Programs
Pete Marks – VP, National Mortgage Testing & Education Programs
Rich Madison – Director, Mortgage Education Programs

3. Conference Call Logistics
*6 to mute and un-mute your call
Please do not put your phone on hold
Please identify yourself and your organization when asking a question

4. Agenda SAFE MLO Test: Background Info & Test Specifications
Pieces of the Test Development Puzzle
1. Subject Matter Experts
2. Content Outlines
3. Test Questions
4. Review Committees
5. Test Forms
Test Administration
Preparing to Take the Test
The Test Center
Computer Test Operations and Functionality
Other Testing Information
Q & A

5. SAFE MLO Test: Background and Overview

6. Background Information
SAFE Act requires NMLS to develop tests for MLO’s
NMLS contracted FINRA and its subcontractor, Pearson VUE, to develop the SAFE MLO tests.
NMLS contracted FINRA to administer the SAFE MLO tests using its current dual-vendor channels - Pearson VUE and Prometric’s high-stakes testing facilities.
SAFE Mortgage Loan Originator Test
National Component
State-Specific Test Component

7. National Test Component Basics
All mortgage loan originators must take and pass the national test component
Test questions = 100
90 scored
10 unscored/pretest
Passing score = 75%
National Test Component launched July 30th

8. State Test Components Basics
Test questions = (varies; determined by state)
45-55 scored
10 unscored/pretest
Passing score = 75%
10 state tests currently available
AZ, IA, ID, LA, NH, NC, PA, RI, VT, WA
More state tests to be released shortly

9. SAFE Deadlines for Testing
Both National and State Components
All new licensees: July 31, 2010
All existing licensees: December 31, 2010
Several states have elected to have earlier deadlines
* Please check with each particular state’s regulatory authority

10. Pieces of the
Test Development
Puzzle

11. Subject Matter Experts (SME’s)
Part 1.
Subject Matter Experts (SME’s)

12. SME’s are: selected by each states’ regulators and approved by NMLS
well versed in laws and regulations
a mix of state regulators and industry professionals
able to devote their time, effort, and expertise to the development of the SAFE test

13. “What do SME’s really do?”

14. SME’s: serve on one or both of the following test
development committees:
National Mortgage Review Committee
State Test Development Committee

15. SAFE National and State Test Committees
Reviewed and provided input on national or specific state test component content outlines
Examined, revised, and approved all test questions on national and state test components
Will participate in future content outline and test question maintenance cycles

16. Part 2.
Content Outlines

17. “What are Content Outlines?”

18. Here’s what they look like…

19. Here’s what they look like…

20. Content outlines are: The basic structure or blue print for the test
The roadmap for constructing test questions
A tool to help candidates prepare for the test

21. “Tell me more about content outlines…”

22. Content Outlines Each test component has its own outline
national component
all state specific components
Outlines map out:
specific test content
and
question distribution (weightings)

23. “How were the content outlines created?”

24. Test Development Fast Track – national and first 11 states
(AZ, IA, ID, LA, MA, NH, NC, PA, RI, VT, WA)
Normal Track– all remaining states

25. Fast Track Content Outlines
National Component
Drafted by regulator work group
Reviewed and approved by national committee
State Components
Template established by regulator work group
Approved by national committee
Customized for each state by state specific test committee

26. Normal Track Content Outlines
Job Analysis
In April 2009, a job analysis survey was conducted
Individuals from the 55 states/jurisdictions were invited to participate
Invitations were sent to regulators and industry professionals
Over 10,000 people responded

27. Normal Track Content Outlines
The job analysis survey results:
produced only minor revisions to state outline template weightings
validated the original national outline
Moving forward:
all future states will use new content outline template
during annual test maintenance cycle, FT states will incorporate new outline template

28. “Can I access these content outlines?”

29. Yes, Absolutely! Outlines are on the NMLS Resource Center
Professional Requirements> Testing> Test Availability Schedule and Content Outlines
Currently posted:
National outline
State outlines
AZ, IA, ID, LA, MA, MI, NH, NC, PA, RI, VT, WA
DC, GA, KY, MD, NJ, VA, WY (just posted)

30. Content Outlines
content outlines

31. And we should also mention…
Content outlines are an overview of what will be on the test.
They can be utilized as a study tool for test takers.
No additional study guides or study materials will be developed
or endorsed by NMLS.

32. Coming Soon: More Outlines
In early November, more content outlines will be posted (SD, NE, AR, NM, NY, MS)
As new states come on board, new outlines will be developed and published
Check back often

33. Part 3.
Test Questions

34. “How are test questions created?”

35. Test questions are: Written by subject matter experts
Reviewed, edited, and approved by committees
Created from content outline categories and subcategories
Supported by reference materials (laws, regs, etc.)

36. Test Questions Debunked
All questions are 4-option multiple choice
No short answer, fill-in-the-blank or true/false
Only one correct answer and 3 incorrect options
No “All of the above” or “None of the above”

37. Test Questions Debunked
No trick questions
No questions with multiple correct answers
No questions without a correct answer
Good wrong answers
No proper names
“A borrower is applying for a loan...”
not
“John is applying for a loan…”

38. “Why are some questions not scored?”

39. Un-Scored (Pretest) Questions
Un-scored questions are being “tried out.” We only want to put onto our tests questions that meet high quality standards.

40. The Process for Un-Scored Questions
We gather statistics on them first
When we have enough “hits”, psychometricians analyze the stats
They are reviewed to determine if a question is problematic
Eventually, questions that “make the cut” become scored questions

41. Un-Scored Questions: Do not affect a candidate’s score
Are not marked in any visible way to the test-taker (candidates can’t distinguish between a scored or an un-scored question)
Are scattered randomly throughout the test

42. “What if a law changes that affects a test question?”

43. Test and Item Bank Maintenance
All questions are reviewed annually
If a new law affects a question, it will be addressed
During maintenance, pretest questions can become scored and old scored questions may be retired
After maintenance, new versions of the tests are released
If an outline changes, it will be updated online (with new date posted)

44. Test and Item Bank Maintenance
What if a law change goes into affect prior to the scheduled maintenance cycle?
Updates to laws and rules are consistently monitored
If a question is found that is no longer accurate, the following steps may occur:
SMEs may be called on to access the validity and edit or revise if necessary
A question may be retired and replaced with another

45. Part 4.
Review Committees

46. Who is included in the State Review Committees?

47. State Review Committee
6 – 8 state industry representatives
Experienced in mortgage industry (time and professional associations)
Balance of regulators and industry professionals
Requires time and dedication

48. “How much time and dedication does this take?”

49. Virtual Meetings All meetings are conducted by Webinar
There are a total of 8-9 meetings
The committee has one meeting a week
The committee agrees on the same day and time for each weekly meeting

50. Why virtual meetings?
Maximum flexibility in scheduling and requires only a computer and a separate phone line
Pearson VUE can work with multiple states concurrently
Each committee member can see test questions directly with no security risks of paper copies
Ability to track progress and customize the next steps, including which items to present, depending on the state’s unique issues

51. What do the meetings accomplish?
The first meeting is focused on developing content outline and is 2 hours
The next 6-8 meetings are focused on approving test questions and are 4 hours each

52. Same Goal, Different Role
Test development knowledge versus content knowledge
Pearson VUE is expert in how to test
Subject matter experts know what to test
Every process step is introduced by a training presentation of about 30 minutes

53. Committee Process Steps
Review content outline template and develop a customized, state-specific test blueprint
Review test question root pool; customize content and approve questions for use on test
Develop new test questions for unique state-specific content areas
Classify test questions to the areas of the content outline
Reference test questions to state law, regulations, and rules

54. Part 5.
Test Forms

55. “What are test forms?”

56. Test Forms Definition: Forms are the different versions of a test
Purpose: The purpose of having multiple forms in a test is to enhance test security

57. Test Forms For example, the national test component has
90 scored and 10 un-scored questions.
The different forms (versions) of the test all have the same number of questions.
To ensure fairness and reliability, different forms of the national test are all of equal difficulty.
Questions present in random, computer-generated order each time for each form.

58. Test Administration

59. Preparing to Take the Test

60. MLO Testing Handbook
Everything you need to know about testing

61. Enrolling: Step-by-Step Instructions

62. Test Enrollment Basics
A candidate does not have to complete education prior to enrolling or taking a test.
Before enrolling, a candidate must have a unique NMLS ID number (must either already have made a filing or created a base record).
A candidate must request and pay for an enrollment in NMLS before scheduling a test session (*enrollment fees are non-refundable– listed on testing page).
Each enrollment window is open for 180 days, during which the candidate must schedule to take the test.

63. “After candidates create enrollments, how do they schedule test appointments?”

64. Next Step: Scheduling a Test

65. Basics of Test Scheduling
To schedule, must have an NMLS ID number
500 test centers for candidates to choose from (Pearson and Prometric)
After selecting a state on “Find a Test Center” page, a list of all test centers in that state will appear
Scheduling– online or over the phone

66. Basics of Test Scheduling– More Info
All tests are available at all test centers
(i.e. candidate may take AZ test component in PA)
A company can request a group appointment
(see Prometric or Pearson web sites for more info)
*Special accommodations requests may also be made if necessary.

67. Basics of Scheduling– More Info
“Can a test-taker schedule the national test component and a state component on the same day?”
Yes, but only if a test center has two back-to-back appointments available.
However, please encourage candidates not to overlap their testing appointments or they can run out of time!
(*The test time for the national component is 3 hours and each state component is 2 hours.)

68. “What if a candidate needs to reschedule a test appointment?”

69. Rescheduling a Test
Must contact Pearson VUE or Prometric directly to reschedule
Must be rescheduled no later than 12 p.m. local time two business days prior to appointment
Appointment scheduled for: Must be cancelled or rescheduled no
later than 12:00 p.m. local
Sunday, Monday Thursday of the preceding week
Tuesday Friday of the preceding week
Wednesday Monday of the same week
Thursday Tuesday of the same week
Friday Wednesday of the same week
Saturday Thursday of the same week
*If a holiday occurs during the normal cancellation/reschedule period, this
schedule adjusts to define "timely cancellation" as one day earlier than stated
above.

70. The Test Center

71. Entering the Test Center
Arrive 30 minutes prior to test time
Must bring a current, government-issued ID with candidate’s photo and signature.
Will be photographed; finger or palm print may be taken
Candidate may not bring anything into testing room (may be asked to show empty pockets)
All belongings such as notes, cell phones, watches, wallets must be placed in locker
Basic calculator (not financial calculator) and scratch paper are provided
Must read and agree to the Rules of Conduct

72. Test Security-- Policies
All candidates must read and agree to the “Rules of Conduct” prior to taking the test.
They stress the confidentiality of all of the test questions and test materials, as they are the copyrighted, proprietary property of NMLS. 
Any attempt to modify, publish, transmit, or create derivative works of the test materials, in whole or in part, is a direct violation of copyright laws, the Agreement and the Rules of Conduct and will subject you to federal and/or state sanctions.

73. Test Security--Policies
Also, when creating a base record, NMLS users agree to use the system and its test “only for activities related to obtaining or maintaining a mortgage loan originator license”.
Candidates who violate or may violate the Agreement, the Rules of Conduct, or copyright laws, who own, operate, are employed by, or are otherwise affiliated with organizations that are NMLS Approved Course Providers or organizations that intend to seek NMLS Course Provider Approval status, may cause their NMLS Course Provider applications to be denied or their Approval status to be revoked, suspended, or subject to other sanctions.

74. Computer Test Operations and Functionality

75. Test Time-- National National Component
2.5 hours test time (150 minutes)
15 minutes allotted for tutorial (before test)
15 minutes allotted for optional survey (after test)
Total seat time = 3 hours
100 questions in 150 min. = 1.5 min. per question

76. Test Time– State Components
State Test Components
1.5 hours test time (90 minutes)
15 minutes allotted for tutorial (before test)
15 minutes allotted for survey (after test)
Total seat time = 2 hours
Approx. 60 questions in 90 min. = 1.5 min. per question
*State test lengths vary from 55 to 65 total questions

77. “What is the computer test functionality. Is it simple to use
“What is the computer test functionality? Is it simple to use? How does it work?”

78. Tutorial and Test Functionality
Tutorial walks candidates step-by-step through test functionality.
Some key features that candidates can utilize:
Toggle back and forth between test questions
Mark a question for review and then come back to it later on
Change an answer

79. Exit Survey The exit survey is completely optional.
All of the data collected in the survey is monitored and shared with us.
Candidates may also provide feedback to the test center administrator on the enrollment process, test center facilities, content of test questions, etc. All comments will be reported.

80. Score Reports At the end of the test session, candidates will:
see their test scores on their computer screens
receive a paper copy of their score report
These scores do not become official until they are entered in NMLS.
Within 48 hours of completing a test, candidates will receive an informing them that their official scores are now ready to be viewed in NMLS.

81. “What happens if a candidate fails?”

82. If a candidate fails…
Score report has a diagnostic table that informs candidates of how they performed on each outline content area.

83. Retake Policy

84. Retake Policy Must wait 30 days between each test taking attempt
May create an enrollment and schedule an appointment, but can’t take the test prior to the end of prescribed wait time
After every fourth failed attempt, a 180 day wait time is instituted
*After passing a test component, a test taker will not be allowed to retake that particular test component.

85. Other Testing Information

86. State Test Certification
Some states that had valid tests prior to the passage of the SAFE Act may elect to certify their state tests.
In these cases, candidates who previously passed that state’s test may not be required to take the SAFE state test component.
Candidates must check with their state regulators.
All candidates, however, must take and pass the national component.

87. Test Enrollment Update
National AZ IA ID LA MA NH NC PA RI VT WA
Total Enrollments
1230 8 53 15 40 7 19
232 96 9 1 64
1774

88. Tests Taken Update National AZ IA ID LA MA NH NC PA RI VT WA
Total Tests Taken
318 3 17 8 21 2 7 46 1 30
470

89. Any Questions?

90. Your Hosts Anne Altman – Manager, Mortgage Education Testing Programs
Pete Marks – VP, National Mortgage Testing & Education Programs
Rich Madison – Director, Mortgage Education Programs