Microsoft Certified Professional (MCP) exams are available for several Microsoft OSs, development tools, and desktop and server applications. Microsoft's Certification Strategy and Development group develops the exams. Exam development professionals, not experts in the exam topic areas, staff this group. To add topic expertise for the exams, Microsoft relies on internal Microsoft groups (the product group that markets and develops the product and the product support specialists who support the product), independent contractors (such as I), and users outside Microsoft. (For information about becoming a technical contributor, see Table 1 on page 74.)
For several years, I've helped develop a variety of MCP exams. My area of expertise is Visual Basic (VB). I wrote about half the questions for the VB 3.0 exam and modified other questions. I also constructed part of the VB 4.0 exam that Microsoft released on April 8, 1996.
If you've taken or are planning to take an exam, if you rely on exam results for hiring and personnel-related decisions, or if you want to help develop an exam, you'll be interested in what goes into making an MCP exam. Here's a brief overview.
Exam Development
Although Microsoft constantly refines the process for developing MCP exams,
all exams go through several major steps. These steps are job analysis,
objective domain development, exam blueprint survey, item pool development, exam
alpha (ensuring technical accuracy), exam beta, cut score, certification exam
publication, localization, and performance monitoring.
Some steps require explanation. After the exam developers create the exam objectives, the developers survey about 30 internal and external reviewers. This exam blueprint survey helps refine the objectives. Following this review, the developers solidify the exam topics.
Adding an item for a topic not on the list of objectives is almost impossible. I can't tell you how often developers submit a great question only to hear, "What objective does this item map to?" If an item doesn't map to an objective, it doesn't get on the exam.
The cut score is a number that determines the pass or fail cutoff for each exam. Each developer decides what percentage of minimally competent candidates must answer each item correctly. The developers then sum all these percentages and divide by 100 to calculate a suggested cut score. They then average the cut scores from everyone contributing in this phase to determine a final cut score for the entire exam.
Bloom's Taxonomy
Microsoft gives test developers a lot of guidance, including instruction on
Benjamin Bloom's Taxonomy of Educational Objectives for the
Cognitive Domain. Bloom's taxonomy has six levels:
The idea behind this taxonomy is to test the student's ability to use knowledge. Microsoft instructs developers to prepare items in high categories such as evaluation, synthesis, and analysis, instead of testing raw knowledge.
The 7 Cs
The internal developers thoroughly evaluate each item before it appears on
an alpha and then beta exam. Microsoft internally identifies this evaluation
process as the 7 Cs, for correct, clear, consistent, complete, concise, current,
and culturally unbiased. The developers use several questions to evaluate the 7
C's, including Is the topic relevant? Is the wording straightforward? Is the
item free of contradictions? Are there at least three distractors that are
plausible to an uninformed person? Are the answers short and simple? Is the
information current? Is the item fair? Is the item free of ethnic, cultural, and
gender bias?
Microsoft gives external developers a shorter list. For example, they typically don't need to worry whether an item is culturally unbiased; the editorial staff addresses such concerns later in the development process.
Types of Items
Exam items can take several forms. Multiple-choice questions can have one
correct answer, or multiple correct answers (and the student knows how many are
correct), multiple correct answers (and the student doesn't know how many are
correct), multiple-response questions with one scenario and several questions
(the student must select the best answer), free-form text (for example, command
lines where several possible syntax alternatives can work; one such command is
DIR), and graphics with hot-spots (the student points and clicks on a correct
answer).
I make the exams as difficult as possible. I favor items with multiple correct answers where the student doesn't know how many are correct. These questions are like a string of several true/false questions, and the student knows only that at least one question is true. Unfortunately, these items give more unpredictable test results than other types of items. They would probably be more useful if a mechanism were available for calculating partial credit.
Although the exams emphasize questions with one correct answer, some items (especially the multiple-response questions) require students to select the best answer. Microsoft uses this approach for questions on troubleshooting and planning, where professional judgment is important.
Item (Question) Development
The taxonomy and 7 Cs are somewhat abstract, but Microsoft gives developers
specific instructions. For example, no item can be a verbatim copy from
instructional or reference materials. Also, each item must reflect facts, not
opinions; use straightforward language; not be tricky; be relevant, not obscure
or trivial; contain specific determiners (all, none, always, only, never, best,
most) only when there are no exceptions; contain no negative words (no, not);
and contain no indefinite qualifiers (some, many, several, few, usually). In
addition, the answers must present one unquestionably correct answer; be
mutually exclusive (i.e., not overlap); contain no phrasing such as "all of
the above," "none of the above," or "both A and B."
Distractors (incorrect alternatives in multiple-choice questions) must be
plausible to an uninformed person and unquestionably incorrect to an informed
person.
| Contact Info |
|
Microsoft Certification Development Team One Microsoft Way, Redmond, Washington 98052 Email: tcon-mcp@microsoft.com Fax: 206 936-1311 |
Sample Question
The sidebar, "How an MCSE Exam Question Is Created," on page 72
shows the development of a typical exam item. Embedded comments are visible to
help you understand the gradual refinement process.
Several things are wrong with the original version of the item: Generic references to a network administrator make the item hard to follow; the item doesn't relate to a scenario, and answers don't follow from the question; the item contains a hidden negative and sentences in passive voice; and answer C is the only one that contains two conditions.
Each MCP exam involves a lot of work and a lot of people at various steps. Microsoft evaluates exam statistics to continually refine the process. To obtain more information about individual tests, visit the Microsoft Training and Certification Web page at http://www.microsoft.com/train_cert.
| TABLE 1: Becoming a technical contributor | |
|
A non-Microsoft person may want to help develop a Microsoft Certified Professional exam for several reasons. You might take a professional interest in the exam, or you might want to network with other professionals in your area, gain professional credibility, or simply enjoy and take pride in participating. Technical contributors participate in several phases in the exam development process, each of which involves a time commitment. | |
| Exam Development Phase | Estimated Time |
| Objective Development (OD) | 2 hours |
| Blueprint Survey (BP) | 2 hours |
| Question Development (QD) | 12 days |
| Alpha Review (AR) | 8 hours |
| Beta Exam (BE) | 3.5 hours |
| Cut-Score Survey (CS) | 4 hours |
To become a technical contributor for Microsoft certification exams, send the following information by mail, email, or fax to the Certification Development Team:
Developing an exam doesn't give you an edge in taking the exam. You generally see only some items, which aren't in final form, and several months can pass between helping develop the items and taking the exam. That's just enough time for everything--including the distractors--to look vaguely familiar.