Designing Online Tests for Traccess

Introduction:

Traccess has a built in software package to allow developers to build tests in a multiple choice format that can be easily administered.  This testing product is linked directly to the database tables and generates the self-assessment/qualification tracking specific to "cognitive" learning folders.  If a "test only" folder is selected in Traccess the folder will automatically turn black when a user has answered 100%  of the questions correctly.

Designing the tests in Traccess needs to be done with care and attention to detail.  Understanding the use and effectiveness of different application or wording of questions is extremely important.  Refer to Theory Supporting Multiple Choice Questions document to understand the foundations of testing the different cognitive levels of learning.

Designing Multiple Choice Questions  

Definition:

A "multiple choice question" or "MCQ" is a question where one alternative out of a list of choices is selected in response to a "question stem".

Example Question Stem: "Why do tests exist in the Operations Support System?"

Example Alternatives:
    (A) to frustrate the operators

    (B) to prove incompetence

    (C) to allow self-directed learning

    (D) to raise internal competition

In this example (C) is the correct alternative (the answer).  Incorrect alternatives are also labelled as "distractors".

One feature of this type of question is that there is a finite possibility of selecting the correct alternative by purely guessing.  In fact, in this example you have a 1 in 4 chance in answering the question correctly even if you don't know the subject matter.

True/False questions are also considered to be MCQ's as there is always a question stem and two alternatives, one being a distractor.

Example Question Stem: "There is little skill involved in designing tests as they are based purely on subject matter."

Example Alternatives:

    (A) True

    (B) False

In this example (B) is the correct answer as there are many facets that need to be considered when designing and building tests using MCQ's in Traccess.  Practice in both writing questions/answers and editing is essential.

Advantages to Multiple Choice Questions

MCQ's can be designed for different cognitive levels: application, synthesis, analysis, and evaluation.
1. You can simply challenge a learners ability to recall facts. (remember the names of components of equipment)
2. Test for ability to apply factual knowledge to given situations. (sequence of events of a process)
3. Test problem solving ability. (work through a formula or mathematical/physics question and choose the correct answer)

MCQ's can be designed with a diagnostic end in mind, in order to find out whether specific areas of a given subject are adequately known, or in order to detect misconceptions.  This can provide feedback on the effectiveness of the instructional materials.

Compared to an essay type question, learners with poor reading skills and second-language barriers need not be at a disadvantage, as long as questions are appropriately written.

MCQ's allow for variation in asking for the same outcome.  Large banks of test questions can be designed relatively quickly to allow for a wider choice of randomisation from the Traccess testing software.

In a traditional paper based test, MCQ's are easy to mark, evaluate and compile for statistical purposes.  Traccess testing software makes this even easier and allows for user level self-assessment and qualification on knowledge folders.

Types:

• question/right answer
• incomplete statement
• best answer

Potential Problems with Multiple Choice Questions

Construction of good test questions requires special care and is time-consuming.

As the test writer, it is easy to fall into the trap of depending on "recall" type questions as they are the easiest to design.

MCQ's have a reputation for being trick questions; wording needs to be concise to provide confidence in the competency based learning process and eliminate any questions/answer that may be perceived as trick questions.

Ambiguous MCQ's plague both new and experienced test designers.  Be clear about the outcome you are looking for.  Always validate the tests against your user guide and "MCQ's Do's & Don'ts" checklist with other subject matter experts for clarity of the question.

Do's & Don'ts Specific to Traccess

• The software does not allow the learner to select more than one answer; do not use more than one correct response in a set of alternatives.
• Do not use the DELETE KEY on your keyboard, it will either delete the whole question or the whole answer line.  When editing, use the backspace key to remove text.
• Save your test often when typing directly into the Traccess testing bank; click on the save icon in Traccess.
• Choose your Must Ask questions carefully, keep the end learning objective in mind.
• Use graphic attachments, built directly from diagrams within your descriptive materials.
• Always test the test as a User to be sure that attachments view correctly.
• Word, wordpad or notepad can be used to type questions in, spell check and import into the Traccess test bank.  You can create new tests and append to existing tests this way.  Careful considerations around spacing and format are necessary (see: Traccess help).
• Be consistent with question/answer format, punctuation, capitalization and sentence structure.
• Do not use "all of the above" as the alternatives or answers randomise in position automatically.  If you have to use this type of answer be sure to type "all answers are correct".

MCQ's - Do's & Don'ts (general advice)

• Test for significant learning outcomes.
  The questions should be designed to test the learning objectives of the Traccess folder (see User Guide & Performance Standards).  Be careful of trivia associated with the subject matter.  Questions need to be relevant to the goals of the learning.
• Test for the intended intellectual skills.
  A question my unintentionally be a "test within a test".  If you can't find the answer to the question within the specific Traccess folder don't ask the question.
• Pay special attention to the language used.
  The level of language should be written to the lowest common denominator of reading ability within your audience.
  Use correct grammar.
  Avoid the use of jargon unless specifically testing terminology.
• Be sensitive to cultural and gender issues.
  Avoid turns of phrase and figures of speech that could be construed as racists, sexists or which may have a cultural bias.
• Avoid giving clues.
  Unintentional clues linking the stem to the answer.  Grammatical/punctuation/format clues linking the stem.  ("an" at the end of the stem is the only article acceptable for the word starting with an "a". Use alternative ending "a/an".
• Avoid using interrelated items.  Questions should not overlap although sometimes it is difficult to develop enough questions for a proper "bank" to pull randomising questions from without repeating the root question information.  Be sure not to have overlapping questions checked as "must ask".

The Question Stem:

• State the problem clearly, simply and as concretely as possible.
• Avoid vague generalizations.
• Do not include irrelevant information.
• No evasive or ambiguous wording or statements.
• Should contain most of the wording in order to reduce the reading load.
• Avoid negative stems.  If this cannot be done, the negative words should always be highlighted by capitalisation.

The Alternatives (answers):

• should be equally plausible if at all possible.
• should be internally consistent (some similar criteria).
• try to ensure that they are approximately the same length.
• avoid "all answers" (analysis shows that these questions tend to be the easiest.  This response shows that the question designer has run out of ideas!)
• do not use "none of the answers" as it doesn't do anything to reinforce the learning.

TRUE/FALSE Question Advantages:

• knowledge level testing;
• users can answer 3-4 questions per minute;
• can test for large amounts of content;
• especially for concepts with 2 logical responses;
• check understanding of popular misconceptions.

TRUE/FALSE Question Disadvantages:

• often easy;
• 50/50 chance of getting the right answer by guessing;
• need a large number of "must asks" from a substantial bank of questions to get fair results.

Tips for Writing Good TRUE/FALSE Questions:

• avoid double negatives;
• avoid long/complex sentences;
• use specific determinants with caution: never, only, all, none, always, could, might, can, may, sometimes, generally, some, few;
• use only one central idea in each item;
• don't emphasise the trivial;
• use exact quantitative language;
• don't lift items straight from the book;
• make more false than true (60/40) (guessers are more likely to answer true).

Matching Question Advantages:

• good for knowledge level;
• if appropriately covered can test comprehension;
• excellent for covering a lot of facts.

Types:

• terms with definitions;
• phrases with other phrases;
• causes with effects;
• parts with larger units;
• problems with solutions.

Tips for Writing Good Matching Questions (also for Pop Quizzes):

• need 15 items or less;
• give good directions on basis for matching;
• use items in response column more than once (reduces effects of guessing);
• use uniform material in each exercise;
• make all responses plausible;
• put all items on a single page;
• put response in some logical order (chronological, alphabetical, etc);
• responses should be short.

Glossary

For Document Control Purposes Only:

If any of the resource materials can be improved or if they are incorrect or out-of-date, write the changes on the documents.  Attach them to this sheet and give it to your Team Leader.  The materials will be updated promptly so the next person using them will have up-to-date information. Thanks!