Build the Item Bank Before the Quiz

A quiz with one fixed version has a short shelf life. Learners compare notes, and a retake ends up measuring who remembers the answer key. The score stops describing what anyone can do.

The fix is a bank of questions that each quiz draws from at random. Building a good one is mostly planning done before the first question gets written. This is the method.

The strict definition and the working definition differ

In testing research, an item bank is a set of items that have been calibrated, meaning each one has measured statistics from real test takers. An item pool is a set of items grouped by content that hasn’t been through that process. A medical education paper traces the term item bank to the mid-1960s and says the distinction hasn’t been widely adopted. Most people now use the two words interchangeably.

Learning platforms use a working definition. Moodle calls the library a question bank, and a quiz can pull random questions from a category in it when each learner starts an attempt. Its documentation gives the example of 10 random questions drawn from a category of 30. Canvas New Quizzes calls its library an item bank, and an author can use every question in it or a random selection.

So in practice the bank is the library you maintain, and the pool is the set a single quiz draws from. Pick one definition, write it at the top of your assessment spec, and keep to it.

Write the blueprint before the first question

A blueprint is a grid. The rows are your learning objectives, the columns are the cognitive levels you want to test, such as recall and application, and each cell holds the number of items the quiz needs. Every item you write maps to exactly one objective. If you can’t name the objective, the item doesn’t go in.

Then size each pool. A pool has to hold more items than a quiz draws from it, or every learner sees the same questions. Moodle’s example draws 10 from 30. Decide how much overlap between attempts you can live with, and size the pool from that. Write the application-level items as scenarios: a short situation, a decision, and answer options that are each a plausible choice.

Pool items have to be interchangeable

A random draw only works if any item in the pool is a fair stand-in for any other. That means the same objective, the same cognitive level and similar difficulty. Put a recall question and a four-constraint scenario in one pool and two learners can sit tests of different difficulty because of the luck of the draw.

Tag every item with its objective, topic, cognitive level, expected difficulty, author and date written. Then check what your platform can filter on before you lock the tags. One university’s guide to Canvas New Quizzes says questions to be randomized together must sit in their own item bank, so there one pool means one bank. Moodle can narrow a random draw by category and tag.

Pilot first, then read three numbers

Run the draft items past a small group and review the results before any score counts. Classical item analysis gives you three checks.

The first is difficulty, the share of learners who answered the item correctly. A common guideline puts well-functioning items between 0.30 and 0.90, with about 0.50 giving the most separation between strong and weak performers. The second is discrimination, how well the item separates high scorers from low scorers overall. A point-biserial correlation of 0.20 or higher is a common floor, and some guides want 0.25. The third is distractor performance. The correct option should correlate positively with the total score, each wrong option should correlate negatively, and every wrong option should attract at least a few learners. An option nobody picks is padding.

Treat the numbers as diagnostics. Difficulty and discrimination shift with the group taking the test, and a small pilot gives noisy results. Flag the items that miss, read them, and revise the wording or the distractors before you drop anything.

Retire items on a schedule

A bank goes stale. The same medical education paper warns that random extraction from a bank fails in fast-moving fields because items go out of date. The same applies to anything with a policy, process or screen that changes.

Give every item a review date. Track which items each learner has seen, so a retake draws fresh ones. Moodle tries to serve a retake a different set of questions where it can, which only works when the pool has room. When the content changes or the statistics drift, revise the item or retire it. Mark a retired item as retired in its metadata and leave it in the bank, so the next author can see why it went.