More Secrets to Writing Effective Assessments

By Joseph Berman, Editorial Director, Science

Writing effective exam questions is a nuanced process, particularly when it comes to assessing critical thinking. At the 2024 NSTA National Conference in New Orleans, I presented a session on “How to Write, Evaluate, and Master High-Quality Assessments,” where I shared insights into creating questions that go beyond simply identifying correct answers. The most impactful assessments allow students to showcase their understanding and skills, serving not only as a gauge of current knowledge but also as a roadmap for future learning. In part one of our blog series on assessments ‘The Secret to Writing Effective Assessments‘, we explored the criteria for effective assessment design. This post explores how to craft assessment questions that boost students’ confidence in their knowledge and skills.

If you are a classroom teacher, especially in high school, I will wager that you have listened to students complain about unfair assessment. Students often feel annoyed (at best) or angry (at worst!) when they encounter an assessment item that they deem to be trivial or ambiguous, or simply wrong for whatever reason. And good for them! They are treating assessment seriously, as we want them to do.

In my experience as a teacher, I found students’ arguments to be justified on some occasions, but not always. Often, the problem was that the students ignored a detail in the question, or they tried for a shortcut that did not really apply.

Below is an example of a legitimate item that my students sometimes misinterpreted.

Many scientists argue that viruses are not alive. Which statement provides the strongest evidence for this argument?

1. Viruses contain protein and genetic material.
2. Viruses cause disease by infecting cells.
3. Viruses cannot reproduce independently. ✔
4. Viruses can remain dormant inside a cell.

The students argued that answer A, B, or D is a true statement, and therefore should also be correct.

This is a teachable moment.

The underlying problem is the mindset—a faulty one—that a multiple choice question is really a true-false question in disguise. The student identifies a true statement among the answer choices, and then moves on.

While this strategy can lead to some success, it often fails, especially on items that assess critical thinking. As students progress through high school and then college, they will find more and more instances where a true statement is not the correct answer.

For a trivial example, consider this question-and-answer conversation:

Ayisha: Steve, when did you wake up this morning?

Steve: I ate a bowl of cereal with milk for breakfast.

Not a realistic exchange, but it illustrates a point. Steve’s reply may be truthful and accurate. However, his reply is incorrect—or at least, not appropriate—because it does not answer his friend’s question.

Here is the same example as a multiple choice item:

When did Steve wake up this morning?

1. He ate cereal for breakfast.
2. He woke up at 7:00 A.M. ✔
3. He rode the bus to school.
4. He studied for his science test.

The lesson for test-taking is simple: The correct choice must answer the question. Steve may have eaten cereal and rode the bus and done many other things that morning, but the question demands his wake-up time.

Look again at the item on viruses. The question stem asks for the strongest evidence that viruses are nonliving, not merely a random fact about viruses. All four answer choices are true statements, but only choice C serves as the evidence requested.

More Item Analysis

Of course, complaints about assessment can also be reasonable and justified. Let’s look at some examples of poorly-constructed assessment items and find ways to improve them.

1. Denying the Premise

Poor item:

Why do all organisms depend on sunlight?

1. They use sunlight directly as an energy source.
2. They use sunlight indirectly as an energy source.
3. They depend on sunlight to sense their environment.
4. Some don’t! They get energy from chemicals. ✔

Critique:

Choice D denies the premise of the question stem, adding a level of confusion that is not necessary. This type of logic might work in a conversation between a teacher and student, but it makes for a poor assessment. Rethinking the question stem can resolve the problem, although in this case it also adds to the length of the question stem.

Improved item:

Devin claims that all organisms depend on sunlight for energy. Talia suggests that Devin should replace the word all with nearly all in his claim. Which statement best explains this suggestion?

1. Plants and algae use sunlight to perform photosynthesis.
2. Animals get energy from sunlight indirectly through food chains.
3. Organisms that live in caves get energy by decomposition.
4. Some marine organisms get energy from a chemical source. ✔

2. Ignoring the Purpose

Poor item:

Who aimed a beam of alpha particles at a thin sheet of gold foil?

1. Ernest Rutherford ✔
2. J. J. Thomson
3. Niels Bohr
4. Erwin Schrödinger

Critique:

This item is not terrible. It has one correct answer (choice A, Ernest Rutherford) and it assesses knowledge of an important experiment in the history of chemistry. However, the item ignores the purpose of Rutherford’s work, which is the heart of the information that the student should know.

Improved item, although with more text:

Which experimental procedure allowed Rutherford to discover the atomic nucleus?

1. aiming alpha particles at a sheet of gold foil ✔
2. observing electrically-charged droplets of oil
3. arranging the elements in order of atomic mass
4. measuring the ionization energies of various elements

3. Wrong Format

Poor item:

Which two organelles are each surrounded by a double membrane and contain their own DNA?

________ and _________

Critique:

The intended answers are mitochondria and chloroplasts. However, a biology student might include nucleus as an answer. In this case, the student would be correct. The nucleus also has a double membrane and contains DNA. The item-writer surely knows about the nucleus but managed to ignore it when constructing the assessment.

The item also presents difficulties if it is meant to be scored by computer. Neither mitochondria or chloroplasts is easy to spell. The two words could be presented in either order, and either singular or plural.

For these reasons, the item might work better as a multiple-select multiple choice. Notice the boldface for the word two, helping the student identify the item format.

Improved item:

Which two organelles are each surrounded by double membranes and contain their own DNA? Select two answer choices.

1. rough endoplasmic reticulum
2. smooth endoplasmic reticulum
3. mitochondrion ✔
4. central vacuole
5. chloroplast ✔

Summary

High-stakes assessments are important for many stakeholders. But everyday classroom assessments are also important, and not only for the gradebook and report card. When students successfully complete an assessment, they gain confidence in their knowledge and skills. Assessments also are incentives for learning. How many students gain mastery of a topic by preparing for the final exam?

In educational publishing, we may be charged with preparing assessments to accompany instructional lessons, to prepare students for a standardized test, or as stand-alone components in a curriculum. The assessments also must meet other requirements, including grade-appropriate readability, lack of bias, coverage of state or national standards, and targets for depth of knowledge (DOK) and Bloom’s taxonomy.

That’s a heavy lift for the assessment writer. It’s also one reason why KGL treats a writer’s submission as only the first step in the publishing process. Every assessment item is subjected to at least three rounds of review before we release it to our client.

Joseph has over 25 years of K–12 science and math product development experience. He edited high school science textbooks at Prentice Hall and served as Science Editor and later Editorial Director of Technology at Macmillan/McGraw Hill. At Scholastic, he developed math programs such as MATH 180. Joe oversees science development for KGL and can be reached at info@kwglobal.com.