Creating Constructed Response Items

What are constructed-response items?

Constructed response items require students to write out or construct a response to a question. Two types of these items are:

1) Fill-in-the blank
2) Essay

Fill-in-the Blank
(Completion Items)

Students complete the sentence or phrase by writing in the missing word or words into the blank space.

Advantages of Fill-in-the Blank

1) They are easy to construct.
2) They require students to provide answers.
3) The teacher can include many items in a test.

Limits of Fill-in-the Blank

1) Items are primarily limited to assessing the recalling of information.
2) Multiple answers can be accepted and be less objective.

Characteristics of Quality Fill-in-the Blank Items

1) Items should measure a specific skill.

-This should measure the skills identified in the performance objective or table of specifications.

-Do not create the item until the skill to be measured has been identified.

Objective: Students will describe the stages of the life cycle of a frog.

Example: The life cycle stage of a frog where the frog is fully developed with lungs, legs, and no tail emerges from the water is know as the _________ stage.

Non Example: The difference between the adult stage and the tadpole stage is ___________.

2) The reading skills required by this item should be below the students' reading ability.

-The teacher will not be able to decipher if the wrong answer is due to the lack of knowledge or poor reading skills.

3) There should be one or a similar cluster of answers that are correct for each item.

-Create items so that there is only one or or a similar cluster of answers that are correct. This reduces errors caused by questions that have several valid answers.

-It may help to first create the answer and then develop a question for which the answer is the only appropriate response.

Example of possible error: In the adult stage of the life cycle of a frog, the frog has ____________.

(Possible Answers: no tail, fully developed lungs, fully developed legs, emerged from water)

4) The item should not be copied directly from the book or materials that are given during instruction.

-Students may try to simply memorize the information, thus eliminating higher-ordered thinking.
-Most information during instruction requires the preceding or subsequent information to be able to complete specific sentences.

Example of a sentence directly taken from the book that is not a high quality fill-in-the blank item:

Unless the item carefully restricts the correct answer, you may not be able to anticipate all the defensible answers when providing the key to the __________.
(answer: computer)

5) The item should contain an identified unit of measure when there is a numerical answer.

-The students may write a converted numeral that is also correct, such as, 1 foot or 12 inches. This may not be recognized by a computer graded system. If the unit is given, then there is no discrepancy as to the appropriate answer.

Example:
How much time did you spend waiting for the bus in all, if you waited for 10 minutes in morning and 20 minutes in the afternoon? _____ minutes

-Possible answers if MINUTES were not indicated: 30 minutes, half of an hour, or half an hour.
-While all are correct answers, the computer may not recognize them all as a correct answer.
-It is also much faster to check if grading by hand.

6) The blank in the sentence should represent a key word.

-It is important that students are able to identify the key words and vocabulary in the blanks. When the blanks are not key words, then the test is not assessing knowledge of the content as much as it is assessing the comprehension of reading.

Example: The third planet from the sun is ________.

7) The blank part of the sentence should be placed at or near the end of the sentence.

-This technique has shown to improve student understanding and increase correct answers. It allows students to answer the blank the first time and eliminate the need to read the sentence again.

8) The number of blanks for each item is limited.

-Time on an item increases when there are too many blanks. When less blanks are used, more items can be included on the test, allowing more of a variety of content to be assessed.

-Too many blanks allows for more possible options for answers.

Example:
I _____ the _______is______.

Possible answer:
I see the ball is rolling.
I know the sky is blue.
I understand the problem is easy.
I hear the school is closing.

This example shows many possible answers.

Essay Items

Students respond to a prompt or questions by writing multiple sentences and/or paragraphs.

Advantages of Essay Items

1) They generally are able to assess behaviors identified by the instructional objective.

2) They demonstrate a student's ability to communicate their ideas through writing.

Limits of Essay Items

1) They limited the variety of content assessed.
2) The scoring of essays is subjective.
3) They can be tedious to score.

Characteristics of Quality Essay Items

1) Items should measure a specific skill.

-This should measure the targeted skills identified in the performance objective.
-Students should not be allowed to choose from a subset of essay items, as this hinders the content-related validity.
-Students may have choice in how they answer the essay item if the same skills are being assessed and they are being assessed by the same rubric as their peers.

Objective: Students will describe the stages of the life cycle of a frog.

Example: Write a story about the life of a frog, from the time it's born to after they become an adult. Be sure to include details about how the frog changes during their life.

Non Example: Write a story about a day in the life of a frog.

2) The reading skills required by this item should be below the students' reading ability.

-The writing prompt or question should include simple sentences and have wording that all students are able to understand.

3) Essay items should take no longer than 10 minutes to brainstorm and write.

-Online assessments should include brief-response essay items (10 minutes or less).

4) The scoring guide (rubric) must be consistent for all learners and raters of the essay.

-Scores should be comparable for those who grade the essay, using the assigned rubric.

5) The scoring guide (rubric) should identify a correct and complete response.

-This can be difficult if the essay item is a broad question.
-To reduce the broad question, it is suggested that the question be broken down into smaller questions that take less than 10 minutes to answer.
-Collaboration with creating and assessing rubrics with colleagues is the best way to develop a rubric that is complete and correct.

6) The essay item must be created so that students are able to relate the item to the scoring guide.

-Students can be knowledgeable, but don't understand how to demonstrate the knowledge that is being assessed for. When they are able to understand the knowledge that is being assessed, then they are able to demonstrate the correct knowledge.

Scoring Guides

Scoring Guide (Rubric) Characteristic

1) Students should be given the total number of points that the item is worth.
2) The rubric identifies the characteristics that will be measured.
3) The rubric should be clear and concise as to the exact points that are given for each level of proficiency in each characteristic.

There are two main scoring guides (rubrics) that we will look at and discuss how to create them.

1) Analytic Rubrics
2) Holistic Rubrics

Analytic Rubrics

Analytic rubrics outline the criteria of an assignment and divide it into parts. This allows the students be able to see each criteria detailed into what they need to do to demonstrate that they have a proficient knowledge of the criteria or how the criteria can be enhanced.

Advantages of Analytic Rubrics

1) They provide clear and identifiable criteria that need to be met in order to demonstrate a specific level of knowledge and a score can be determined for each level of criteria.
2) They allow teachers to be able to provide more detailed feedback to students without having to explain the feedback to students.

Disadvantages of Analytic Rubrics

1) They are tedious and time consuming to make.

2) Due to the detail of the rubric, students may overlook important details.

The following six slides discuss the process to create an analytic rubric.

1st Step
Decide on the different skills and abilities that the students need to be able to know and do. Decide what assessments they will complete or create to show evidence of their knowledge. Decide what it looks like for a student's evidence to be "excellent," and also "emerging". These will be some of the descriptors for each criteria. The criteria will be broken up into different skills and the skills should be characterized by a brief phrase or word. The demonstrated skills should be measurable through the assessment of the work sample. The criteria will be listed down the first column on the left of the rubric.

Examples of Criteria (not an exhaustive list):

1) Content-The content is addressed by identifying a topic and using supporting details and examples.
2) Organization-The essay is written in a logical sequence of reasons and supporting information.
3) Conventions-The essay uses proper grammar, spelling, punctuation and word usage.

2nd Step
Identify the measure of success of what the students are expected to be able to display for each level. These are often identified using descriptors and can include or be numbers. This gives you the number of columns that you need and should be placed at the top of each row. There are normally at least 4 to 6 descriptors.

Example of Descriptors:
-poor, below average, average, above average, excellent
-novice, intermediate, proficient, advanced
-beginning, developing, competent, exemplary
-ineffective, developing, effective, highly effective
-needs improvement, satisfactory, good, accomplished

3rd Step
Create the description for each achievement level of each criteria. For example, what does poor, below average, average, above average, and excellent convention look like from student work? It is helpful to start with the highest level of achievement and write a description of this level. From this description, you can identify how the student performed and modify that performance level for the subsequent achievement levels.

Example of performance levels:

-one or less spelling, grammar, punctuation, and usage mistakes to many spelling, grammar, punctuation, and usage mistakes
-clear and concise details to vague details
-complete sentences to incomplete sentences
-all the time to none of the time

Each detailed criteria should be placed in a table format with the criteria parts going down the left column and the scoring (in descending order) on the top row going towards the right. The characteristics for each level of criteria would also be in descending order going towards the right and matching the appropriate point value. The next slide shows an example of an analytic rubric using the same criteria and scoring from the previous six slides.

Untitled Slide

Holistic Rubrics

Holistic rubrics are very similar to analytic rubrics, but the criteria for each point value is combined into one section, instead of being broken into different categories. For examples, all of the criteria for each category that would award students 4 points on an analytic rubric would be consolidated and students would need to demonstrate all of those skills or criteria to get 4 points. This description would also be written more broadly and with less details for students .

Advantages of Holistic Rubrics

1) They are easier to create, taking less time than an analytic rubric.
2) They make grading an assignment quick and easy.

Disadvantages of Holistic Rubrics

1) They are less detailed, providing less feedback for students.
2) They provide less guidance as to the specifics of the assignments.

The following ten slides outline the creation of a holistic rubric. This is a very similar process as creating an analytic rubric, so don't be alarmed if the information looks similar as previously seen.

1st Step
Decide on the different skills and abilities that the students need to be able to know and do. Decide what assessments they will complete or create to show evidence of their knowledge. Decide what it looks like for a student's evidence to be "excellent," and also "emerging". These will be some of the descriptors for each criteria. The criteria will be broken up into different skills and the skills should be characterized by a brief phrase or word. The demonstrated skills should be measurable through the assessment of the work sample.

Examples of Criteria (not an exhaustive list):

1) Content-The content is addressed by identifying a topic and using supporting details and examples.
2) Organization-The essay is written in a logical sequence of reasons and supporting information.
3) Conventions-The essay use proper grammar, spelling, punctuation and word usage.

2nd Step
Identify the measure of success of what the students are expected to be able to display for each level. These are often identified using descriptors and can include or be numbers. This gives you the number of rows that you need and should be placed down the first column on the left. There are normally at least 4 to 6 descriptors.

Example of Descriptors:
-poor, below average, average, above average, excellent
-novice, intermediate, proficient, advanced
-beginning, developing, competent, exemplary
-ineffective, developing, effective, highly effective
-needs improvement, satisfactory, good, accomplished

3rd Step
Create the description for each achievement level of each criteria. This is where the rubrics start to differ, as the descriptions will include a description of each category in the same achievement level. For example, what does excellent content, organization, and conventions look like from student work? These will all be in the same description of “excellent”. It is once again helpful to start with the highest level of achievement and write a description of this level and the subsequent levels thereafter.

Example Description (Advanced 4 points)

• There is a clear topic, identified opinion on the topic, and 3 or more strong reasons and details to support your opinion. There is a strong conclusion to the essay. Sentences vary in length using linking words (because, therefore, since, etc.). All sentences are complete. A few or no spelling, grammar, punctuation, and usage mistakes.

Example Description (Proficient 3 points)

• There is a topic, identified opinion on the topic, and there are two supporting details to support your opinion. There is a conclusion to the essay. Sentences vary in length using linking words. Most sentences are complete. There are some spelling, grammar, punctuation, and usage mistakes.

Example Description (Developing 2 points)

• There is a vague topic, opinion, and there is one or less reasons or the details are vague. The essay is missing the topic sentence, opinion, and/or the conclusion to the essay. Few sentences varying in length with many simple sentences. Some sentences are complete. There are many spelling, grammar, punctuation, and usage mistakes.

Example Description (Emerging 1 points)

• Topic and opinion haven’t been identified. There are little to no reasons or details to support the opinion or a conclusion to the essay. Sentences and ideas have no clear organization. Most sentences are run-ons or incomplete. Many spelling, grammar, punctuation, and usage mistakes.

Example Description (Beginning 0 points)

• Did not complete or turn in assignment.

Each score and criteria description should be placed in a table format with the achievement level and/or score (in descending order) going down the left column and the criteria description going down the right column. The description can be written in paragraph form, but it may be easier to read and understand if the criteria description for each score is broken down into bullets, identifying different characteristics of the criteria, as displayed in the next slide.

The next slide shows an example of a holistic rubric using the same criteria and scoring from the previous ten slides and the same writing prompt from the analytic rubric. The scoring for this rubric includes only the points for each achievement level.

Untitled Slide

That was a quick (or maybe not so quick) guide to creating constructed-response items. I hope this was helpful in your journey for knowledge on constructed-response items.