Content Prioritization for Standards-based Education

Content Prioritization for Standards-based Education[edit | edit source]

In contrast to the other methods of content prioritization presented here, some designers are not given latitude to disregard or eliminate content. Most often this is because the content is directly related to externally or internally imposed standards, such as OSHA-compliant safety awareness programs, or state and district academic standards required of primary and secondary schools.

Schools in particular are required to achieve a considerable number of standards, and the general consensus among administrators and teachers alike is that there are too many standards to cover uniformly and fully. Noted education consultant Dr. Robert Marzano notes that in order to cover all standards “You would have to change K-12 to K-22. The sheer number of standards is the biggest impediment to implementing standards.”

Yet state and local standards must be met. Content prioritization is essential in this case as well, though rather than discarding content altogether, the prioritization is of time and energy devoted to accomplishing each standard.

There are two main difficulties users will encounter when attempting to execute instructional design using prioritized standards:

1. Evaluating standards against the specific criteria of your school or organization.
2. Achieving consensus among the stakeholders regarding high, medium, and low priority standards.

Thus, the method needed will need to be adaptable and democratic. An adaptable nature will allow it to be tailored to the organization’s unique situation, and being democratic will ensure that the results are well rounded and - just as importantly – fair, meaning that teachers, trainers, and/or designers are more likely to buy into the decisions made based on the consensus.

About this Method[edit | edit source]

The method that you will learn about in this section has two stages:

1. Develop additional criteria using Nominal Group Technique.
2. Prioritize standards using a Decision Matrix.

The method detailed below uses an example that takes place in a K-12 setting, but it can be used for other organizations that use externally-implemented standards and/or high-stakes standardized testing.

After reading the section about the Nominal Group Technique, you will understand how it allows decision-makers to develop criteria that are important and specific to their institution and how the group ranking process promotes fairness in the adoption of the criteria in the decision-making process.

After reading the section on processing the resultant data with a Decision Matrix, you will understand how the criteria are used to prioritize the standards.

Nominal Group Planning[edit | edit source]

Nominal Group Planning allows participants to anonymously submit their ideas and rank the group results. Group discussion allows participants to debate the pros and cons for each criterion, and then anonymously revise their rankings.

Steps for Conducting a Nominal Group Planning Session

1. Gather all decision makers into a group. If more than ten people are present, it may be advisable to break into subgroups, as participant attention and enthusiasm will wane if the proceedings take too long.
2. The facilitator (chosen beforehand) explains the purpose and process of prioritization to all participants. This should include a clear statement of the objective of the meeting.
3. Next, each member of the group should write down criteria which they feel will be useful in evaluating the standards. This should be strictly individual, as to preserve the anonymity of the author at a later stage. This is the most unstructured step in the whole prioritization process, and because it is done individually and early in the process, it is likely that the facilitator will need to supply plenty of encouragement for participants who are still uncertain in their purpose. It is likely that a few criteria will already be established externally, or may be particularly obvious, so the facilitator may want to use those criteria as an example.
4. The facilitator will take all lists of suggestions and compile them into a master list that will be manipulatable by the facilitator and visible to all participants (such as a dry erase board or transparency on an overhead projector). The master list should not have grouped entries from the same individual list, but should be random. If the number of decision makers and/or the number of criteria offered is substantial, offering the group a break will give the facilitator the time to compile the large list without boring the participants.
5. Through group discussion, the entries in the large list should be consolidated and simplified. Undoubtedly some specific criteria will be subsumed in more general criteria, but the facilitator should warn against over-consolidating into a few criteria that are too general or broad. For instance, ‘opportunity cost’ and ‘pecuniary cost’ should not be rolled into a general ‘cost’ criteria, as each is a very different dimension, and ‘cost’ will mean very different things to different people.
6. Having reduced the master list into a consolidated list, each participant should individually (again, to preserve anonymity) compile a ranked list of criteria, where #1 is what they consider the most important criteria of prioritization. It should not be necessary for participants to rank more than ten of the criteria if the total exceeds that number. The facilitator should remind the group that they will have the opportunity to re-rank their decisions at a later stage, so it is not imperative to take a lot of time and consideration when ranking their lists.
7. The facilitator will tally the results, one individual list at a time. One point is assigned to the top ranked criteria on each list, 2 points to the second, and so forth. The individual scores of each criterion are summed, and the criteria are ranked according to the group scores. For example, if the group consisted of three participants with three criteria, the individual lists might look like this:

The ranked tally would then look like this:

• Criteria X: 1 + 1 + 2 = 4
• Criteria Y: 2 + 3 + 1 = 6
• Criteria Z: 3 + 2 + 3 = 8

According to the group, criteria X is the most important, Criteria Z the least, and Y in between.

8. Next, discuss the results as a group. Participants are encouraged to voice concerns about why a high-ranked criteria should not be held in such positive regard, or why a low-ranked criteria should be elevated. The facilitator should encourage discussion among all participants, not just a vocal few.
9. The penultimate step in the process is the same as step 6, only this time participants should be given more time to rank their lists.
10. Facilitator once again tallies and ranks the results. The top criteria are selected for use in the Decision Matrix. The number of criteria selected depend on the situation and if some criteria were established previously to the session. However, if there is a natural break in the ranking data (such as a 50% difference in the score of adjacently ranked criterion) it generally indicates a good cutoff point.

Narrative Walkthrough for Conducting a Nominal Group Planning Session[edit | edit source]

Context: Teachers in a large California school district have faced considerable difficulty in covering the ever-increasing number of state standards during the academic year, and the issue has been a source of friction between the educators and the demanding district school board. The board has responded by enacting a program of standards prioritization that will aid the teachers in refining their curriculum to be functional and to improve concurrency of content taught in each classroom. This is where you come in. Let’s say that you have been charged with acting as a facilitator for a group of five sixth grade social studies teachers who are prioritizing their content standards. You and the teachers have received a “prioritization packet” created under supervision of the district board that explains the general idea of prioritization and a few criteria that should be used when evaluating the standards. During a summer time Teachers’ Conference meeting your job is to guide the teachers through a Nominal Group Planning brainstorming session with the goal of discovering other criteria that can be used to prioritize the standards.

• Beforehand: You reserve a conference room that is quiet and largely free from distractions. You make sure that there is adequate seating for you and the five teachers who will be participating in the process. Knowing that the process requires each participant to write, you place a stack of lined paper in the middle of the conference table along with more pencils than will be needed. In order to indicate their seats, you place an itinerary (complete with the steps used in the Nominal Group Planning session) in front of each participant’s chair. You are also aware that a group display is needed, and you have decided to use the conference room’s whiteboard. Erasers are accounted for and the markers are in working order, so you are confident that the meeting is ready to proceed.

• Step 1: The participants arrive and are seated. Because there are relatively few participants, only a single group is required.

• Step 2: Although the teachers are aware of the purpose of this meeting, it is still a good idea to have a discussion. You ask the teachers to volunteer reasons why prioritization will be valuable to their practice as well as any concerns they may have. The teachers respond immediately and are very vehement in their frustrations in covering all standards, and commiserate with one another over their mutual hardships. After a few minutes you sense that the discussion is slipping too far into negativity and attempt to turn the conversation constructive by asking the teachers how they think prioritization will help them to cover all standards. One response is “We will finally be able to downplay the topics that aren’t of much use to our students.” Another teacher adds “We will all be on the same page when teaching, rather than some classrooms concentrating on one topic for a long time when the others only spend a few days on it.” The other teachers voice their agreement, and you thank each for invoking the two central goals of the meeting. You ask the teachers to help you formulate a goal statement for this session, and after brief discussion it is decided. You write ‘What factors should we, acting and teaching as a group, consider when deciding what content students will need to succeed in school and life?’ at the top of the dry erase board. You ask the participants to keep this in mind during the session, and take a few minutes to review the steps of the process with the teachers, using the model on the itinerary as an aid.

• Step 3: You ask everyone to take a piece of paper and a pencil for the next step. Even though you have just explained the process, you review the actions that need to be taken during this step. Keeping in mind that the teachers may require some additional guidance due to their relative inexperience using this decision making method, you use some of the criteria already established by the district board as an example. You write ‘Readiness for School’ on the whiteboard and, as if thinking aloud to yourself, explain that this is an important criterion to consider because the lessons taught today become the foundation of lessons taught later on. You write ‘State Test’ and think aloud about how students (and teachers) must cover some material if only because it has a significant presence in standardized testing. Finally you write “Usefulness in Life” and think aloud how some topics may have a small presence on tests or later on in school, but have substantial significance or use in one’s life, such as learning basic economic principles. You ask the participants to come up with at least four or five other criteria that they think should be considered during prioritization, reminding them to keep their list to themselves.

• Step 4: After several quiet minutes you ask the participants to finish up, and then collect each list from each person yourself. You explain that you will combine these lists into a consolidated list of all suggestions and that the participants may take a short break while you accomplish this.

The five sheets you obtain read as below:

You realize that the anonymity afforded by working individually will be worthless if you just wrote each list in total, one after another, on the board. Instead, you randomize the responses as best you can, crossing out each response as you write it on the board. The result looks like the below:

• What can we teach easily?
• Are students interested?
• Cost of ancillary materials
• Student weaknesses
• District’s past performance
• How familiar are students with the topic already?
• Cohesiveness with other standards
• Interactivity
• Greatest room to improve on CAT
• Do we have any (more exciting) resources available besides the textbook?
• Is it easy to teach?
• Interest level
• Time/effort needed to teach adequately
• School needs
• Student interest
• School’s past performance
• Available internet resources
• What materials are available?
• What do we need to raise social studies scores?
• Time
• Teacher interest
• Alignment with school’s mission

• Step 5: Once the teachers have returned from their break, you remind them that in this step of the process group discussion will be used to combine criteria that are essentially the same idea. The first few are immediately called out and unanimously agreed upon. ‘What can we teach easily?’ and ‘Is it easy to teach’ are combined, as are ‘are students interested?’ and ‘student interest.’ Mindful of the difficulties that can arise later if the criteria are too general, you take the opportunity to ask the group exactly what “easy to teach” means. They are silent for a moment until one teacher responds, “Well, to me it means that I don’t have to fight with the kids about it. Something they like and will work at without me standing over them every second.” Another volunteers, “To me, easy means that I don’t have to do a lot of prep work. If it’s a pre-planned lesson that I can tweak on the fly, that makes my job much easier.” You point out that the criteria as it is currently phrased means different things to different people. The first teacher points out, “True, and what each of us got out of it is represented by criteria that are also already on the board – ‘student interest’ and ‘what materials are available.’” After some more discussion, the group elects to remove ‘easy to teach’ from the board, as it is too general, and can be represented by several other criteria listed.

After further consideration, consolidation, combination and elimination, the group agrees on the following consolidated list of criteria.

• Availability of ancillary materials
• School’s past performance
• Cohesiveness with other standards
• Teacher’s interest in topic
• Students’ interest in topic
• Time needed to teach thoroughly
• Alignment with school’s mission
• Students’ previous knowledge

• Step 6: You ask the teachers to once again take a blank piece of paper. You ask them to rank the consolidated list according to what they believe are the most important criteria for prioritizing state standards. Remind them that they should keep their responses confidential and that they shouldn’t split hairs over their ranking, as they will have opportunity to change their minds later.

• Step 7: After a few minutes you ask the participants to finish their ranking and gather the papers as before. You offer them another break as you pore over the results, which are below:

You assign one point to each top ranking criteria, two to the second, and so on. You then add up the points awarded to each criteria. Finally, you arrange the results into a group-ranked master list, which lists the lowest scoring criteria first and the highest last.

Show me the math used in step 7 and the group-ranked list that is the result

• Step 8: You transfer the group-ranked master list to the whiteboard for everyone to examine. You open the discussion by asking the teachers if they see a low-ranked criterion that should be higher. “I think the ancillary materials could be higher,” one participant ventures with some hesitation. “If a topic has a lot of quality materials available, then we can give the kids a wider choice in what learning tools work best for them. It would be a good way to reach out to more learning preferences and styles.” Heads nod in agreement and another participant quickly volunteers “I think ‘teacher’s interest’ is way too high. God knows some lessons are dull as dishwater for us, but its not really about us – it should be about the students.” A few other points are made, but after a few minutes the discussion peters out and you take it as your cue to move on.

• Step 9: You ask the teachers to take a piece of paper yet again. You explain that they are going to rank the master list one more time, this time for keeps. Assure them that they will have plenty of time to think and consider this time.

• Step 10: Once all participants are finished, you collect their papers for the final time and once again sum their scores and arrange the criteria into a final group-ranked master list. When finished, you notice that there is a strong break between the third ranked criterion and the fourth. Realizing that this indicates a strong preference for the first three criteria, you announce that they will be carried over to the standards prioritization session, while the others must be discarded.

Show me the math used in step 10 and the final group-ranked list that is the result

Using Results of Nominal Group Planning in a Decision Matrix[edit | edit source]

There are many possible methods for using criteria in a decision matrix to prioritize standards, ranging from simple grids to complex formulas and mathematically-determined weights. Because prioritization to external standards is almost always a group-based project, it is recommended that the simplest possible method should be used. This is so the manner in which prioritization decisions are made are transparent to all participants. If a complex statistical method is used, some participants may not trust that the results are representative of the “true” group consensus because they fail to understand the math. Such weighted calculations may be more accurate, but they do little good if some participants are not willing to accept them.

Steps for Conducting a group Decision Matrix

1. Gather all decision makers if possible. Participants can submit their worksheets electronically or via post, but having all physically present reinforces the idea that this is a group project and results in a group consensus.
2. Distribute a matrix worksheet, either on paper or electronically. The worksheet should already have any pre-existing criteria listed, along with the standards to be prioritized. Blank columns should be available so that participants can fill in the criteria that were created during the nominal group planning session.
3. Ask the participants to fill in the blank columns with the selected criteria.
4. Explain that participants will evaluate each standard by each criterion in terms of importance, where ‘1’ means that teaching the standard is very important in meeting the criterion and ‘5’ means that it is of little or no importance.
5. Allow participants time to fill in their matrices. Frequent breaks should be allowed so that participants do not “burn out” in the repetitive task of filing in the worksheet. If the sheer number of cells required seems guaranteed to burn out participants (and return inaccurate scores) you will need to break the process into multiple sessions.
6. Gather the finished worksheets. If the amount of data generated is extensive (i.e. it will take several hours to compute) it is advisable to adjourn the session and report the results at a later meeting.
7. Sum the results for each standard across all of the criteria and all participants. For instance if you have three participants each using two standards (3 x 2), you will be adding six digits when calculating the score for each standard.
8. Use the resulting score to classify each standard as high, medium, or low in importance. This is done by calculating the largest possible score for each standard and the lowest possible score. All possible scores will fall in the range between these two numbers. If a standard's score falls in the lowest third of the range, it should be classified as high priority. Scores that fall in the highest third of the number range are classified as low priority. Standards that fall in the middle third are of medium priority.

Then What?

As has already been mentioned, this method values group consensus over statistic accuracy. This is in part because participant buy-in is an important factor, but also because small gradations in priority are difficult to put into practice in the real world. Do you make a 90 minute lesson on a high-medium-high standard and 84 minutes on a high-medium-medium standard? You could plan it that way, but any instructor can tell you that problems arise and the successful teacher is a flexible teacher. The classification of high, medium, and low standards is not intended as input for an algorithm that will spit out the exact duration of a lesson. Instead, it should be used as a general reference when planning instruction.

Narrative example of Using Results of Nominal Group Planning in a Decision Matrix.[edit | edit source]

• Step 1. In the previous narrative example, the five social studies teachers have established three additional criteria:

• Students’ interest in topic
• Availability of ancillary materials
• School’s past performance

You announce that the group will now move from defining criteria to actually using the criteria to prioritize the state standards.

• Step 2. You pass out a worksheet that has eleven individual decision matrices, one for each major standard. Above each matrix is printed the accompanying standard and its indicators (sub-standards). In order to keep this example short, we will examine only one of these matrices. The blank worksheet matrix appears below:

Step 3. You request that each participant fill in the recently generated criteria. The result is below.

• Step 4. You explain to the participants that they will judge each standard by each criteria. For instance, if one feels that indicator 6.4.1 (Discuss the connections between geography and the development of city-states in the region of the Aegean Sea, including patterns of trade and commerce among Greek city-states and within the wider Mediterranean region) will be of utmost importance in later lessons or classes, then writing ‘1’ in the intersecting cell would be appropriate. If one feels that 6.4.1 is of no use in later schooling, then a score of ‘5’ might be applied.

• Step 5. You realize that completing all eleven matrices requires filling in over a thousand cells. You give the participants the option to complete them now, or to complete a portion of the sheet now, and finish at a later date. The teachers, eager to complete this task vote to finish it in the current meeting. You let them know that they are free to take a break at any time while completing the sheet. However, within a half-hour you notice that most of the teachers are wearily rubbing their eyes and shifting uncomfortably in their seats. You realize that the long day has taken its toll on the participants and would take its toll on the accuracy of their results as well. You ask the teachers if they would reconsider finishing the scoring at a later time, provided it was done via email. With some relief, they agree. You collect the worksheets and find that all participants have completed up to at least standard 6.4.

Each day for the next four days you email the teachers an Excel spreadsheet with two matrices and request that they fill it out and return it. Wary that some participants may save up their matrices to fill out at the last moment (and taking little care in their rush to complete it) you send a polite reminder when a teacher fails to return their file by the following evening.

• Step 6. Finally on the fifth day you have collected all of the complete matrices. Below are the five matrices for standard 6.4

• Step 7. You add the scores assigned to each criteria for each indicator by each participant. You are secretly glad that the participants decided to turn in their later answers via excel, but still calculated the first four matrices by hand.

Show me the math used when scoring each indicator

• Step 7 You establish the range of possible scores by finding the highest and the lowest possible scores.

Show me how find the highest and lowest possible scores

Once you've established the range of possible scores, you divide the range into thirds, rounding when necessary.

Show me how the range of high, medium, and low priority is established.

You categorize standards that fall in the lowest-scoring third as high priority, the middle third as medium priority, and the highest third as low priority.

Now What?

You report the results of the prioritization session to the decision makers. You tell them what they largely already know - that these classifications of standards as high, medium, and low priority are intended to be guidelines for their upcoming content planning sessions, and while they should definitely be referenced during that process, slavish adherence to the classifications will be detrimental to the planning, and morale on the whole.

Navigation[edit | edit source]