REPORT

 

Alignment Analysis of STATE F

Language Arts Standards and Assessments

Grades 5, 8, and 11

 

 

Norman L. Webb

Wisconsin Center for Education Research

Madison, Wisconsin

 

November 30, 2001

 

 

 

 

 

For the Technical Issues of Large-Scale Assessment Group of the Council of Chief State School Officers

Washington, D. C.

 

 


REPORT

 

Alignment Analysis of State F

Language Arts Standards and Assessments

Grades 5, 8, and 11

 

Norman L. Webb and Sharon O’Neal

November 30, 2001

 

 

 

 

 

 

 

This study is one of the alignment studies conducted and supported by the Technical Issues of Large-Scale Assessment (TILSA) group of the Council of Chief State School Officers. The alignment institute was held May 21-24, 2001. The report consists of a description of the four criteria used to judge the alignment between State F standards and assessments in language arts and the tables listing the results from the analysis of the coding by six reviewers. Assessment officials in State F also were given Excel files that contain supplementary information, the rating guidelines for the depth-of-knowledge levels assigned by reviewers to the objectives, the summary of coding by all reviewers, the objectives coded by each reviewer by item, and the summary tables.


Table of Contents

 

 

Introduction ..................................................................................................................... 1

 

Alignment Criteria Used for This Analysis ......................................................................... 3

     Categorical Concurrence ............................................................................................. 3

     Depth-of-Knowledge Consistency .............................................................................. 4

     Range-of-Knowledge Correspondence ....................................................................... 7

     Balance of Representation ........................................................................................... 7

     Source-of-Challenge Criterion ..................................................................................... 8

 

Findings ........................................................................................................................... 8

     Standards ................................................................................................................... 8

     Grade 5 Reading ....................................................................................................... 11

     Grade 8 Reading  ...................................................................................................... 12

     Grade 11 End of Instruction English II ....................................................................... 14

     Depth-of-Knowledge Reliability for the Three Grade Levels ...................................... 15

 

References ..................................................................................................................... 16

Tables

Reviewers’ Comments

Reviewers’ Codes by Item

 

 


Alignment Analysis of State F Language Arts Standards and Assessments

Grades 5, 8, and 11

 

Norman L. Webb and Sharon O’Neal

 

Introduction

 

The alignment of expectations for student learning with assessments for measuring students’ attainment of these expectations is an essential attribute for an effective standards-based education system. Alignment is defined as the degree to which expectations and assessments are in agreement and serve in conjunction with one another to guide an education system toward students learning what they are expected to know and do. As such, alignment is a quality of the relationship between expectations and assessments and not an attribute of any one of these two system components. Alignment describes the match between expectations and assessment that can be legitimately improved by changing either student expectations or the assessments. As a relationship between two or more system components, alignment is determined by using the multiple criteria described in detail in a National Institute of Science Education (NISE) research monograph, Criteria for Alignment of Expectations and Assessments in Mathematics and Science Education (Webb, 1997).

 

A four-day Alignment Analysis Institute was conducted May 20 through May 24, 2001. Six people, including state assessment consultants, content experts, and researchers, analyzed the agreement between the state’s language arts standards and assessments. The institute was coordinated by the Council of Chief State School Officers (CCSSO) as a function of the Technical Issues in Large-Scale Assessment (TILSA) collaborative among states. At the institute, as two teams analyzed the language arts standards and assessments from four states, two other teams of specialists analyzed mathematics standards and assessments from three of these states. In language arts, the alignment of standards and assessments was analyzed for three grade levels for three states and for four grade levels for one state.

 

           States use a variety of labels for identifying levels of expectations. For the purposes of this analysis, we have employed the convention of standards, goals, and objectives to describe three levels of expectations for what students are to know and do. Standard is the most general; it is divided into goals, which are further subdivided into objectives. It is assumed that all of the goals under a standard span the content knowledge expressed in the standard and all of the objectives under a goal span the content knowledge expressed in the goal.

 

A major outcome of the institute was to further develop a systematic process and analytic tools for judging the alignment between standards and assessments based on the criteria developed in conjunction with CCSSO and National Institute for Science Education (Webb, 1997). In addition to learning the process, reviewers were asked to provide suggestions for improving the process. Reviewers were trained to identify the depth-of-knowledge of objectives and assessment items. This training included reviewing the definitions of the four depth-of-knowledge levels and then reviewing examples of each. As the first step of the review process, the team of reviewers read each objective under each standard and reached consensus on the appropriate depth-of-knowledge level for that objective. In completing this step, the team of the reviewers gained increasing familiarity with the objectives themselves and the four depth-of-knowledge levels. Before independently coding the items from each assessment, the reviewers independently coded a sample of 5 to 10 items from the assessment instrument. They then compared what they had assigned as the depth-of-knowledge level and the content objective to each item. In this way, the reviewers calibrated their coding of the depth-of-knowledge level and the assigned objective. The overall process is not designed for the purpose of enabling reviewers to reach exact agreement. To derive the results from the analysis, the reviewers’ responses are averaged. Any variance among reviewers is considered legitimate, with the true depth-of-knowledge level for the item falling somewhere in between the two or more assigned values. Such variance could signify a lack of clarity in how the objectives were written, the robustness of an item that can legitimately correspond to more than one objective, and/or a depth of knowledge that falls in between two of the four defined levels. Reviewers were allowed to identify one assessment item as corresponding to up to three objectives—one primary hit (objective) and up to two secondary hits. However, reviewers could only code one depth-of-knowledge level to each assessment item even if the item corresponded to more than one objective.  

 

Reviewers were instructed to focus primarily on the alignment between the state standards and assessments. However, they were encouraged to offer their opinion on the quality of the standards or of the assessment activities/items by writing a note about the item. Reviewers could also indicate whether there was a source-of-challenge issue with the item—i.e., a problem with the item that might cause the student who knows the material to give a wrong answer, or enable someone who does not have the knowledge being tested to answer the item correctly. For example, a mathematics item that involves an excessive amount of reading may represent a source-of-challenge issue because the skill required to answer is more a reading skill than a mathematics skill.

 

 The results produced from the institute pertain only to the issue of agreement between the state standards and the state assessment instrument. This alignment analysis does not serve as external verification of the general quality of a state’s standards or assessments. The results of the alignment institute do provide the evaluations of content area experts, independent of any of the participating states, who are very familiar with state and national standards. The means of the reviewers’ coding were used to determine whether the alignment criteria were met. When reviewers did vary in their judgments, the means lessened the error that might result from any one reviewer’s finding. Standard deviations are reported, which give one indication of the variance among reviewers.

 

This report describes the results of an alignment study of standards and grade- level tests in language arts for grades 5, 8, and 11 in State F. The study addressed specific criteria related to the content agreement between the state standards and grade-level assessments. Four criteria received major attention: categorical concurrence, depth-of-knowledge consistency, range-of-knowledge correspondence, and balance of representation.

 

Alignment Criteria Used for This Analysis

 

This analysis judged the alignment between the standards and the assessment on the basis of four criteria. Information is also reported on the quality of items by identifying items with sources of challenge and other issues. For each alignment criterion, an acceptable level was defined by what would be required to assure that a student had met the standards.

 

Categorical Concurrence

 

           An important aspect of alignment between standards and assessments is whether both address the same content categories. The categorical-concurrence criterion provides a very general indication of alignment if both documents incorporate the same content. The criterion of categorical concurrence between standards and assessment is met if the same or consistent categories of content appear in both documents. This criterion was judged by determining whether the assessment included items measuring content from each standard. The analysis assumed that the assessment had to have at least six items measuring content from a standard in order for there to be an acceptable level of categorical concurrence between the standard and the assessment. The number of items, six, is based on estimating the number of items that could produce a reasonably reliable sub-scale for estimating students’ mastery of content on that sub-scale. Of course, many factors have to be considered in determining what a reasonable number is, including the reliability of the sub-scale, the mean score, and cutoff score for determining mastery. Using a procedure developed by Subkoviak (1988) and assuming that the cutoff score is the mean and that the reliability of one item is .1, it was estimated that six items would produce an agreement coefficient of at least .63. This indicates that about 63% of the group would be consistently classified as masters or nonmasters if two equivalent test administrations were employed. The agreement coefficient would increase if the cutoff score is increased to one standard deviation from the mean to .77 and, with a cutoff score of 1.5 standard deviations from the mean, to .88. Usually states do not report student results by standards or require students to achieve a specified cutoff score on sub-scales related to a standard. If a state did do this, then the state would seek a higher agreement coefficient than .63. Six items were assumed as a minimum for an assessment measuring content knowledge related to a standard, and as a basis for making some decisions about students’ knowledge of that standard. If the mean for six items is 3 and one standard deviation is one item, then a cutoff score set at 4 would produce an agreement coefficient of .77. Any fewer items with a mean of one-half of the items would require a cutoff that would only allow a student to miss one item. This would be a very stringent requirement, considering a reasonable standard error of measurement on the sub-scale.


Depth-of-Knowledge Consistency

 

Standards and assessments can be aligned not only on the category of content covered by each, but also on the basis of the complexity of knowledge required by each. Depth-of-knowledge consistency between standards and assessment indicates alignment if what is elicited from students on the assessment is as demanding cognitively as what students are expected to know and do as stated in the standards. For consistency to exist between the assessment and the standard, as judged in this analysis, at least 50% of the items corresponding to an objective had to be at or above the level of knowledge of the objective: 50%, a conservative cutoff point, is based on the assumption that a minimal passing score for any one standard of 50% or higher would require the student to successfully answer at least some items at or above the depth-of-knowledge level of the corresponding objectives. For example, assume an assessment included six items related to one standard and students were required to answer correctly four of those items to be judged proficient—i.e., 67% of the items. If three, 50%, of the six items were at or above the depth-of-knowledge level of the corresponding objectives, then for a student to achieve a proficient score would require the student to answer correctly at least one item at or above the depth-of-knowledge level of one objective. Some leeway was used in this analysis on this criterion. If a standard had between 40% to 50% of items at or above the

depth-of-knowledge levels of the objectives, then it was reported that the criterion was “weakly” met.

 

In language arts, four levels of depth-of-knowledge were used to judge both reading and writing objectives and assessment tasks. These levels are based on Wixson, Fisk, Dutro, & McDaniel (1999).

 

Reading

 

Reading Level 1

 

           Level 1 requires students to receive or recite facts or to use simple skills or abilities. Oral reading that does not include analysis of the text as well as basic comprehension of a text is included. Items require only a shallow understanding of text presented and often consist of verbatim recall from text, or simple understanding of a single word or phrase. Some examples that represent, but do not constitute all of, Level 1 performance are:

·        Support ideas by reference to details in the text.

·        Use a dictionary to find the meaning of words.

·        Identify figurative language in a reading passage.

 

Reading Level 2

 

           Level 2 includes the engagement of some mental processing beyond recalling or reproducing a response; it requires both comprehension and subsequent processing of text or portions of text. Inter-sentence analysis of inference is required. Some important concepts are covered but not in a complex way. Standards and items at this level may include words such as summarize, interpret, infer, classify, organize, collect, display, compare, and determine whether fact or opinion. Literal main ideas are stressed. A Level 2 assessment item may require students to apply skills and concepts that are covered in Level 1. Some examples that represent, but do not constitute all of, Level 2 performance are:

·        Use context cues to identify the meaning of unfamiliar words.

·        Predict a logical outcome based on information in a reading selection.

·        Identify and summarize the major events in a narrative.

 

 

Reading Level 3

 

Deep knowledge becomes a greater focus at Level 3. Students are encouraged to go beyond the text; however, they are still required to show understanding of the ideas in the text. Students may be encouraged to explain, generalize, or connect ideas. Standards and items at Level 3 involve reasoning and planning. Students must be able to support their thinking. Items may involve abstract theme identification, inference across an entire passage, or students’ application of prior knowledge. Items may also involve more superficial connections between texts. Some examples that represent, but do not constitute all of, Level 3 performance are:

·        Determine the author’s purpose and describe how it affects the interpretation of a reading selection.

·        Summarize information from multiple sources to address a specific topic.

·        Analyze and describe the characteristics of various types of literature.

 

 

Reading Level 4

 

Higher-order thinking is central and knowledge is deep at Level 4. The standard or assessment item at this level will probably be an extended activity, with extended time provided for completing it. The extended time period is not a distinguishing factor if the required work is only repetitive and does not require the application of significant conceptual understanding and higher-order thinking. Students take information from at least one passage of a text and are asked to apply this information to a new task. They may also be asked to develop hypotheses and perform complex analyses of the connections among texts. Some examples that represent, but do not constitute all of, Level 4 performance are:

·        Analyze and synthesize information from multiple sources.

·        Examine and explain alternative perspectives across a variety of sources.

·        Describe and illustrate how common themes are found across texts from different cultures.


Writing

 

Writing Level 1

 

Level 1 requires the student to write or recite simple facts. This writing or recitation does not include complex synthesis or analysis but basic ideas. The students are engaged in listing ideas or words, as in a brainstorming activity, prior to written composition, are engaged in a simple spelling or vocabulary assessment, or are asked to write simple sentences. Students are expected to write and speak using the conventions of Standard English. This includes using appropriate grammar, punctuation, capitalization, and spelling. Some examples that represent, but do not constitute all of, Level 1 performance are:

·        Use punctuation marks correctly.

·        Identify Standard English grammatical structures and refer to resources for correction.

 

Writing Level 2

 

Level 2 requires some mental processing. At this level, students are engaged in first-draft writing or brief extemporaneous speaking for a limited number of purposes and audiences. Students are expected to begin connecting ideas, using a simple organizational structure. For example, students may be engaged in note-taking, outlining, or simple summaries. Text may be limited to one paragraph. Students demonstrate a basic understanding and appropriate use of such reference materials as a dictionary, thesaurus, or web site. Some examples that represent, but do not constitute all of, Level 2 performance are:

·        Construct compound sentences.

·        Use simple organizational strategies to structure written work.

·        Write summaries that contain the main idea of the reading selection and pertinent details.

 

Writing Level 3

 

Level 3 requires some higher-level mental processing. Students are engaged in developing compositions that include multiple paragraphs. These compositions may include complex sentence structure and may be expected to demonstrate some synthesis and analysis. Students show awareness of their audience and purpose through focus, organization, and the use of appropriate compositional elements. The use of appropriate compositional elements includes such things as addressing chronological order in a narrative or including supporting facts and details in an informational report. At this stage, students are engaged in editing and revising to improve the quality of the composition. Some examples that represent, but do not constitute all of, Level 3 performance are:

·        Support ideas with details and examples.

·        Use voice appropriate to the purpose and audience.

·        Edit writing to produce a logical progression of ideas.

 

Writing Level 4

 

Higher-level thinking is central to Level 4. The standard at this level is a multi-paragraph composition that demonstrates synthesis and analysis of complex ideas or themes. There is evidence of a deep awareness of purpose and audience. For example, informational papers include hypotheses and supporting evidence. Students are expected to create compositions that demonstrate a distinct voice and that stimulate the reader or listener to consider new perspectives on the addressed ideas and themes. An example that represents, but does not constitute all of, Level 4 performance is:

·        Write an analysis of two selections, identifying the common theme and generating a purpose that is appropriate for both.

 

Range-of-Knowledge Correspondence

 

For standards and assessments to be aligned, the breadth of knowledge required on both should be comparable. The range-of-knowledge criterion is used to judge whether a comparable span of knowledge expected of students by a standard is the same as, or corresponds to, the span of knowledge that students need in order to correctly answer the assessment items/activities. The criterion for correspondence between span of knowledge for a standard and an assessment considers the number of objectives within the standard with one related assessment item/activity. Fifty percent of the objectives for a standard had to have at least one related assessment item in order for the alignment on this criterion to be judged acceptable. This level is based on the assumption that students’ knowledge should be tested on content from over half of the domain of knowledge for a standard. This assumes that each objective for a standard should be given equal weight. Depending on the balance in the distribution of items and the need to have a low number of items related to any one objective, the requirement that assessment items need to be related to more than 50% of the objectives for a standard increases the likelihood that students will have to demonstrate knowledge on more than one objective per standard to achieve a minimal passing score. As with the other criteria, a state may choose to make the acceptable level on this criterion more rigorous by requiring an assessment to include items related to a greater number of the objectives. However, any restriction on the number of items included on the test will place an upper limit on the number of objectives that can be assessed. Range-of-knowledge correspondence is more difficult to attain if the content expectations are partitioned among a greater number of standards and a large number of objectives. If 50% or more of the objectives for a standard had a corresponding assessment item, then the range-of-knowledge criterion was met. If between 40% to 50% of the objectives for a standard had a corresponding assessment item, the criterion was “weakly” met.

 

Balance of Representation

 

In addition to comparable depth and breadth of knowledge, aligned standards and assessments require that knowledge be distributed equally in both. The range-of-knowledge criterion only considers the number of objectives within a standard hit (a standard with a corresponding item); it does not take into consideration how the hits (or assessment items/activities) are distributed among these objectives. The balance-of-representation criterion is used to indicate the degree to which one objective is given more emphasis on the assessment than another. An index is used to judge the distribution of assessment items. This index only considers the objectives for a standard that have at least one hit—i.e., one related assessment item per objective. The index is computed by considering the difference in the proportion of objectives and the proportion of hits assigned to the objective. An index value of 1 signifies perfect balance and is obtained if the hits (corresponding items) related to a standard are equally distributed among the objectives for the given standard. Index values that approach 0 signify that a large proportion of the hits are on only one or two of all of the objectives hit. Depending on the number of objectives and the number of hits, a unimodal distribution (most items related to one objective and only one item related to each of the remaining objectives) has an index value of less than .5. A bimodal distribution has an index value of around .55 or .6. Index values of .7 or higher indicate that items/activities are distributed among all of the objectives at least to some degree (e.g., every objective has at least two items) and is used as the acceptable level on this criterion. Index values between .6 and .7 indicate the balance-of-representation criterion has only been “weakly” met.

 

Source-of-Challenge Criterion

 

            The source of challenge criterion is only used to identify items on which the major cognitive demand is inadvertently placed and is other than the targeted language arts skill, concept, or application. Cultural bias or specialized knowledge could be reasons for an item to have a source-of-challenge problem. Such item characteristics may result in some students not answering an assessment item, or answering an assessment item incorrectly, or at a lower level, even though they possess the understanding and skills being assessed.

 

Findings

 

Standards

                                   

           Six reviewers reached consensus on the depth-of-knowledge level of each objective for each standard. They used the same definitions of the depth-of-knowledge levels to rate each objective as they did to rate each assessment item. The distribution of items by levels gives some indication of how the reviewers judged the cognitive demands of the set of objectives in the standards. Some objectives can cover a very broad range of content and of what students would be expected to do to meet the objective. Also, students’ knowledge can vary greatly and what is demanding for one student can be  trivial for another student. Reviewers were asked to code the depth-of-knowledge level of an objective as the level that would be most commonly expected for a student at the grade level to satisfactorily demonstrate he or she had met the expectation of the objective. Because the six reviewers, all well familiar with the content area, reached consensus on the level for each objective, the assigned level to each objective is based on the judgment derived from discussion and agreement of experts.

 

Sometimes when reviewers tried to match an item to an objective within a standard, they did not find a match with any of the objectives, but felt that the item appropriately matched the standard. In these cases, the reviewers assigned the item to a “generic” objective under the standard, or a “generic” objective under a goal. In the tables under the “objective number,” the values represent the mean number of objectives available to the reviewers. If this number is more than the actual number of objectives under a standard, this indicates that one or more of the reviewers had to use a generic objective. The fractional values indicate how many of the reviewers used generic objectives—e.g., 6.17 indicates that one of six reviewers assigned at least one item to a generic objective beyond the 6 existing objectives. If a high percentage of the reviewers had to use generic variables, this indicates a possible problem in the coverage of content by the set of objectives. Each generic objective was assigned the depth-of-knowledge of the mode level assigned for the objectives within the goal or standard.

 

Grade 5 Reading

 

State F’s grade 5 reading standards consist of three standards:

I.                    Reading Process: The student will apply a wide range of strategies to comprehend, interpret, evaluate, and appreciate texts.

II.                 Responding to Text: Students read, construct meaning, and respond to a wide variety of literary forms.

III.               Information and Research: Students read widely to acquire knowledge, conduct research, and organize information.

 

The Reading Process standard is further specified by two goals—vocabulary and comprehension. The Reading Process standard had 8 objectives, 6 under vocabulary and 2 under comprehension. The Responding to Text standard had 12 objectives. The Information and Research standard had 4 objectives.

 

           Reviewers were able to find objectives corresponding to nearly all of the grade 5 items. Only one reviewer could not find a corresponding objective and goal for one item and coded this item as corresponding to a generic goal/objective for the Reading Process standard (Table LA5-1). As a result, the total number of objectives for the Reading Process standard was 8.17, rather than 8. Reviewers rated the majority of the objectives at a Level 3, designating that deep knowledge was expected to be a focus. Fifty-two percent of the 24 objectives were rated at a Level 3, 40% of the objectives were rated at a Level 2 (requiring comprehension and subsequent processing), and only 8% of the objectives were rated at a Level 1 (requiring simple skills). All three standards had nearly the same distribution of objectives by level.

 

Grade 8 Reading

                                   

           Grade 8 expectations had the same three standards as grade 5, but the nature of the objectives varied from those of grade 5. As for grade 5, only one of the six reviewers could not find a corresponding objective for one item and coded the item as corresponding to the generic objective for the Responding to Text standard. Thus, the total number of grade 8 objectives was 34.17, rather than 34 (Table LA8-1). Reviewers coded at least some objectives at each of the four depth-of-knowledge levels. Half of the 34 objectives were coded at a Level 3 (deep knowledge becomes a focus) and one third of the objectives were coded at a Level 2 (requiring comprehension and subsequent processing). Only two objectives (6%) were coded at a Level 1 (requiring simple skills). Four objectives (12%) were coded at a Level 4 (higher-order thinking).

  

Grade 11 End of Instruction English II

 

           Grade 11, English II, had five standards:

I.                    Reading Process: Students will apply a wide range of strategies to comprehend, interpret, evaluate, and appreciate texts.

II.                 Responding to Text: Students read, construct meaning, and respond to a wide variety of literary forms.

III.               Information and Research: Students read widely to acquire knowledge, conduct research, and organize information.

IV.              Grammar/Usage and Mechanics: Demonstrate appropriate practices in speaking and writing and correct use of standard English usage.

V.                 Literature

 

Standard I and Standard IV had two goals. All of the other standards had only one goal. One reviewer could not find a corresponding objective for an item related to Standard I and assigned it to a generic objective. Two reviewers did not find a corresponding objective and goal for an item related to Standard IV and assigned a generic goal and objective. Thus, the total number of coded objectives was 25.5, rather than 25 (Table LA 11-1). A higher percentage of the grade 11 objectives were coded at a lower depth-of-knowledge level than either the grade 5 or grade 8 objectives. Forty percent of the grade 11 objectives were coded at a Level 1 (requiring simple skills), 20% were coded at a Level 2 (requiring comprehension and subsequent processing), 28% at a Level 3 (deep knowledge becomes a focus), and 12% at a Level 4 (higher-order thinking).

 

Summary

 

           Grades 5 and 8 had a reasonably high percentage of objectives at Levels 3 and 4. This is in contrast to the grade 11 standards that had the majority of items at Level 1 or 2. The set of objectives at grades 5 and 8 covered the content measured on the assessment since nearly all of the reviewers found objectives corresponding to the assessment items. At grade 11, this was generally true, except for a small number of items. A high percentage of the reviewers found an objective corresponding to each assessment item. This indicates that the content assessed is included within the content of the objectives.


Grade 5 Reading

 

Categorical Concurrence

 

           On average, about half of the 40 assessment items were coded as corresponding to two objectives. The mean number of total hits was 62.33 (Table LA5-1). Three-quarters of the hits were for objectives under Standard II, Responding to Text. Only 5.33 of the total hits were for objectives under Standard III, Information and Research. This standard did not meet the acceptable level for categorical concurrence of at least six corresponding assessment items. The other two standards did have an acceptable level of items.

 

Depth-of-Knowledge Consistency

 

           On only one of the three grade 5 standards did the reviewers find an acceptable level on the depth-of-knowledge criterion of 50% or more of the corresponding items with a depth-of-knowledge level at or above the level of the corresponding objective (Table LA5-2). Standard I, Reading Process, had about 11 corresponding items of which 66% of them were judged to have a depth-of-knowledge equivalent or higher than the depth-of-knowledge level of the corresponding objective. In contrast, of the 45 items coded as related to Standard II, Responding to Text, 76% had a depth-of-knowledge below the corresponding objectives. Of the five items coded as related to Standard III, Information and Research, 64% had a depth-of-knowledge below the corresponding objectives. Reviewers coded the depth-of-knowledge levels of objectives under both Standards II and III as mainly Level 3 with about one-fourth Level 2. Nearly all of the items corresponding to these standards had a depth-of-knowledge rated as Level 1 or 2. For these two standards the items were not as demanding as expected by the standards.

 

Range-of-Knowledge Correspondence

 

One of the three standards, Standard II, Responding to Text, had an acceptable range-of-knowledge correspondence with the assessment instrument (Table LA5-3). On average across the six reviewers, about 10 of the 12 objectives had at least one corresponding item. Even though the other two standards had fewer objectives, only about 40% of their objectives had at least one corresponding item. This was sufficient to indicate that these two standards weakly but not fully met the acceptable level for range of knowledge. To fully have met this criterion, at least 50% of the objectives would need to have corresponding items.

 

Balance of Representation

 

One of the three standards had an acceptable level on the balance-of-representation criterion (Table LA5-3). The five items coded as relating to Standard III, Information and Research, were evenly distributed among the objectives with hits. This standard had a balance index value of .86, well above the acceptable level of .7. However, this finding has to be considered in light of the fact that the standard had only four objectives and weakly met the range-of-knowledge criterion. This indicates that, on the average, items corresponded to two of the four objectives, but the items were equally distributed among those two. The other grade 5 standards only weakly met the balance-of-representation criterion. For these standards and considering only the objectives with hits, one or two objectives had a slightly higher frequency of hits that the other objectives. For Standard I, Reading Process, reviewers coded a higher number of hits for objective 1B, “Read and comprehend both fiction and nonfiction that is appropriately designed for the second half of grade 5.” For Standard II, Responding to Text, reviewers coded a higher number of hits for two objectives, 2A6, “Locate and identify important details that support a main idea or describe a key concept,” and 2A8, “Make inferences and draw conclusions/generalizations from evidence presented in the text and the reader’s prior knowledge.”

 

Summary Grade 5 Reading

 

           Alignment between the grade 5 reading assessment and standards can be improved in a number of ways. On each of the four alignment criteria, only one or two of the standards met an acceptable level for the criterion. More items need to be included on the assessment instrument to measure content knowledge of Standard III, Information and Research. These additional items should be at a depth-of-knowledge level comparable to those of the objectives under the standard and address the objectives not already assessed. Some of the items now corresponding to Standard II, Responding to Text, need to be replaced with those at a higher depth-of-knowledge level or the objectives under this standard need to be revised so as not to be too demanding. Although there is an adequate number of items assessing content related to Standard I, Reading Process, at an appropriate depth-of-knowledge level, not enough of the objectives have corresponding items. Either existing items need to be replaced or additional items need to be included so that more of the objectives are assessment. In replacing items or adding items to the assessment instrument for Standards I and II, some attention needs to be given to the distribution of items. Currently, existing items tend to concentrate on a few of the objectives, rather than being more evenly distributed among the objectives.

 

Grade 8 Reading

 

Categorical Concurrence

 

           Reviewers coded fewer multiple hits at grade 8 than at grade 5. The mean total number of hits at grade 8 for the 40 assessment items was 53.83 (Table LA8-1). As in the case of grade 5, only two of the three standards had an acceptable number of items to meet the criterion for categorical concurrence. The Information and Research Standard, on average, had only 2.17 hits, far below the acceptable level of six.


Depth-of-Knowledge Consistency

 

           None of the three grade 8 standards had an acceptable level on the depth-of-knowledge criterion (Table LA8-2). Reviewers rated the depth-of-knowledge level of 59% of the items related to Standard I, Reading Process, as below the depth-of-knowledge level of the corresponding objective. Standard I only weakly met this criterion. For Standards II and III, reviewers rated the depth-of-knowledge level of 90% and 92%, respectively, of the items as below the depth-of-knowledge level of the corresponding objectives. On Standards II and III, half or more of their objectives were rated at a depth-of-knowledge Level of 3 or 4. With the exception of one reviewer on one or two items, all of the assessment items were coded with a depth-of-knowledge level of 1 or 2 below the demands of the majority of the objectives.

 

Range-of-Knowledge Correspondence

 

Standard II, Responding to Text, met the acceptable level for range-of-knowledge correspondence (Table LA8-3). Seventy percent of the 15 objectives had one or more corresponding items. Standard I, Reading Process, weakly met the acceptable level by having 48% of its 11 objectives with at least one corresponding assessment item. The number of items measuring knowledge related to Standard III, Information and Research, was insufficient to claim that the range of content in the eight objectives under the standard was adequately measured.

 

Balance of Representation

 

All of the three grade 8 standards met an acceptable level of a balance-of-representation index of .70 or higher (Table LA8-3). This means that those items corresponding to the objectives within a standard were evenly spread among the objectives hit. For Standards III to meet the balance-of-representation criterion is not very meaningful, since on the average reviewers only rated two or three items as

corresponding to that standard. Because these two or three items corresponded to different objectives, there was balance, but, as noted above, the range of knowledge was unacceptable.

 

Summary Grade 8 Reading

 

           Alignment between the standards and assessment on grade 8 reading can be improved by increasing the number of items assessing students’ knowledge related to Standard III, Information and Research. These items should be selected to correspond to a range of the objectives under Standard III. For all three grade 8 standards, at least some existing corresponding items, nearly all at Level of 1 or 2, need to be replaced by those at depth-of-knowledge Level 3. Some of the existing items corresponding to Standard I need to be replaced by items that assess objectives currently not measured. Within the current assessment instrument, the items are now distributed evenly among the objectives with corresponding items. Caution is needed to maintain this balance should current items be replaced or new items added.

 

Grade 11 End of Instruction English II

 

Categorical Concurrence

 

            On the average, reviewers coded one-third of the 100 assessment items as corresponding to more than one objective. The mean number of hits by the six reviewers was 134.33 (Table LA11-1). The largest number of hits, 79.17, was for Standard IV, Grammar/Usage and Mechanics. Two standards, Standard III, Information and Research and Standard V, Literature, did not have an acceptable number of hits and did not meet the criterion for categorical concurrence. The other three standards had a sufficient number of hits to meet the acceptable level of six items.

 

Depth-of-Knowledge Consistency

 

One grade 11 standard, Standard IV, Grammar/Usage and Mechanics, had an acceptable proportion of items, 50% or more, at a depth-of-knowledge as high or higher than the corresponding objectives (Table LA11-2). One other standard, Standard I, Reading Process, only weakly met this criterion by having 44% of the items with a depth-of-knowledge as high or higher than the corresponding objectives. Two other standards had nearly all of the related items, 87% for Standard II and 90% for Standard V, below the depth-of-knowledge level of the corresponding objectives. Reviewers did not code any items as related to Standard III, Information and Research, so there was an insufficient number of items to make a judgment on this criterion for that standard. Three-fourths and more of the objectives under Standards II and V were rated at Level 3 or 4, indicating that the students were expected to apply deep thinking or higher-order knowledge. Nearly all of the items rated as corresponding to Standards II and V were rated depth-of-knowledge Level of 1 or 2. Standard V received only three hits on the average, the minimal level for making any judgment on the values of the criteria. All of these items were rated by reviewers at a Level of 1 or 2. Only Standard IV, Grammar/Usage and Mechanics, had an acceptable level on depth-of-knowledge, primarily because most of its objectives were rated with a depth-of-knowledge of

Level 1, as were nearly all of the items.

 

Range-of-Knowledge Correspondence

 

Four of the five grade 11 standards met an acceptable level on the range-of-knowledge corresponding criterion (Table LA11-3). Only Standard III did not meet an acceptable level because none of the assessment items corresponded to objectives for the standard. For the other four standards, 75% or more of the objectives for each standard had at least one corresponding assessment item. One factor that contributed to the standards having an acceptable level on range-of-knowledge correspondence is the relatively low number of objectives for three of the four standards; each of these three standards had six or fewer objectives.

 

 

Balance-of-Representation

 

Three of the five grade 11 standards had an acceptable level on the balance of representation (Table LA11-3). Since Standard III, Information and Research, did not have any hits, it was inappropriate to judge whether the standard had met the balance-of- representation criterion. One standard, Standard II, Responding to Text, only weakly met the acceptable level for balance of representation with a balance index of .60. Four of the six reviewers coded over half of the items related to this standard as corresponding to one objective, 2A5, “support ideas, make inferences, and draw conclusions from evidence presented in the text.” The two other reviewers coded about the same number of items to objective 2A5, but one coded an equal number of items to 2A4, “Explain how the use of sound devices in poetry supports the subject and mood” and 2A2, “Summarize fiction/nonfiction by determining the main idea and supporting details.” Thus, reviewers coded items as relating to other objectives under Standard II, but coded a far greater number of items as corresponding to objective 2A5, so that the balance among items for this standard was low and in need of improvement.

 

Summary Grade 11 End of Instruction English II

                                   

           Alignment between the five standards and assessments for end of instruction English II was moderate. There were three main alignment issues. The assessment instrument did not include any items measuring students’ knowledge of content related to Standard III, Information and Research. Too few of the items on the assessment corresponded to Standard V, Literature. Only a sufficient proportion of items coded to Standard IV were at or above the depth-of-knowledge level of the corresponding objectives. The alignment could be improved on all of the other four standards by replacing existing items or increasing the number of items with items at a higher depth-of-knowledge level. With the exception of Standard III, the assessment had corresponding items relating to a sufficient number of the objectives within a standard and these were distributed appropriately among the objectives. However, the items corresponding to Standard II could be distributed more evenly among the objectives.                              

 

Depth-of-Knowledge Reliability for the Three Grade Levels

                                   

           Reliability among the six reviewers in coding the depth-of-knowledge levels for the items was .79 for grade 5, .60 for grade 8, and .85 for grade 11. This indicates there was reasonable agreement among the reviewers, except for grade 8, where reviewers disagreed more. Disagreement among the reviewers could signify that a greater number of items were on the borderline between levels.


 

References

 

Subkoviak, M. J. (1988). A practitioner’s guide to computation and interpretation of reliability indices for mastery tests. Journal of Educational Measurement, 25 (1), 47-55.

           

Webb, N. L. (1999). Alignment of science and mathematics standards and assessment in four states (Monograph No. 18). Madison: University of Wisconsin, Council of Chief State School Officers and National Institute for Science Education Research.

 

Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education (Monograph No. 6). Madison: Wisconsin Center for Education Research, Council of Chief State School Officers and National Institute for Science Education.

 

Wixson, K. K., Fisk, M. C., Dutro, E., & McDaniel, J. (1999). The alignment of state standards and assessments in elementary reading. A report commissioned by the National Research Council’s Committee on Title I Testing and Assessment. Ann Arbor: The University of Michigan.


 

 

 

Tables


Brief Explanation of Data in the Alignment Tables by Column

 

Tables LA5-1, LA8-1, LA11-1

 

            Goals #            Number of goals (second level) for each standard.

 

Objs #              Average number of objectives (third level) for reviewers. If the number is greater than the actual number in the standard, then at least one reviewer coded an item for the goal/objective, but did not find any objective in the goal that corresponded to the item.

 

Level                The Depth-of-Knowledge level coded by the reviewers for the objectives for each standard.

# of objs    

by Level           The number of objectives coded at each level

 

% w/in std

by Level           The percent of objectives coded at each level

 

Hits

Mean & SD     Mean and standard deviation number of items reviewers coded as corresponding to standard. The total is the total number of coded hits.

Cat. Conc.

Accept.            “Yes” indicates that the standard met the acceptable level for criterion. “Yes” if mean is six or more. “Weak” if mean is five to six. “No” if mean is less than five.

 

Tables LA5-2, LA8-2, LA11-2

 

First eight columns are the same as Table 1.

                                   

Level of Item

w.r.t. Stand      Mean percent and standard deviation of items coded as “under” the Depth-of-Knowledge level of the corresponding objective, as “at” (the same) the Depth-of-Knowledge level of the corresponding objective, and as “above” the Depth-of-Knowledge level of the corresponding objective.

            Depth-of-

            Know.

            Consistency

            Accept.            “Yes” indicates that 50% or more of the items were rated as “at” or

                                    “above” the Depth-of-Knowledge level of the corresponding

                                     objectives.

“Weak” indicates that 40% to 50% of the items were rated as “at” or “above” the Depth-of-Knowledge level of the corresponding objectives.

“No” indicates that less than 40% items were rated as “at” or “above” the Depth-of-Knowledge level of the corresponding objectives.

 

Tables LA5-3, LA8-3, LA11-3

 

            First eight columns are the same as Table 1 and 2.

 

            Range of Objectives

                                   

# Objs Hit        Average number and standard deviation of the objectives hit coded by reviewers.

 

% of Total        Average percent and standard deviation of the total objectives that had at least one item coded.

Range of

Know.

Accept.            “Yes” indicates that 50% or more of the objectives had at least one coded objective.

                        “Weak” indicates that 40% to 50% of the objectives had at least one coded objective.

                        “No” indicates that 40% or less of the objectives had at least one coded objective.

 

Balance Index

 

% Hits in

Std/Ttl Hits       Average and standard deviation of the percent of the items hit for a standard of total number of hits (see total under the Hits column).

 

Index                Average and standard deviation of the Balance Index

Note: BALANCE  INDEX     1 – (∑ │1/(O) – I (k) /(H )│)/2                                                                         k=1

                                    Where O    = Total number of objectives hit for the standard

                                                I (k) = Number of items hit corresponding to objective (k)

            H    = Total number of items hit for the standard

 

 

Bal.of Rep

Accept.            “Yes” indicates that the Balance Index was .7 or above (items evenly distributed among objectives).

                        “Weak” indicates that the Balance Index was .6 to .7 (a high percentage of items coded as corresponding to two or three objectives).

                        “No” indicates that the Balance Index was less than .6 (a high percentage of items coded as corresponding to one objective.)

 

 

 

Tables LA5-4, LA8-4, LA11-4

Summary if standard met the acceptable level for the four criteria by each standard.

 

 


Table LA5-1

Categorical Concurrence Between Standards and Assessment as Rated by Six Reviewers

State F Grade 5 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)   

                                                         

Standards

Level by Objective

Hits

Categorical

Concurr.

Acceptable

Title

Goals

#

Objs

#

Level

# of objs by Level

% w/in std by Level

Mean

S.D.

I. Reading Process

31

8.17

1

2

3

2

6

1

22

67

11

11.50

6.60

YES

II. Responding to Text

1

12

2

3

3

9

25

75

45.50

13.14

YES

Text Box: T- 4III. Information and Research

1

4

2

3

1

3

25

75

5.33

5.65

NO

      Total

5

24.17

1

2

3

  2

10

13

  8

40

52

62.33

17.55

 

 

                     1Includes one generic goal because coded items did not correspond to existing goals.


Table LA5-2

Depth-of-Knowledge Consistency Between Standards and Assessment

as Rated by Six Reviewers

State F Grade 5 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)

Standards

Level by Objective

Hits

Level of Item w.r.t. Standard

Depth-of-Knowledge Consistency

Acceptable

% Under

% At

% Above

Title

Goals

#

Objs

#

Level

# of objs by Level

% w/in std by Level

M

S.D.

M

S.D.

M

S.D.

M

S.D.

 

I. Reading Process

31

8.17

1

2

3

2

6

1

22

67

11

11.50

6.60

33

26

64

37

2

4

YES

II. Responding to Text

1

12

2

3

3

9

25

75

45.50

13.14

76

46

21

31

3

12

NO

III. Information and Research

1

4

2

3

1

3

25

75

5.33

5.65

64

38

36

28

0

0

NO

Text Box: T-5      Total

5

24.17

1

2

3

  2

10

13

  8

40

52

62.33

17.55

65

44

33

33

3

8

 

 

1Includes one generic goal because coded items did not correspond to existing goals.


 

 

Table LA5-3

Range-of-Knowledge Correspondence and Balance of Representation Between Standards and Assessment as Rated by Six Reviewers

State F Grade 5 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)

 

Standards

Level by Objective

Level 1=Recall

Level 4=Complex Reasoning

Hits

Range of Objectives

Range of Know. Accept.

Balance Index

(1 perfect-0 no balance)

Balance of Representation

Acceptable

# Objs Hit

% of Total

% Hits in Std/Ttl Hits

Index

 

Title

Goals

#

Objs

#

Level

# of objs

by

Level

% w/in std by Level

Mean

S.D.

Mean

S.D.

Mean

S.D.

 

Mean

S.D.

Mean

S.D.

 

Text Box: T- 6I. Reading Process

31

8.17

1

2

3

2

6

1

22

67

11

11.5

6.60

3.33

1.03

41

11

WEAK

18

7

.68

.13

WEAK

II. Responding to Text

1

12

2

3

3

9

25

75

45.5

13.14

9.67

.52

81

  4

YES

74

9

.61

.06

WEAK

III. Information and Research

1

4

2

3

1

3

25

75

5.33

5.65

1.67

1.21

42

30

WEAK

  9

8

.86

.21

YES

      Total

5

24.17

1

2

3

  2

10

13

  8

40

52

62.3

17.55

4.89

3.66

54

26

 

33

30

.71

.17

 

 

       1Includes one generic goal because coded items did not correspond to existing goals.


 

 

Table LA5-4

Summary of Attainment of Acceptable Alignment Level on Four Content Focus Criteria

State F Grade 5 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)

 

          Standards

 

Alignment Criteria

 

 

Categorical Concurrence

 

Depth-of- Knowledge Consistency

 

Range of Knowledge

 

Balance of Representation

I. Reading Process

YES

YES

WEAK

WEAK

II. Responding to Text

YES

NO

YES

WEAK

III. Information and Research

NO

NO

WEAK

YES

Text Box: T-7

 


Table LA8-1

Categorical Concurrence Between Standards and Assessment as Rated by Six Reviewers

State F Grade 8 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)    

                                                                    

Standards

Level by Objective

Hits

Categorical

Concurr.

Acceptable

Title

Goals

#

Objs

#

Level

# of objs by Level

% w/in std by Level

Mean

S.D.

I. Reading Process

2

11

1

2

3

4

1

6

3

1

9

55

27

9

11.67

3.14

YES

Text Box: T-8II. Responding to Text

1

15.171

2

3

4

  2

11

  2

13

73

13

39.50

10.03

YES

III. Information and Research

1

8

1

2

3

4

1

3

3

1

12

38

38

12

2.17

.75

NO

      Total

4

34.17

1

2

3

4

  2

11

17

  4

6

32

50

12

53.83

10.78

 

 

                   1 Includes one generic objective because coded items did not correspond to existing objectives.


 

Table LA8-2

Depth-of-Knowledge Consistency Between Standards and Assessment

as Rated by Six Reviewers

State F Grade 8 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)

Standards

Level by Objective

Hits

Level of Item w.r.t. Standard

Depth-of-Knowledge Consistency

Acceptable

% Under

% At

% Above

Title

Goals

#

Objs

#

Level

# of objs by Level

% w/in std by Level

M

S.D.

M

S.D.

M

S.D.

M

S.D.

I. Reading Process

2

11

1

2

3

4

1

6

3

1

9

55

27

9

11.67

3.14

59

41

38

33

0

0

WEAK

Text Box: T-9II. Responding to Text

1

15.171

2

3

4

  2

11

  2

13

73

13

39.50

10.03

90

48

10

22

0

0

NO

III. Information and Research

1

8

1

2

3

4

1

3

3

1

12

38

38

12

2.17

.75

92

42

0

0

8

14

NO

      Total

4

34.17

1

2

3

4

  2

11

17

  4

6

32

50

12

53.83

10.78

81

49

17

21

1

7

 

 

1 Includes one generic objective because coded items did not correspond to existing objectives.


 

Table LA8-3

Range-of-Knowledge Correspondence and Balance of Representation Between Standards and Assessment as Rated by Six Reviewers

State F Grade 8 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)

 

Standards

Level by Objective

Level 1=Recall

Level 4=Complex Reasoning

Hits

Range of Objectives

Range of Know. Accept.

Balance Index

(1 perfect-0 no balance)

Balance of Representation

Acceptable

# Objs Hit

% of Total

% Hits in Std/Ttl Hits

Index

Title

Goals

#

Objs

#

Level

# of objs

by

Level

% w/in std by Level

Mean

S.D.

Mean

S.D.

Mean

S.D.

 

Mean

S.D.

Mean

S.D.

Text Box: T-10I. Reading Process

2

11

1

2

3

4

1

6

3

1

9

55

27

9

11.67

3.14

5.33

.82

48

7

WEAK

22

5

71

10

YES

II. Responding to Text

1

15.171

2

3

4

  2

11

  2

13

73

13

39.50

10.03

10.67

1.51

70

8

YES

73

7

71

9

YES

III. Information and Research

1

8

1

2

3

4

1

3

3

1

12

38

38

12

2.17

.75

2.17

.75

27

9

NO

4

2

89

13

YES

      Total

4

34.17

1

2

3

4

  2

11

17

  4

6

32

50

12

53.83

10.78

6.06

3.75

49

20

 

33

30

77

13

 

 

  1Includes one generic objective because coded items did not correspond to existing objective      


 

 

Table LA8-4

Summary of Attainment of Acceptable Alignment Level on Four Content Focus Criteria

State F Grade 8 Language Arts

(Number of Assessment Items—40 Multiple Choice Items)

 

          Standards

 

Alignment Criteria

 

 

Categorical Concurrence

 

Depth-of- Knowledge Consistency

 

Range of Knowledge

 

Balance of Representation

I. Reading Process

YES

WEAK

WEAK

YES

II. Responding to Text

YES

NO

YES

YES

Text Box: T-11III. Information and Research

NO

NO

NO

YES

 

 


Table LA11-1

Categorical Concurrence Between Standards and Assessment as Rated by Six Reviewers

State F Grade 11 Language Arts

(Number of Assessment Items—100 Items)

 

                                                                    

Standards

Level by Objective

Hits

Categorical

Concurr.

Acceptable

Title

Goals

#

Objs

#

Level

# of objs by Level

% w/in std by Level

Mean

S.D.

I. Reading Process

2

2.172

2

2

100

8.33

1.63

YES

Text Box: T-12II. Responding to Text

1

6

2

3

4

1

4

1

17

66

17

29.83

5.91

YES

III. Information and Research

1

2

3

2

100

0

0

NO

IV. Grammar/Usage and      Mechanics

31

11.33

1

2

10

1

91

9

79.17

25.51

YES

V. Literature

1

4

2

3

4

1

1

2

25

25

50

3

0

NO

Total

8

25.5

1

2

3

4

10

5

7

3

40

20

28

12

134.33

24.43

 

 

                   1Includes one generic goal because coded items did not correspond to existing goals.

                   2Includes one generic objective because coded items did not correspond to existing objectives.


 

 

Table LA11-2

Depth-of-Knowledge Consistency Between Standards and Assessment

as Rated by Six Reviewers

State F Grade 11 Language Arts

(Number of Assessment Items—100 Items)

Standards

Level by Objective

Hits

Level of Item w.r.t. Standard

Depth-of-Knowledge Consistency

Acceptable

% Under

% At

% Above

Title

Goals

#

Objs

#

Level

# of objs by Level

% w/in std by Level

M

S.D.

M

S.D.

M

S.D.

M

S.D.

I. Reading Process

2

2.172

2

2

100

8.33

1.63

56

37

44

33

0

0

WEAK

II. Responding to Text

1

6

2

3

4

1

4

1

17

66

17

29.83

5.91

87

45

13

24

0

0

NO

Text Box: T- 13III. Information and Research

1

2

3

2

100

0

0

0

0

0

0

0

0

INSUFFICIENT #

IV. Grammar/Usage and      Mechanics

31

11.33

1

2

10

1

91

9

79.17

25.51

7

20

88

45

4

11

YES

V. Literature

1

4

2

3

4

1

1

2

25

25

50

3

0

90

49

10

20

0

0

NO

      Total

8

25.5

1

2

3

4

10

5

7

3

40

20

28

12

134.33

24.43

44

42

54

44

2

7

 

 

 1Includes one generic goal because coded items did not correspond to existing goals.

 2Includes one generic objective because coded items did not correspond to existing objectives.

 


Table LA11-3

Range-of-Knowledge Correspondence and Balance of Representation Between Standards and Assessment as Rated by Six Reviewers

State F Grade 11 Language Arts

(Number of Assessment Items—100 Items)

 

Standards

Level by Objective

Level 1=Recall

Level 4=Complex Reasoning

Hits

Range of Objectives

Range of Know. Accept.

Balance Index

(1 perfect-0 no balance)

Balance of Representation

Acceptable

# Objs Hit

% of Total

% Hits in Std/Ttl Hits

Index

Title

Goals

#

Objs

#

Level

# of objs

by

Level

% w/in std by Level

Mean

S.D.

Mean

S.D.

Mean

S.D.

 

Mean

S.D.

Mean

S.D.

I. Reading Process

2

2.172

2

2

100

8.33

1.63

2.17

.41

100

0

YES

8.33

1.63

.84

.14

YES

Text Box: T-14II. Responding to Text

1

6

2

3

4

1

4

1

17

66

17

29.83

5.91

5.17

.75

86

13

YES

29.83

5.91

.60

.08

WEAK

III. Information and Research

1

2

3

2

100

0

0

0

0

0

0

INSUFFICIENT #

0

0

-

-

INSUFFICIENT #

IV. Grammar/

Usage and      Mechanics

31

11.33

1

2

10

1

91

9

79.17

25.51

10.33

1.21

91

8

YES

79.17

25.5

.70

.07

YES

V. Literature

1

4

2

3

4

1

1

2

25

25

50

3

0

3.00

0

75

0

YES

17.00

3.29

.77

.14

YES

      Total

8

25.5

1

2

3

4

10

5

7

3

40

20

28

12

134.33

24.43

4.13

3.63

70

37

 

20

21

.73

.14

 

1Includes one generic goal because coded items did not correspond to existing goals.

2Includes one generic objective because coded items did not correspond to existing objectives.


 

 

Table LA11-4

Summary of Attainment of Acceptable Alignment Level on Four Content Focus Criteria

State F Grade 11 Language Arts

(Number of Assessment Items—100 Items)

 

          Standards

 

Alignment Criteria

 

 

Categorical Concurrence

 

Depth-of- Knowledge Consistency

 

Range of Knowledge

 

Balance of Representation

I. Reading Process

YES

WEAK

YES

YES

II. Responding to Text

YES

NO

YES

WEAK

Text Box: T- 15III. Information and Research

NO

INSUFFICIENT #

INSUFFICIENT #

INSUFFICIENT #

IV. Grammar/Usage and      Mechanics

YES

YES

YES

YES

V. Literature

NO

NO

YES

YES

 

 

 

 

 

 

 


 

 

 

 

Table LA-1

Reliability of Depth-of-Knowledge Levels Ratings for

State F Language Arts

 

Grade

Number of Reviewers

Number of Items

Alpha*

95% CI

Lower-Upper

5

6

38

.79

.66-.88

8

6

35

.60

.36-.78

11

6

88

.85

.79-.89

 

*  Average Measure Intraclass Correlation

 

Text Box: T-16 


 

 

 

 


Reviewers’ Comments

 

State F

Sources of Concern

Grade 5

 

Item #12

Not certain reader will be able to delete answer B as a possible choice.

 

Item #13

Answer “B” may also be correct

 

Item #15

Article is headed with “Today’s Health” in the upper left corner. This could be misleading.

 

Also of concern is the overall notion that titles reflect the main idea of a selection. Difficult to code to curriculum

 

Item #18

No single correct answer provided. All answers could be correct.

 

Item #20

Two people in our coding group felt there were two possible answers. “A” is a legitimate answer.

 

Item #30

The answer choices are poor. The distractors did not seem strong enough. The correct answer is not quite synonymous with the highlighted word.

 

Item #34

Why is it important to ask students about what is not important?  Good readers focus on important information.

 

Item #40

No correct answer provided.


State F

Sources of Concern

Grade 8

 

Issue#1

Overall the curriculum standards are somewhat problematic at this level. There is no reference to locating information, yet some of the test items ask the students to locate information using headings or other parts of a text or reference material.

 

Issue #2

The first text selection (about the stray cat and perseverance) is an excellent, well-written story. Some in our group felt the text was too easy for grade 8. Others felt is was appropriate and fine writing simply aided in the students’ ability to process the text easily.

 

Issue #3

Content of poem on page 28 is too elementary for grade 8.

 

Issue #4

Selection on page 26 is archaic. No speech teacher in 2001 would encourage their students to stand behind a lectern and avoid audiovisual materials/special effects.

 

In addition, some may feel the use of the word “sportsmanlike” is sexist.

 

Item #10

Not sure the answer reflects exactly what occurred. Was watching the cat considered and “experience?”

 

Item #16

Answer is not text dependent.

 

Why is it important to ask students about what is not important?  Good readers focus on important information.

 

Item #18

Why is it important to ask students about what is not important?  Good readers focus on important information.

 

Item #21

Answer is not in the text.

 

The distractors are of poor quality.

 

Answer may not be text dependent.

 

Item #22

Prompt does not match information to which reader is directed.

Item#24

Answer is not text dependent. Reader can answer this question is s/he knows the definition of “content.”

 

Item #26

Falling of leaves more literally means that the leaves are “going to bed.” Correct answer may need to be more precise.

 

Item #29

Correct answer not text dependent.

 

Items #35 and 36

These items seemed to be asking the reader to locate information rather than move the reader to reflect an understanding of cause and effect. This item written at a much lower level than the curriculum standard.

 

Item #39

It was difficult to match this item to the curriculum.


State F

Sources of Concern

End of Course

English II

 

Issue #1

Selections on pages 2-3 and 6-7 interesting and well-written. Most appropriate for English II EOC exam

 

Item #3

This item could not be coded to the curriculum standards unless the general reading standard (#5) was used.

 

Item #6

Why is it important to ask students about what is not important?  Good readers focus on important information.

 

Item #7

No correct answer provided. The inference is weak. Mr. Michner kept paying the shipping bills because he had already invested substantial amounts of money.

 

Item #15

Poor item; weak answer and distractors.

 

Item #18

Answers “C” and “D” are both correct given context of poem.

 

Item #19

Question focuses on what is NOT in poem. Why is it important to ask students about what is not important?  Good readers focus on important information.


STATE F Grade 5 LA Reviewers’ Comments by Item

Rater

Rater 1

Rater 2

Rater 3

Rater 4

Rater 5

Rater 6

Rater

Item #

 

 

 

 

 

 

Item #

12

Not coded

 

 

 

 

 

12

13

Not coded

Prior K-interferes

 

 

 

 

13

14

 

 

 

 

 

 

14

15

Not coded

Poor answer choices

 

 

 

 

15

18

Not coded

 

 

 

 

 

18

20

Not coded

 

 

 

 

 

20

29

Not coded

 

 

 

 

 

29

30

Not coded

 

 

 

 

 

30

34

Not coded

 

 

 

 

 

34

38

Not coded

 

 

 

 

 

38

Text Box: C-540

Not coded

 

The correct answer is not given

 

 

 

40

General

Items with two P's –

6(2A5, 2A8), 20(2A3, 2A9)

Items with two P's –

1(2A8, 1B1), 3(2A2, 1B1), 4(2A8, 1B1), 5(2A8, 1B1), 6(2A8, 1B1), 20(2A3, 2A9)

Items with two

P's –

31(3A3, 1B1), 39(2A6, 1B1), 3(1B1, 2A6)

Items with two P's –

1(2A8, 1B1) 14(1B1, 3A4. Item with only S - 9.

Items with two P's –

1(1B1, 2A8), 6(2A5, 2A8), 11(2A5, 3A3), 16(2A5, 2A8), 24(3A2, 3A3). Item with only S -  8, 17

Items with two P's - 1(2A8,1B1), 3(2A6,1B1), 4(2A5, 2A8), (2A6,2A8),

6(2A, 2A8),7(2A8,2A11),

13(2A6, 1B1),

14(2A6, 2A11),

16(2A5, 2A8,1B1),

17(2A6, 2A8),

18(2A5, 2A8),

19(1A3, 2A9, 1B1), 22(2A6, 1B1),

23(2A6, 1B1),

24(2A6, 1B1),

25(2A6, 1B1),

26(2A6, 1B1),

27(2A8, 2A11),

33(2A5, 2A8),

35(2A8, 2A11),

36(2A8, 2A11),

37(2A5, 2A8)

General


STATE F Grade 8 LA Reviewers’ Comments by Item

Rater

Rater 1

Rater 2

Rater 3

Rater 4

Rater 5

Rater 6

Rater

Item #

 

 

 

 

 

 

Item #

16

Answer is not text dependent

Does not fit the codes

 

 

 

 

16

17

 

 

 

 

Given in text

 

17

18

Why is it important to ask students about what is not important? Good readers focus on important information

 

 

 

 

 

18

20

 

Not text dependent

 

 

 

 

20

Text Box: C-621

Answer is not in the text

 

 

 

 

 

21

22

 

 

Missing item

 

Item answers are poor fit to prompt

 

22

24

Answer is not text dependent

Missing item

 

 

There is no content info to match

 

24

28

 

 

 

Missing item

 

 

28

34

 

Missing item

 

 

 

 

34

39

 

 

 

 

Very uncertain category match

 

39

General

 

Item with two P's - #33(1A3, 2A12). Item with only S - # 17

Item with two P's - # 24(1B2, 2A11)

Item with two P's - # 29(1A6, 2A9)

 

 

General

 


STATE F Grade 11 LA Reviewers’ Comments by Item

Rater

Rater 1

Rater 2

Rater 3

Rater 4

Rater 5

Rater 6

Rater

Item #

 

 

 

 

 

 

Item #

3

This item could not be coded to the curriculum standards unless the general reading standard was used

No category - locate info.

 

 

 

 

3

7

 

Not really right answer

 

 

 

 

7

12

Missing item

 

 

 

 

 

12

Text Box: C-713

Missing item

 

 

 

 

 

13

14

Missing item

 

Missing item

 

 

 

14

15

Missing item

 

 

 

 

 

15

16

Missing item

 

 

 

 

 

16

18

Answer C and D are both correct given context of poem

 

 

 

 

 

18

19

Question focuses on what is NOT in poem. Why is it important to ask students about what is not important? Good readers focus on important information.

 

 

 

 

 

19

20

 

 

 

Missing item

 

 

20

30

Have to code poetry questions pp. 16-17. Standards written at higher level

 

 

 

Not certain of correct answer

 

30

31

Have to code poetry questions pp. 16-17. Standards written at higher level

Missing item

 

 

 

 

31


 

STATE F Grade 11 LA Reviewers’ Comments by Item (continued)

Rater

Rater 1

Rater 2

Rater 3

Rater 4

Rater 5

Rater 6

Rater

Item #

 

 

 

 

 

 

Item #

32

Have to code poetry questions pp. 16-17. Standards written at higher level. Difficult to code poetry questions pp. 16-17 standards written at higher level

 

 

 

 

 

32

33

Difficult to code poetry questions pp. 16-17 standards written at higher level

 

 

 

 

 

33

34

Difficult to code poetry questions pp. 16-17 standards written at higher level

 

 

 

 

 

34

Text Box: C-835

Missing item

 

 

 

 

Missing item

35

36

Difficult to code poetry questions pp. 16-17 standards written at higher level

Missing item

 

 

 

 

36

37

 

 

 

Missing item

 

 

37

40

Difficult to code. No pre-reading skills addressed in standards

 

 

 

Two correct answers (A & C)

 

40

47

Much lower than standard

 

 

Missing item

 

 

47

50

Much lower than standard as written

 

 

 

 

 

50

71

 

 

 

 

 

Missing item

71

85

No primary hit

 

 

 

 

 

85

86

 

 

 

 

Answer is in prompt

 

86

90

No primary hit

 

 

 

 

 

90

Rater

Rater 1

Rater 2

Rater 3

Rater 4

Rater 5

Rater 6

Rater

Item #

 

 

 

 

 

 

Item #

93

No primary hit

 

 

 

 

 

93

97

No primary hit

 

 

 

 

 

97

 

Item with two P's – 32. Items with just S - 85, 90, 93, 97

Items with only S - 37, 75, 93, 97. Items with two P's - 6(2A5, Standard Genric), 30(5A2, 2A5), 38(1B1, Stnd Generic)

Item with two P's - 13(2A5,5A1)

Items with two P's - 19(2A4,A2), 39(1A1,4B11), 46(5A2, 2A3)

 

 

General

Text Box: C-9


 

 

 

 

 

 

Reviewers’ Codes by Item


State F Grade 5 Reading Item by Reviewer Depth-of-Knowledge and Objective

 

State F Reading Grade 5 Item codes by Reviewers

 

 

 

 

 

 

 

 

 

 

 

 

#IT

A1

DSTATE F

 Obj

A2

DSTATE F

Obj

A3

DOK

Obj

A4

DOK

Obj

A5

DOK

Obj

A6

DOK

Obj

1

2

2A6,2A8

2

1B1,2A6,2A8

2

1B1,2A6,2A8

2

1B1,2A6,2A8

2

1B1,2A6,2A8

1

1B1,2A6,2A8

2

2

1A1,2A8

2

1A1,1B1,2A8

2

1A2,1B1,2A8

2

1A1,2A8

2

1A1,2A8

2

1A1,1B1,2A8

3

2

2A2,2A6

2

1B1,2A2,2A6

2

1B1,2A1,2A6

3

2A1

3

2A2,2A6

2

1B1,2A2,2A6

4

2

2A5,2A8

2

1B1,2A5,2A8

2

2A8

2

2A8

2

2A5,2A8

3

1B1,2A5,2A8

5

2

2A6

3

1B1,2A6,2A8

2

2A6,2A8

3

2A6

2

2A5,2A8

3

1B1,2A6,2A8

6

2

2A5,2A8

2

1B1,2A1,2A8

2

2A1,2A5,2A8

2

2A5

2

2A5,2A8

2

1B1,2A5,2A8

Text Box: I-17

2

2A5,2A8,2A11

3

2A5,2A8,2A11

2

2A11

2

2A11

2

2A5,2A11

2

2A8,2A11

8

2

2A6

2

2A8,2A10

2

2A8

2

2A10

2

2A10

2

2A6,2A8

9

2

1A1,2A10

3

1A1,2A8,2A10

2

1GEN,2A10

2

1A1

2

2A2

3

2A8,2A10

10

1

2A1,2A6

1

3A2

2

1B1,2A2

1

1B1

2

3A2

1

1B1,2A6

11

1

2A6

2

3A3

1

2A7

2

1B1,2A8

2

2A5,3A3

1

1B1,2A6

12

3

2A8

3

3A3

2

2A7,2A8

2

2A6

2

3A3

1

1B1,2A6,2A8

13

3

2A8

2

2A8,3A3

3

2A8

2

2A8

2

2A5

2

1B1,2A6

14

2

3A4

2

3A3,3A4

2

1A1,2A8,3A4

2

1B1,3A4

2

3A4

2

2A1,2A6,2A11

15

2

2A5

2

3A3

2

2A5,2A8

2

2A8

2

2A5

2

2A5,2A8

16

2

2A5

2

2A5,2A7

2

2A5

2

2A5,2A9

2

2A5,2A8

2

1B1,2A5,2A8

17

2

2A8

2

2A8,2A9

2

2A10

3

2A9

2

1A3

2

2A6,2A8

18

UN

 

2

2A5,2A8

2

2A5

2

2A5

2

2A5

2

2A5,2A8


State F Grade 5 Reading Item by Reviewer Depth-of-Knowledge and Objective (continued)

 

19

1

1A3

1

1A3

1

1A3

1

1A3

1

1A3

2

1A1,1A3,1B1,2A9

20

1

2A2,2A9

2

2A3,2A9

1

2A3

1

2A9

2

2A9

2

2A3

21

1

2A6

1

3A3

1

1B1,2A6

1

2A6

2

3A3

1

1B1,2A6

22

2

2A6,2A8

1

3A3

1

1B1,2A6

3

2A6

1

3A3

1

1B1,2A6

23

2

2A8

1

3A3

1

1B1,2A6

1

2A8

2

2A8

1

1B1,2A6

24

2

2A6,2A8

1

3A3

1

2A6

1

2A6

2

3A2,3A3

1

1B1,2A6

25

2

2A11

1

1B1,3A3

1

2A8

2

1B2

2

2A11

1

1B1,2A6,2A8,2A11

26

2

2A2

1

1B1

1

2A6

1

2A6

2

1A3

1

1B1,2A6,2A8

27

2

2A8

2

2A8,3A3

2

2A8

2

2A7,2A11

2

2A8

2

2A8,2A11

28

2

2A8

2

3A3

2

2A8

2

2A7

2

2A12

2

2A6,2A8

Text Box: I-229

UN

 

1

GEN

1

3A3

UN

 

1

GEN

2

1A1

30

2

1A1

2

1A1,2A8

2

1A1

2

1A1

1

1A1

2

2A8

31

1

2A2

2

2A3

1

1B1,3A3

2

2A3

1

2A3

2

2A3,2A4

32

1

2A6

1

2A2

1

2A6

1

2A6

2

2A6

2

2A6

33

2

2A5

2

2A5

2

2A5

2

2A5

2

2A5

2

2A5,2A8,2A11

34

2

2A6

2

2A6

3

2A6

2

2A6

2

2A6

2

2A6,2A8

35

2

2A11

2

2A8,2A11

2

2A8,2A11

2

2A11

2

2A1,2A11

2

2A6,2A8,2A11

36

2

2A8

2

2A8,2A11

2

2A11

2

2A11

2

2A5

2

2A6,2A8,2A11

37

2

2A11

2

2A8,2A11

2

2A11

2

2A11

2

2A11

2

2A5,2A8,2A11

38

2

2A6

2