SmallSample Longitudinal Study of Group Problem Solving and Attitudes of IMP Students and Traditional Mathematics Students—Grade 9 to Grade 11
Grade 11 Report of Class of 1998 Study An Exploratory Study Norman Webb, Maritza Dowling, and Isabel Romero Wisconsin Center for Education Research, University of WisconsinMadison TABLE OF CONTENTS Sample for Small Group Problem Solving Major Findings from Year 3 Analyses Appendix A: Performance Assessment Activities Appendix B: Mathematics Problem Solving and Discourse Coding Criteria Appendix C: Analysis of Students' Mathematical Knowledge Exhibited in Solving Two Performance Assessment Activities
SmallSample Longitudinal Study of Group Problem Solving and Attitudes of IMP Students and Traditional Mathematics Students—Grade 9 to Grade 11
Grade 11 Report of Class of 1998 Study An Exploratory Study Norman Webb, Maritza Dowling, and Isabel Romero Wisconsin Center for Education Research, University of WisconsinMadison This is the third and final brief report of the longitudinal study designed to monitor the growth in students' ability to solve problems while working in small groups and their development in using communication skills in doing mathematics. The study was one of several conducted by the Interactive Mathematics Program Evaluation Project. The available resources could not support a rigorous study to answer all of the questions associated with the complex nature of problem solving and group communications. This study provides some information and insights into how students working in groups perform while solving mathematics problems. In two California high schools, we observed students solving problems four times during their high school careers—at the beginning of grade 9, at the end of grade 9, at the end of grade 10, and at the end of grade 11. Students enrolled in both the Interactive Mathematics Program (IMP) and students enrolled in the more traditional curriculum participated in the study. We began the study in fall 1994 and continued the study through spring 1997. The study was not designed to nor did it provide enough information or data for a comprehensive description of all that students gained from their mathematics programs. Students being able to communicate effectively and solve problems in mathematics is receiving more and more attention from professional groups such as the National Council of Teachers of Mathematics, higher education, and employers. This study was designed to observe how students develop in these important areas during their high school experiences. The Interactive Mathematics Program is designed, among other goals, to have a broad range of students become capable problem solvers and communicators of mathematics who are very comfortable in using mathematics in their daily lives. The more traditional algebra, geometry, and precalculus curriculum also is being changed by teachers incorporating more opportunities for students to work in groups and to solve more extended problems. The purpose of this study is not to judge either IMP or the more traditional curricula—the study is not extensive enough to do this—but to document how students grow in their facility to communicate mathematically and to use mathematics to solve problems in different learning experiences.
Sample for Small Group Problem Solving In fall 1994, 48 students (25 IMP Year 1 students and 19 Algebra I and 4 Math A) were randomly selected from their respective groups at two California high schools. One group included the 159 students who were enrolled in IMP Year 1. The second group was made up of students enrolled in the more traditional mathematics programs, Algebra 1 and Math A. Over the three year period, from 199495 school year to the 199697 school year, some of the students participating in the study transferred schools or changed mathematics program within their school. In general, these students were replaced by other students with similar gender, ethnicity, and mathematics program. There also was attrition in the study due to students being absent or having other responsibilities on the days testing was done. In spring 1996, year two of the study, followup data were collected on nearly the same cohort enrolled in the mathematics programs, schools, and courses as in the first year of the study. A total of 42 students participated, 22 students from School A and 20 students from School B. Twentyone students were enrolled in IMP Year 2 (50 percent of the students tested that year), 15 students were enrolled in Geometry (36 percent), and the remaining 6 students (14 percent) were taking Algebra 1. Females slightly outnumbered males (55 percent females compared to 45 percent males). In May 1997, the third and final year of the study, data were collected by observing 37 students as they worked in small groups of three or four students. Of these students, 22 students (16 IMP students and 6 Advanced Algebra/Algebra II students) had participated in the study in the previous year, 199596. Only 14 of the original 48 students participated in testing all three years. In 1998, 19 students were enrolled in IMP Year 3, 10 from School A and 9 from School B, and 18 were enrolled in either Advanced Algebra (6 from School A), Algebra II (6 from School B), and Geometry (6, three from each of the schools). In the total group there were 18 female students and 19 male students (Table 1). The secondyear algebra course at School A was entitled Advanced Algebra and at School B, Algebra II. IMP and Advanced Algebra/Algebra II mathematics students who were tested in 199697 were, in general, comparable in mathematics achievement prior to entering high school. However, there was some variation in general grade point average (through grade 11) and mathematics grade point average (through grade 11; Table 2). IMP Year 3 students and Advanced Algebra students from School A who participated in the study had attained comparable mean scores on the grade 7 Comprehensive Test of Basic Skills (CTBS), 78.71 and 76.5, respectively. Both of these groups had a higher mean score than those who were enrolled in Geometry. The Advanced Algebra students had a higher overall grade point average than IMP Year 3 students, 3.64 compared to 3.04, but there was less of a difference in the mathematics grade point averages of the two groups, 3.12 for the Advanced Algebra group and 2.69 for the IMP Year 3 group. For School B, the IMP Year 3 students and the Algebra II students differed little. The nine IMP Year 3 students had a mean score of 71.67 on the Iowa Test of Basic Skills taken in Grade 8 compared with the six Algebra II students who had a mean of 68.5. The two groups varied little on the mean overall grade point average, 3.19 for IMP Year 3 students and 3.10 for Algebra II students, and on the mean mathematics grade point average, 3.18 for IMP Year 3 students and 2.92 for the Algebra II students. The three students from School B who were enrolled in Geometry were comparable on the average with the students from the other two courses from School B. Performance on these concomitant variables indicate that the groups tested were in general comparable to each other on these factors, dispelling any argument that one group was of higher ability than another group. Students were video taped as they worked in groups of three or four solving two performance assessment activities during one class period. Students were tested at the end of May 1997, nearly at the end of their third year of high school. They were given up to 20 minutes to work each of the two problems. Groups that had not completed a problem after 20 minutes were asked to record what they had accomplished. If time permitted after they had completed the two problems, students were asked a few questions about their mathematics course work during that school year. Two performance assessment activities prepared for the Wisconsin Student Assessment System, under the auspices of the Wisconsin Department of Public Instruction, were given to the groups of students to solve (Appendix A). Each activity required students to construct a response by solving a multistep mathematics problem, generalizing the results obtained, and writing an explanation of the reasoning and procedures used. One activity, Roll 'Em and Win, required students to consider the product of rolling two dice and deciding what criteria to set to determine a win, rolling a product less than 13 or less than 18. The second activity, Street Wise, required students to draw a scale drawing of two parallel streets, one street perpendicular to these, and one street intersecting the three others forming right triangles. They then were asked to find the distance along the hypotenuse using two methods. Possible methods students could use included measuring the scale drawing with a ruler, using the Pythagorean theorem, using trigonometry, and using the distance formula to calculate the distance between two points on a coordinate system. Students at School B did not have rulers available to them and had to derive some other means for measuring or estimating the distances. Students at School A used rulers. Students' work on each activity was scored using a sixpoint rubric (advanced response, proficient, nearly proficient, minimal, attempted, and not scorable). A proficient or advanced score on one activity, Roll 'Em and Win, indicates that the students demonstrated skills in solving problems, reasoning, developing and testing conjectures, identifying all possibilities, computing probabilities, and writing a clear and correct argument. A proficient or advanced score on the second activity, Street Wise, indicates that the students drew an accurate scale drawing, accurately marked the streets and points of intersection, computed the distance using two distinct methods, and adequately explained their methods and the results. Work by each group on each activity was scored independently by two raters. If these raters disagreed, the group’s work was scored by a third rater. In the latter case, the group was assigned the modal score. The two raters had exact agreement on 19 of the 24 scores (79%). The five times the raters disagreed, all in scoring groups’ work on Street Wise, the raters only disagreed by one score. In four of these disagreements, one rater gave a score of 2 and the other gave a score of 3; in the fifth, one rater gave a score of 3; and the other gave a score of 4. In addition, a qualitative analysis was performed on all of the groups' work to identify the problemsolving approaches used by each group and the errors they made. The video tapes and transcriptions of students' protocols were rated and analyzed on six interaction variables—problem solving, information, questioning, reasoning, group collaboration, and group cooperation (Appendix B). Major Findings from Year 3 Analyses Group scores for IMP Year 3 students, Advanced Algebra/Algebra II students, and Geometry students on the two performance assessment activities are summarized in Table 3 and Figure 1. Groups of IMP students performed proficiently on solving complex performance assessment activities. Four of six IMP groups performed proficiently on at least one of two performance assessment activities. One group of IMP students performed proficiently on both performance assessment activities. One group of IMP students performed proficiently on one activity and nearly proficient (conceptually correct with a minor flaw) on the second. Two IMP groups, both from School B, did not demonstrate strong understanding of the mathematics required on either of the two performance assessment activities. Three of the six groups of IMP students, all from School A, demonstrated proficient work on the Roll ‘Em and Win activity that requires knowledge of probability to solveone topic given some emphasis in the IMP curriculum. Two of the six groups of IMP students, one from School A and one from School B, demonstrated proficient work on the Street Wise activity. Street Wise could be solved using mathematical concepts and procedures included in most high school mathematics curricula. A higher percentage of groups of IMP students than groups of Advanced Algebra/Algebra II students performed proficiently on solving complex performance assessment activities. Whereas only one of four groups of Advanced Algebra/Algebra II students (25%) achieved a proficient score on at least one of the two performance assessment activities, four of the six groups of IMP students (67%) did. This finding cannot be generalized beyond the groups that were tested because of the small sample size. (The finding replicates one from the previous year, 199596. In this study only one of four groups of those from the traditional grade 10 mathematics course, Geometry, performed proficiently on either activity tested.) This one group performed proficiently on both activities administered. In contrast, five of the six IMP groups tested in 199596 achieved a proficient score on at least one activity. These findings from this study of group problem solving are consistent with those of other studies (Dowling & Webb, 1997; Webb & Dowling, 1997) that indicate IMP students are able to solve complex performance assessment activities with a higher frequency than students enrolled in the more traditional mathematics courses. The finding from this small study suggests the results from these other studies are applicable to the performance of groups of students. Groups of students enrolled in all the courses did not demonstrate strong group communication skills conducive to solving complex problems. Of the 12 groups of students tested, only four of the groups, two IMP groups and one from each of the other two courses, exhibited complete collaboration while solving a problem (Appendix B, Tables B3 & B4). Each of these groups exhibited this level of collaboration only when solving one of the two activities. Thus, students’ collaboration may be problemdependent and not a consistent characteristic of any one group of students observed over two problems. Complete collaboration indicated that the students (a) were generally able to maintain a dialogue that built coherently on each others’ ideas to promote collective understanding, (b) were supportive of ideas presented by others, and (c) were willing to challenge ideas presented by others when appropriate. Criteria used for analyzing group communication attributes are given in Appendix B. At grade 11, more groups did show complete collaboration than they had the previous year. In the 1996 testing, two groups of IMP Year 2 students exhibited complete collaboration, the only groups judged to do this from the 12 groups tested at the two high schools. In general, when working in groups, students interacted with each other, but they did not interact with each other at a level of discourse that would lead to effective problem solving. Students rarely questioned the work or an idea of another student, but tended to listen to the idea and then continued to work on his/her own. This resulted in students failing to consider all of the conditions of the problem, to consider relevant questions, and to challenge wrong assumptions about the problem. As a result of poor group communication, students in most groups were unable to organize the information given in the problem and make productive progress toward a solution. Both IMP students and Advanced Algebra/Algebra II students demonstrated knowledge of a variety of mathematical topics including probability, geometry, and trigonometry. Analysis of students’ solvingproblem approaches to the two activities are given in Appendix C. From the analysis of the first problem, Roll ’Em and Win, we can observe that, in almost all the features of mathematical knowledge that can be assessed using this task, IMP groups performed better than nonIMP groups. Also, the differences clearly favored School A. This suggests that there is an interaction between the school and the curriculum. In regard to knowledge about combinatorics, four IMP groups, all of them from School A, versus only one Advanced Algebra group from School A, were able to use the most reliable methods for generating the total number of possibilities for rolling two dice (rug table and tree diagram). A conceptual error of counting combinations instead of permutations was made by three groups from School B: one IMP, one Algebra II, and one Geometry. All of the IMP groups referred to probability to discuss the arguments given by the two parts of the club (two nonIMP groups did not mention probability). Three IMP groups from School A versus two Advanced Algebra/Algebra II groups, one from School A and one from School B, correctly calculated the probability for the product of the dice to be under 13. One more IMP group from School B made just a computational error in calculating this probability. Concerning the discussion of each part of the club’s claim, three IMP groups from School A, compared to one Algebra II group from School B, properly argued why the club should win by reasoning correctly using probability. One more Advanced Algebra group from School A properly argued in favor of the first claim, although they did not make use of probability. Nevertheless, the discussion of the second part of the club’s claim was not successfully accomplished by almost any of the groups. Even though more IMP groups were careful in attempting to address the point, only two nonIMP groups did not use faulty reasoning. From the analysis of the second problem, Street Wise, we can observe that, even though the best results slightly favor IMP groups, there are features of mathematical knowledge in which IMP groups had problems. No significant differences between schools can be detected in most of the features of this task, but when there are differences, these again favor School A. Most groups were able to make a scale drawing following instructions. A few groups made procedural errors. One IMP group and one Advanced Algebra/Algebra II group, however, made a rather inaccurate picture that they used to actually measure some data. No significant differences were found between IMP and nonIMP groups in this aspect, but the groups that made the most mistakes were all from School B. On the ability to give clear explanations of the methods employed, four IMP groups versus two nonIMP groups gave unclear and/or incomplete explanations. On the other hand, only one IMP group from School A gave good verbal explanations explaining their mathematical procedures in both methods. One more IMP group and one Advanced Algebra/Algebra II group were able to do this for one of the methods. Regarding the ability to rely exclusively on a theoretical approach, as compared to physically measuring, at least in one of the methods, two IMP groups and two nonIMP groups (one Advanced Algebra/Algebra II and one Geometry) were able to do it correctly. All of them employed the Pythagorean theorem and similar triangles. Two more groups, one IMP and one Geometry, attempted to rely on trigonometry alone to solve the problem. Both of these groups failed, although the IMP group was better able to relate theoretical procedures with the actual conditions of the problem. One more IMP group tried to use a theoretical method, in this case, the distance formula, but they made a conceptual error by confusing it with the rate formula. Most groups relied on physical measure as a mean to find the solution in the problem, either alone or in combination with the Pythagorean theorem. Only one IMP group had a correct result by combining the Pythagorean theorem with physical measures. However, three IMP groups and one Advanced Algebra/Algebra II group made rather inaccurate measurements or estimations. These students then used the incorrect numbers as the solution in the problem or employed them to get a solution. Moreover, two of the IMP groups tried to match results by using a very inaccurate physical measurement or estimation, instead of using the physical evidence to question the method employed. Only one IMP group from School A was able to figure out two correct and distinct methods to solve the problem and to properly explain the differences between them. A common error, for both IMP and nonIMP groups, was the use of the general statement that physical measurements were less accurate than theoretical ones to justify a lack of ability in finding accurate solutions. In general, IMP students performed better than nonIMP students on the two tasks that were administered. In the first one, IMP groups demonstrated a better understanding of combinatorics and probability. In the second one, the difference between IMP and nonIMP groups was less noticeable. Even though a few IMP groups were more successful in using theoretical methods and physical measures to determine the solution of the problem, and also in being coherent with the verbal explanations, most of them had difficulty in giving clear and complete explanations, in making accurate measurements, and in properly using them to verify theoretical results. No other significant differences were found. As a general feature, we would like to note that none of the groups used trigonometry exclusively to solve the problem. Some of them made a good use of similar triangles, but none of them successfully employed trigonometric identities. The differences between schools clearly favored School A on the first task; on the second task, a slight superiority of IMP School A groups could be observed, but the pattern was not as clear. Dowling, M., & Webb, N. L. (1997). Comparison of IMP students with students enrolled in traditional courses on probability, statistics, problem solving, and reasoning (Project Report 971, Interactive Mathematics Program Evaluation Project). Madison, WI: Wisconsin Center for Education Research. Webb, N. L., & Dowling, M. (1997). Replication study of the comparison of IMP students with students enrolled in traditional courses on probability, statistics, problem solving, and reasoning (Project Report 975, Interactive Mathematics Program Evaluation Project). Madison, WI: Wisconsin Center for Education Research.
Roll 'Em and Win The 123 Math Club of XYZ School developed contests for its school fund raiser. One of the contests involved dice used in this way:
The club is having a disagreement on which rules they should use. One group believes that the club should win if the product is less than 13.
You believe that the club will make money if the product is less than 13. Show the evidence you plan to present to the club to prove your point. Explain how you reached your decision.
Street Wise You are working for the Department of Transportation. A new road is going to be built. You need to draw a diagram of the area in which the road will be built, and find some specified distances. Directions: Make a scale drawing. On your drawing, label the streets. Label the Pine Street points of intersection. All streets are straight lines. (This includes vertical, horizontal, and diagonal lines.) Use the scale of 1.0 cm = l/2 mile. Follow these descriptions to make the drawing on page 2.
Oak Street is 7 miles south of Birch Street.
Pine Street also intersects Birch Street 3 miles west of the railroad at point Y Pine Street intersects Oak Street at point X.
A
2. Describe in words, two of the possible methods which could be used to find the distance between X and Y.
Method 1:
Method 2:
3. Use Method 1 to find the distance between X and Y.
Distance between X and Y:__________
4. Use Method 2 to verify your first result.
Distance between X and Y:__________
5. If the results differ, how would you explain the difference?
SCORING RUBRIC Your responses to the questions on this instrument will be scored according to the following criteria: ADVANCED RESPONSE The response is highly developed and goes beyond what is expected for a proficient response. The response demonstrates creative insights by considering multiple solutions, presenting ideas in many ways (for example with pictures, graphs, and symbols), and may include examples and counterexamples. PROFICIENT RESPONSE The response completely addresses all aspects of the task and includes appropriate application of concepts and procedures. Minor computational errors may be present. Presents coherent use of mathematical words and symbols. Logical conclusions are based upon known facts, properties and relationships. NEARLY PROFICIENT RESPONSE Evidence of inappropriate use of knowledge or skills. Communication may be somewhat unclear, and some conclusions may be faulty or incomplete. MINIMAL RESPONSE The response addresses some of the essential conditions of the task, but includes evidence of major misconceptions. Very little evidence of correct use of mathematical knowledge or skills. Communication is unclear, and conclusions are based upon faulty reasoning. ATTEMPTED RESPONSE The response gives little evidence of addressing the task, and meets none of the essential conditions.
Problem Solving Skills
The analysis of the problem solving skills demonstrated by the videotaped groups focused on the following five subscales or features: (1) selection and organization of information, (2) control over the solution process, (3) checking and confirming the work, (4) level of abstraction, and (5) affects. The students’ work was coded as good, fair, or poor according to the overall level of sophistication shown, and particular characteristics of the subscales considered under problemsolving skills. A complete description of the subscales and coding categories follows.
GOOD Students show a maintained focus on the relevant conditions of the problem. Students are able to select the important information in the problem and use it in a meaningful way.
Students are capable of disregarding irrelevant information presented by group members. Appropriate organization of information in, for example, "inout" tables, charts, pictures or diagrams.

Students present relevant mathematical knowledge and skills to the group. Group members are able to utilize these resources efficiently. The students have a good intuition about the correct strategy and are able to control the path or course of the solution process.
Students are able to come up with examples to verify or contradict the work the group is doing. Students are willing to challenge information presented by other group members and prove the claims or warrants made. They always try to see if the answer is reasonable or "makes sense".
Students show interest in discovering the pattern underlying the problem. Even though the problem may be familiar to them, they look for patterns or structures in the problem, beyond the routine.
Students show interest in solving the problem. They approach the problem in a serious manner, engage in the dialogue, and show confidence and persistence in carrying out computations.
Students look for patterns and generalizations and when they have the intuition that the problem has a general solution which shortens the solution process they concentrate efforts on finding that generalization and do no give up early. They are willing to take risks, they show persistence in following up their ideas and resistance to premature closure.
Students also show flexibility to change the plan when it does not work.
FAIR or AVERAGE
Students show a sustained focus on some of the relevant conditions of the problem and ignore or miss others. Sometimes the group gets distracted with irrelevant information presented by a group member.
Students are able to adequately organize some of the relevant information given. (For example, they may use "inout" tables, or charts and diagrams, but may fail to consider all possible combinations or their work may have computational errors.)
Students make some connections with prior knowledge or mathematical experiences, but are not able to carry on and use their knowledge in a useful way. The depth of the mathematical knowledge shown (content knowledge), might be superficial resulting in an inexact or inaccurate solution. Sometimes students have the correct intuition about the solution of the problem, but they are either uncertain about how to apply prior knowledge to a new situation or the content knowledge is superficial, therefore, they are not able to meaningfully advance the solution process.
Students often check their work, but it is mostly limited to "factual checking". They may fail to take into account relevant conditions or variables.
Occasionally, students challenge ideas, claims, or arguments presented by other members. Most of the time, students agree with each other without questioning.
Although students show persistency in carrying out the computations, they do not show interest in generalizing the results using a more powerful mathematical or algebraic structure. They seem satisfied with only finding the pattern that underlies the problem and get easily distracted with routine computations.
Some students may try to make a relevant generalization, but they might not be supported by other group members, or the group is not able to use the knowledge presented or contribution made to advance the solution.
Often, not all the students in the group get involved in the solution process or they might get easily distracted by irrelevant conversation.
There is usually a lack of critical sense and students seem to trust other group members’ problemsolving ability.
After discovering the pattern, they easily give up on finding a "shorter way" and persist in carrying out the computations. They seem to be more comfortable with this position instead of taking risks and trying something else.
POOR
The group focuses on some of the relevant conditions of the problem, but fails to disregard irrelevant information provided either by the problem itself of by some group members.
Although there might be some organization of relevant variables and data, students fail to use the information in a meaningful way, or may fail to consider all the possibilities before reaching a conclusion. Students are not able to develop a sense of how conditions and variables are related to each other.
Control over the solution process. Connection with previous mathematical knowledge and experiences. Intuition.  
Checking and confirming the work  
Level of abstraction  
Affects  
Selection and organization of information  
Control over the solution process. Connection with previous mathematical knowledge and experiences (intuition).  
Checking and confirming the work  
Level of abstraction  
Affects  
Selection and organization of information 
Some students might be able to make
some connections with prior
mathematical knowledge or experience, but the information is not used, or
developed by the group. The explanations, provided are usually unclear and
superficial. Although there might be some intuition about the way that the problem
should be solved, students fail to integrate relevant data, content
knowledge, and solution strategy. Group does not show sustained control over the solution process. Control over the solution process. Connection with previous mathematical
knowledge and experiences (intuition).
Students fail to check any obtained result against the relevant information provided by the problem. Mistakes may often go unnoticed.
Students do not check whether the results are reasonable or not. They rely completely on each other and do not question the results. No meaningful and sustained examples, counter example, arguments or claims are presented.
Observations made about the problem, variables, or conditions are usually superficial or vague. Content knowledge seems irrelevant or superficial.
Students may have the notion of the existence of a pattern and may show some interest in discovering that pattern, but they are not able to carry out their thoughts in order to find a generalization that would explain the relationships observed.
Students do not show persistence in carrying out the necessary computations or in considering all the possibilities. Students may even question if the problem could have a shorter or more general solution.
Students demonstrate a low level of confidence. Students seem unwilling to take risks. They may attempt to solve the problem in different ways, but they do not have the persistency or the necessary content and experiential knowledge to finalize or complete their thoughts.
The group is often not flexible enough to listen to other members’ points of view in order to make relevant changes, if necessary, to the solution strategy.
Mathematical Discourse
One important component of the instructional process is the analysis of how the communication and interaction process among the students while solving a mathematics task reflect the quality and level of their mathematical thinking and understanding of relevant concepts and principles. The study of the mathematical discourse that goes on in the classroom environment, for example, provides evidence of the students’ ability to engage in the processes of mathematical thinking and, more importantly, the meanings students attribute to mathematics. Recent research efforts have emphasized the importance of taking into account the type of argumentation, discussion orchestration, and classroom discourse the takes place in a learning situation as a measure of how students think mathematically. The following mathematical discourse dimension or scale examines the quality of the interaction and communicative exchange, and forms of discourse that took place among the students in the video taping sessions when solving two performance assessment mathematical tasks in groups of three to five students. Three fundamental aspects or subscales were considered in this scale: the information exchange that occurred among the students, the type and quality of questions that were posed, and the level of mathematical reasoning as evidenced by the verbal interaction. The video taped groups received a score of good, average or fair, or poor for each of these three subscales. A detailed description of the subscales considered within mathematical discourse follows.
I. Information Exchange
This subscale focuses upon the quality and sophistication of the information exchange that took place in the groups while solving the problems. Specifically, it describes the type of information presented by the students to the group, how it was supported and developed, and how the group members reacted to the information presented.
Good:
Checking and confirming the work  
Level of abstraction  
Affects 
The majority of the ideas or information presented goes beyond merely perceptual or contextual data. That is, most of the ideas presented by the students have a substantial content that permits and advances the solution of the problem; either in terms of strategies or ways to approach the problem in the different stages of development, or in terms of conjectures, significant relationships among data, explanation of an underlying pattern or structure that allows to organize results, etc.  
There is an active interaction among the students. Students react or argue about the ideas presented by other members; they may accept the idea presented and work to develop it, or try to understand it and evaluate how it fits the solution strategy, or disagree and provide counterarguments, or counterexamples, or make explicit the points that are not clear, etc.  
Students are able to provide clear and complete explanations most of the time. 
Fair or Average:
Students present some ideas whose content permit advancement in the solution of the problem, as mentioned above. Nevertheless, a good amount of the information presented is highly contextual and refers to specific data or results. Some relevant connections or relationships among the data and results obtained are missed or may not be clearly explicated, and the information shared, although the interaction may be active, tends to be related to computational processes.  
Ideas, conjectures and opinions presented by one member of the group are often accepted by other members and computational processes can be carried out accordingly; but the students may not carefully evaluate all the information presented that permits the advancement of the solution.  
Students’ claims and conjectures are sometimes vague, and they may fail to support them properly with the data or results available.  
Some verbal explanations may be unclear or imprecise. 
Poor:
Most of the information presented by the students refers to numeric results, data given or it is contextual with few or no references to relevant connections or relationships that facilitate the advancement of the solution of the problem.  
Reactions to the information or ideas presented by other members of the group are mostly passive. Most of the information presented is not explained, justified, supported by other members. Therefore, the information exchange or communication lacks fluency and it is limited to more or less isolated and interrupted interventions  
Students may present some opinions or conjectures, but they are usually vague, unclear or imprecise. 
II. Questions
Questioning is an integral component of processing information. The questions that are posed and the manner in which they are phrased and sequenced are indicators of the way students’ level of understanding of the information presented. The analysis of the questioning process also provides evidence of the level of student involvement in the solution process. This subscale focuses on the type, significance and quality of questions that students posed to each other during the problemsolving sessions. Specifically, this subscale attempts to give a description of how the students use questioning to clarify, verify or confirm ideas, hypothesis or conjectures that permit the advancement of the problem.
Good:
Most of the questions posed are relevant to the content of the problem, allow the advancement of the problem, and suggest higher order thinking.  
Students ask questions to clarify content, claims that are made, or explanations that are not understood.  
Students ask questions that require the analysis or synthesis of facts, relationships, patterns or structures.  
Students ask questions that require an evaluation of ideas presented; either to verify or confirm thinking or to obtain a reaction about a conjecture, proposal, idea or opinion, etc.  
Students may ask questions that require the recall of isolated bits of information in terms of results, terminology or specific facts, when needed. They may also pose questions that require the recall of prior knowledge on how to deal with specifics (e.g., conventions, classifications, processes, methodologies, etc.). 
Students ask many questions to clarify specific problemrelated content (i.e., data and/or variables provided by the problem). Students pose some questions to clarify explanations, claims or information presented.  
Students pose some questions that require analysis or synthesis of facts, relationships or structures.  
Students ask some questions that require an evaluation of ideas presented; either to verify thinking, processes, or results, or to obtain a reaction about a conjecture, claim, hypothesis or proposal, etc.  
The most frequent kind of questions posed by the students are factquestions, which are meant for the others to provide isolated bits of information mostly in terms of partial results and/or specific facts. 
Poor:
Overall, little relevant questioning processes.  
Students may ask few questions to clarify explanations, claims or conditions that are not understood, but the relevance of the questions to advance the solution process is minimal.  
Questions that require analysis, synthesis or evaluation are very scarce or nonexistent.  
Questions that require analysis, synthesis or evaluation are very scarce or nonexistent.  
Lowlevel cognitive questions predominate, i.e., highlycontextualized questions and isolated factquestions (e.g., what is the next number? Four? What did you get?, etc.) clearly dominate the questioning process. 
III. Reasoning
This subscale focuses upon students’ ability to make meaningful inferences, manipulate information and ideas by synthesizing, generalizing, explaining, hypothesizing, or arriving at conclusions that produce new meaning and understanding for them (Newmann, Secada, and Wehlage, 1995). The groups were classified in one of the following clusters or categories according to the quality, depth, and level of sophistication of the reasoning skills demonstrated by the group members.
Good:
Students concentrate on looking for patterns and more general structures underlying the data and results obtained. Students have the ability to properly abstract these structures from the data and available results.  
Students make correct inferences from the relationships observed among the data available to analogous or more complex and general cases.  
Students are able to draw relevant conclusions by linking two or more pieces of information that were not previously connected. The reasoning is correct and clear.  
Students usually take time to ponder the relevance of ideas presented.  
Students are systematic in reflecting upon their inferences and conclusions, and care about rigor and correspondence with actual data and results.  
Students are able to make proper use of previous mathematical knowledge which is relevant to the actual conditions of the given problem; and are capable to successfully integrate both. 
Fair or Average:
Students are able to recognize patterns in the data, but they are not able to find, or show no interest in finding, more general or powerful structures underlying the data and results.  
Students make relevant inferences from the relationships observed between the data available and results; but they may fail to translate it into a mathematical or algebraic expression.  
Students are able to draw conclusions by linking two or more pieces of information that were not previously connected. The reasoning may be somewhat faulty or unclear.  
Students may fail to take into account the relevance of ideas presented.  
Students are not systematic in reflecting upon their inferences and conclusions, and they are not very rigorous in checking correspondence between data and results.  
Students may use previous mathematical knowledge, but the relevance to the actual conditions of the problem is not clear; and they may fail to adequately integrate both. 
Poor:
Students are mainly engaged in a computational process, and have little interest or no ability to look for patterns and relevant relationships.  
Students have difficulties making meaningful inferences form the relationships observed that would allow them to advance the solution process. They may also try to make inferences with little reference or attention to the actual relationships among the data and results available.  
Students have serious difficulties making correct or relevant connections.  
Students pay little attention to ideas presented and do not make a serious or concentrated effort to ponder them.  
There might be some reflection upon results obtained or inferences made, but students fail to take into account relevant relationships, data or information given; or make the appropriate connections.  
Students may use previous mathematical knowledge, but they may use it incorrectly and may fail to integrate it with the actual conditions of the problem. 
Social Component
I. Type of Collaboration
A. No collaboration
B. Indirect collaboration: Some students work individually, although they explain themselves aloud and confer with the group or with another member when necessary.
C. Partial collaboration: Students share their ideas with others; for example, explain themselves, confirm information with others, ask each other questions in complete sentences, are supportive of ideas produced by others, etc.
D. Complete collaboration: Dialogue builds coherently on students’ ideas to promote collective understanding, a good part of the dialogue is sustained, and students are willing to challenge ideas presented by others when appropriate.
II. Group Control (Who controls the solution process?)
A. One member of the group has the authority or control.
B. Two or more members of the group have the authority or control and other members intervene occasionally.
C. All members have the same authority or control.
Reference
Newmann, F.M., Secada, W.G., & Wehlage, G.G. (1995). A guide to authentic interaction and assessment: vision, standards and scoring. Madison, WI: Wisconsin Center for Education Research.
Analysis of Students’ Mathematical Knowledge Exhibited in Solving
Two Performance Assessment Activities
ANALYSIS
1. Roll'em and win
1. Method for generating set of possible rolls
Three IMP groups (AG1, AG5 , AG6) and one Algebra group (AG4), all from School A used a rug table showing different rolls. 
One IMP group (BG3) from School B used a tree diagram to generate the set of possible rolls. 
One IMP group (BG1) from School B and three Algebra groups (AG2, BG4, BG6), one from School A and two from School B, and one Geometry group (BG5) from School B counted the different possible rolls following a certain pattern or order. Most of these five groups ordered their work as if in a tree diagram, but without the advantage of organization provided by using a tree diagram. As a consequence, some of the groups tried to save work by altering the order (ex: BG6) (figure 1). 
Figure 1.
Since the number of rolls is small, this strategy allowed the students to be exhaustive in most cases, but the same method for larger numbers is likely to present more problems than the rug method (or the tree diagram, which none of these students used).
Another problem with this method, which is avoided with the rug method, is that some of the groups counted combinations only instead of permutations and, in this way, left out possibilities. For example, groups did not consider (2,1) and (1,2) as different possibilities. One IMP group (BG1), one Algebra group (BG4), and one geometry group (BG5), all from School B, demonstrated this conceptual error (figure 2).
Figure 2
One IMP group (BG2) from School B and one Geometry group (AG3) from School A, did not present a viable method for generating a set of all different rolls (figure 3). This was considered a conceptual error. 
Figure 3.
2. Frequencies for the different rolls
None of the groups provided the frequencies for the different possible rolls. This is not necessary for solving the problem, although it could be useful to have the frequencies for products 13 to 18 and higher in order to discuss the argument with clearer mathematical data. 
3. Probability
Three IMP groups from School A (AG1, AG5, AG6), and two Algebra groups (AG4, BG6), one from School A and one from School B, calculated the probability for the different products up to 13 correctly, although they expressed them in different ways (23 out of 36, 23/36, 64%) (figure 4). 
Figure 4
One IMP group (BG3) from School B calculated the probability incorrectly, because they failed to count the total number of possibilities, even though they generated them properly. This is a procedural error (figure 5). 
Figure 5.
Two IMP groups (BG1, BG2), one Algebra group (BG4), and one Geometry group (BG5), all of them from School B, calculated the probability incorrectly because they had problems in generating the set of possibilities. These were conceptual errors or gaps in understanding combinations that resulted in procedural errors in probability (figure 6). 
Figure 6.
One algebra group (AG2) and one Geometry group (AG3), both from School A, did not mention probability in solving the problem (figures 3 and 7). 
4. Mathematical arguments presented to discuss the rules to be used by the Math Club
a) Specific illusion to the standpoint of the first group
Three IMP groups (AG1, AG5, AG6), from School A, and one Algebra group (BG6), from School B, properly argued why the club should win if the product is less than 13 by reasoning correctly using probabilities. 
One Algebra group from School A (AG4) reasoned correctly using probability but did not specify who should win in its argument, a reasoning error (figure 7). 
Figure 7.
One Algebra group (AG2), from School A, properly argued that the club should win if the product is less than 13 from the results of their table, but they did not make any mention of probability (figure 8). 
Figure 8.
One IMP group (BG3) from School B calculated the probability incorrectly because they failed to count the total number of possibilities, even though they generated them properly. Therefore, the mathematical result was not totally correct or supported by their agreement with the group that defended that the club should win by going under 13, a procedural error in combinatorics that led to faulty reasoning. 
One IMP group (BG1), one Algebra group (BG4), and one Geometry group (BG5), all of them from School B, correctly supported the view that the club should win if the product is under 13, but their mathematical calculations did not support this conclusion, since they counted combinations instead of permutations, conceptual error in combinatorics that led to faulty reasoning (figure 9). 
Figure 9.
One IMP group (BG2) from School B and one Geometry group (AG3) from School A did not have a systematic method for counting. Therefore, their responses were not supported with mathematically sound results, a conceptual error in combinatorics that led to faulty reasoning. 
b) specific allusion to the standpoint of the second group
Only two groups, one Algebra group (AG4) from School A and one Geometry group (BG5) from School B, argued that, in fact, the probability of a product being less than 18 is even higher that being less than 13. 
This is correct but limited in terms of discussing why it is not necessary to go under 18, why a bet for under 13 is indeed an optimal option than betting for under 18 when we take into account the global picturethat is, the club wants to earn money, but also to attract people to bet, or why their reasoning (18 is half of 36) is faulty.
The rest of the groups did not grasp the point in enough detail to discuss a means to convince the second Club group that its option is not better and/or its reasoning is not correct. The following faulty reasoning was held by three IMP groups (AG1, AG6, BG2), two from School A and one from School B: 
Figure 10.
Nevertheless, more IMP groups tried to address the standpoints of both groups in the club, although they may have missed what was relevant for discussing the second standpoint specifically. (Two Algebra groups (AG2, BG6), one from School A and one from School B, one Geometry group (AG3) from School A, and one IMP group (AG5) from School A did not mention anything related to the view of the second group).
Summary:
Four IMP groups, three from School A and one from School B, versus one Algebra group from School A were able to use the most reliable methods for generating the set of all possible rolls of the two dices. The groups from School A used a rug table and the group from School B used a tree diagram. The rest of the groups used some form of organized counting. One conceptual error observed in this part of the problem was the consideration of combinations instead of permutations to figure out the total number of possible results when rolling the dice; one IMP group, one Algebra group, and one Geometry groupall of them from School Bmade this error.
Three IMP groups, all of them from School A, and two Algebra groups, one from School A and one from School B, calculated correctly the probability for the product of the dice to be under 13. One IMP group from School B calculated this probability incorrectly because of a computational error. The rest of the groups either calculated the probability incorrectly because of conceptual errors or appeared to lack an of understanding of combinatorics that led to procedural errors in probability, or did not mention probability. All IMP groups mentioned probability.
Three IMP groups, all from School A and one Algebra group from School B properly argued why the club should win if the product is less than 13 by reasoning correctly in terms of probability. One Algebra group from School A properly argued that the club should win if the product is less than 13, although they did not make use of probability. The rest of the groups made errors in counting the number of combinations which resulted in faulty reasoning, even though the response could be correct. All these errors were conceptual except in the case of one procedural error made by one IMP group from School B.
Most of the groups did not grasp the need to discuss in their response the argument of the second group in the club. Although more IMP groups were careful in trying to address this specific part of the problem, only one Algebra and one Geometry group were able to give an accurate reason. None of the IMP groups did so.
2. Street Wise
1. Picture
Are students able to make a scale drawing using the information provided (1cm=1/2 mile)? Are students able to use the information provided in order to draw their picture.
All the groups from School A were able to make a correct scale drawing following the instructions given in the problem and used all the information provided. The only exception to this was an IMP group (AG6) that used an approximate measure for a given distance and failed to indicate this on the picture, a procedural error (figure 11). 
Figure 11.
None of the groups from School B had a ruler available. 
Two IMP groups (BG2, BG3) and one Algebra group (BG4) were able to make up their own scale and draw a scale picture; nevertheless, one IMP group (BG2) failed to mark on the picture all the measures provided
(figure 12).
Figure 12.
One IMP group ((BG1), one Algebra group (BG6), and one Geometry group (BG5) from School B failed to make a scale drawing even by estimation. One Algebra group (BG6) confused the orientation in drawing one of the streets and had a difficult time after that trying to make sense of the problem. This group (BG6) actually used the picture to estimate some data as did group (BG1) to calculate angles that were wrong (figures 13 and 14).
Figure 13.
Figure 14.
2. Verbal explanation of methods
Do students describe clearly the methods which can be used to find the distance between X and Y?
One IMP group (AG5) from School A, two Algebra groups (BG6, AG4), one from School B and one from School A, were able to give clear explanations of the methods they used to find the required distance, although one of the methods was just measuring with a ruler (figure 15). 
2. Describe in words, two of the possible methods which could be used
to find the distance between X and Y.
Figure 15.
One IMP group from School B (BG3), one Algebra group (BG4) from School B, and one Geometry group (AG3) from School A gave relatively clear explanations for one of the methods but not for the other (figure 16). 
Figure 16.
The explanations of the rest of the groups: four IMP (AG1, AG6, BG1, BG2), from School A and School B, one Algebra group (AG2) from School A and one Geometry group (BG5) from School B were rather incomplete in almost all cases. Mostly they named the procedures (e.g., "we used the Pythagorean theorem") or just the end of the process (figures 17 and 18). 
Figure 17
Figure 18.
Do the descriptions correspond to the solutions/computations provided?
Only one IMP group (AG5) from School A matched clear verbal descriptions with mathematical procedures and solutions in both methods. 
One IMP group (BG3) and one Algebra group (BG4) from School B matched their verbal description and mathematical performance for both methods. However both groups did not clearly explain one of the methods. This made it difficult to determine if the method was appropriate (figure 19). 
Figure 19.
There were no other significant differences between IMP groups and others for the remaining groups. In most groups, the explanation was rather general or incomplete, and the results and procedures were weak, e.g., group (AG6) (figure 20). 
Figure 20.
3. Methods employed
What strategies/methods do students use to find the distance between X and Y? Are the students able to employ the methods correctly?
Classification by groups:
Group AG1: (IMP A) 
1. Used Pythagorean theorem and physical measure of one side of the big triangle to guess the hypotenuse of the big triangle, subtracted an inaccurate measure of the hypotenuse of the small triangle to get the distance XY. They made a computational error.
2. They made an inaccurate physical measure.
Group AG5: (IMP A) 
1. Made a physical measure (within the range 7.9 to 8.3).
2. Used similar triangles to find the side of the rectangle triangle whose hypotenuse is XY (constructed drawing a perpendicular from Y to bottom side of the picture); Pythagorean theorem to find the hypotenuse XY. Correct.
Group AG6 (IMP A) 
1. Used Pythagorean theorem with physical measures (out of range). [Procedural error.]
2. Made a physical measure (out of range). [Made same kind of error as in method 1]
This group did not convert centimeters into miles.
Group AG2 (ALG A) 
1. Applied the Pythagorean theorem only to the small triangle.
2. Made a physical measure (within the range).
Group AG4 (ALG A) 
1. Made a physical measure. They converted centimeters into miles multiplying by 2 instead of dividing; taking that into account, the measure is within the range. [Procedural error]
2. Explained that the second method would be to use the Pythagorean theorem and physical measure of one side of the big triangle to guess the hypotenuse of the big triangle, and to subtract the hypotenuse of the small triangle obtained by means of the Pythagorean theorem applied to the small triangle. Nevertheless, their calculations did not correspond to this explanation. In the interaction recorded in video it is possible to see that the students got their result by trying different operations in order to match the result of the first method [conceptual error]. However, the result is out of range taking into account the wrong conversion between centimeters and miles.
Group AG3: (GEOMETRY A) 
1. Used the Pythagorean theorem to find the hypotenuse of the small triangle; similar triangles to find the missing side of the big triangle, Pythagorean theorem to find the hypotenuse of the big triangle, and subtraction of both hypotenuses. The method was correct, but the students made mistakes in calculations that they did not show in the written work. In the video, we can see that they talked about 12=7+5 being the measurement of the long side of the big triangle and 7.2 the measurement of the short side. Those were the two measurements that the students were going to use in the Pythagorean theorem, but at some moment they lost sight of it and used 7 instead of 12 as the measurement of the long side of the triangle. [Procedural error]
2. Used Pythagorean theorem for the small triangle; attempted to use similar triangles and trigonometric identities. None of the trigonometric methods was actually carried out to get a solution.
Group BG1 (IMP B) 
1. Used trigonometric identity with wrong angle measure to get XY. The triangle to which they applied it was the one constructed by drawing a perpendicular line from Y to the bottom line, and the students assumed that this triangle was equilateral. Then, they used the Pythagorean theorem to get the missing side of the triangle, and applied the Pythagorean theorem again to find out the hypotenuse (XY) which they already had obtained. Besides, the students tried to round the results once more to make them match. [Conceptual and procedural errors].
2. Used trigonometric identities assuming wrong angles, rounding results to make them match with the previous ones. [Conceptual and procedural errors].
Group BG2 (IMP B) 
1. Used rate formula (mistaken for distance formula) and physical measure to get the horizontal coordinate for point X. [Conceptual error].
2. Used estimation [completely out of the range 7.5 to 9.0]. [Procedural error].
Group BG3 (IMP B) 
1. Used Pythagorean theorem applied to the triangle built by drawing a perpendicular line from X to the bottom line. The students obtained one of the sides by accurately estimating its length. Result was within range.
2. Correctly used similar triangles and Pythagorean theorem.
Group BG4 (ALG B) 
1. Correctly used Pythagorean theorem to find the hypotenuse of the small triangle; Pythagorean theorem and similar triangles to find the hypotenuse of the big triangle and subtraction of both hypotenuses.
2. Used Pythagorean theorem to find the hypotenuse of the small triangle; similar triangles to obtain the hypotenuse of the big triangle, and subtraction of both hypotenuses. Computational mistakes. [Procedural error].
Group BG6 (ALG B) 
1. Used estimation (within range).
2. Used Pythagorean theorem and estimation to find the missing side. (Out of range). [Procedural error].
Group BG5 (GEOM B) 
1. Used Pythagorean theorem to find the hypotenuse of the small triangle; Pythagorean theorem to find the hypotenuse of the big triangle. It is not possible to tell how the students figured out the measure of the missing side of the triangle in the written work, but in the video we can observe that they used similar triangles. Then, the students subtracted both hypotenuses. This was a correct result.
2. The students claimed to have used similar triangles to find out the hypotenuse of the big triangle, but the computations made no sense. The result is correct, but there is no way of finding out hints about how they got it in the written work nor in the video recording.
Classification by methods:
a) Pythagorean theorem + trigonometric results necessary to apply the Pythagorean theorem (similarity of triangles):
Two IMP groups (AG5, BG3), one from School A and one from School B, one Algebra group (BG4) from School A; and one Geometry group (BG5) from School B were able to do this correctly. The Geometry group did not show any evidence of employing similar triangles in their written work, but it can be observed in the video recording. The Algebra group also tried to combine similar triangles with the Pythagorean theorem in a different way for the second method, but they made computational mistakes. (figure 21). 
Figure 21
One Geometry group (AG3) from School A applied the correct procedure but they made mistakes in calculations. [Procedural error]. 
One IMP group (BG1) from School B did circular reasoning between the Pythagorean theorem and trigonometry, and they assumed angles that were not correct. (This might be due to the fact that they did not draw a scale picture and the triangle seemed to be isosceles). [Conceptual and procedural errors] (figure 22). 
Figure 22.
b) Pythagorean theorem + ruler or estimation to guess the necessary data to apply the theorem:
Three IMP groups (AG1, AG6, BG3), two from School A and one from School B, and two Algebra groups (AG4, BG6), one from School A and one from School B. The two IMP groups from School A and the Algebra group from School B had inaccurate results. [Procedural errors]. The Algebra group from School A was only able to explain the method but not to actually carry it out to find the solution of the problem (figure 23). 
Figure 23.
c) Measure with a ruler:
Three IMP groups (AG1, AG5, AG6) and two Algebra groups (AG2, AG4), all of them from School A. Two IMP groups’ measures were very inaccurate (AG1, AG6) (figure 24). 
Figure 24.
d) Estimation:
One IMP group (BG2) and one Algebra group (BG6), both from School B. The IMP group’s estimation was very inaccurate (figures 25a and 25b). 
Figure 25a.
Figure 25b.
e) Trigonometry exclusively:
One IMP group (BG1) from School B. They assumed a wrong measure for the angles, although they used the trigonometric relations properly. They assumed that the measures of the other two angles of a right triangle is 45˚ (figure 26). 
Figure 26.
One Geometry group (AG3) from School A very superficially used trigonometric formulas and similar triangles. They were unable to relate this superficial knowledge with the data and conditions of the problem.
f) Algebra (distance formula):
One IMP group (BG2) from School B erroneously used the formula for the rate, a clear conceptual error (See figure 25). 
4. Ability to explain the difference between the methods presented
Only one IMP group (AG5) was able to properly address this issue. It was the only one that had two correct different methods for the solution of the  
problem (figure 27). 
Figure 27.
Three IMP groups (BG1, BG2, BG3) from School B, three Algebra groups (AG2, AG3, BG6), from School A and School B, and the Geometry groups (AG3, BG5), from School A and School B, argued that the ‘theoretical’ method that they employed was more accurate than the physical measure, but they employed physical measures in their ‘theoretic’ method as well. This invalidated their argument, which they were not aware of. 
Two groups (AG1, AG6), both from School A gave a very inaccurate measure with the ruler. This could have been done in order to match wrong results obtained by more theoretical means (figure 28 and figure 24).
Figure 28.
5. Conversion between miles and centimeters
One Algebra group (AG4) from School A made a mistake in the conversion and did it the other way around. That is, they converted 1 mile=1/2 cm. [Procedural error]. 
Summary:
Almost all the groups were able to make a scale drawing following the instructions given in the problem, either using a ruler or makingup a scale in the case of the groups from School B, which did not have a ruler available. The exceptions to this were: One IMP group, one Algebra group (they also confused the orientation of one street), and one Geometry group from School B. One IMP group from School A failed to mark one of the measures provided and used an approximate one instead. The nature of these errors is procedural in some cases, but in others (one IMP and one Algebra group) are considered more as conceptual errors, since they made a rather inaccurate picture which they used afterwards to actually measure some data. The errors in this part of the problem were more related to the school than to program.
More nonIMP groups were able to give clear or relatively clear verbal explanations of the methods they used for one or both of them. Four IMP groups, one Algebra group, and one Geometry group gave rather incomplete verbal explanations of their procedures.
On the other hand, only one IMP from School A could match clear verbal descriptions and mathematical procedures in both methods. One IMP group and one Algebra group, both from School B, matched their verbal description and mathematical performance in both methods. But because one of the methods was not clearly explained, it was more difficult to make sense of the correspondence.
Two IMP groups, from School A and School B, one Algebra group from School B, and one Geometry group from School A had at least one correct theoretical result. They obtained this by combining the Pythagorean theorem and trigonometry (similar triangles). The only group that had another correct second method (measured with a ruler) was the IMP group from School A.
One more group, a Geometry one from School A, also applied correctly trigonometry (similar triangles) and Pythagorean theorem at the theoretical level, but they made procedural mistakes.
One IMP group from School B had a correct approximate theoretical result, that is, the students combined theoretical procedures with accurate physical measures or estimations for some data that they could not or would not obtain theoretically. Two more IMP groups, both from School A, and one Algebra group from School B used the same procedure, but the physical measure was inaccurate.
A number of the groups, both IMP (four groups) and Algebra (three groups) but no Geometry groups, relied on measures or estimation as a strategy to determine the value of the
distance XY. In general, we can observe a problem with the accuracy of the measures or estimations taken by IMP groups. Three IMP groups, from School A and School B, made inaccurate measures or estimations versus only one Algebra group from School B.
As far as relying on trigonometric methods alone to solve the problem, one IMP group from School B attempted to do it. They knew the trigonometric identities, but they assumed the wrong angles. (Since they were not able to make a proper scale drawing, they considered the triangle isosceles.) Another Geometry group, from School A, also tried to rely on trigonometry, but very superficially. They were not able to fulfill the conditions of the problem.
One more IMP group from School B made a conceptual error by confusing the formula for the rate with a distance formula.
Only one IMP group from School A was able to employ correctly two different methods to solve the problem and to properly explain the differences between them. Various other groups had the correct notion that theoretical measures are more accurate than physical ones, but this claim had little relation with the actual results that they presented.
Finally, one Algebra group from School A made a procedural mistake in the conversion between centimeters and miles, they converted 1 mile=1/2 cm.
Conclusion
From the analysis of the first problem, ‘Roll’Em and Win’, we can observe that, in almost all the features of mathematical knowledge that can be assessed in the task, IMP groups performed better than nonIMP groups. Also, the differences clearly favored School A. This suggests that there is an interaction between the school and the curriculum.
In regards to knowledge about combinatorics, four IMP groups, all of them from School A, versus only one Algebra group from School A, were able to use the most reliable methods for generating the total number of possibilities for rolling two dice (rug table and tree diagram). A conceptual error of counting combinations instead of variations was made by three groups from School B: one IMP, one Algebra, and one Geometry.
All of the IMP groups referred to probability in order to discuss the arguments given by the two parts of the club (two nonIMP groups did not mention probability). Three IMP groups from School A versus two Algebra groups, from School A and School B, calculated the probability for the product of the dice to be under 13 correctly. One more IMP group from School B made just a computational error in calculating this probability.
Concerning the discussion of each part of the club’s claim, three IMP groups from School A, compared to one Algebra group from School B, properly argued why the club should win by reasoning correctly in terms of probability, one more Algebra group from School A properly
argued in favor of the first claim, although they did not make use of probability. Nevertheless, the discussion of the second part of the club’s claim was not successfully accomplished by almost any of the groups. Even though more IMP groups were careful in attempting to address the point, only two nonIMP groups did not use faulty reasoning.
From the analysis of the second problem, ‘Street Wise’, we can observe that, even though the best results slightly favor IMP groups, there are features of mathematical knowledge in which IMP groups had problems. No significant differences between schools can be detected in most of the features of this task, but when there are differences, these again favor School A.
In regards to the ability to make a scale drawing following instructions, most groups were able to do it. A few groups made procedural errors. One IMP group and one Algebra group, however, made a rather inaccurate picture which they used to actually measure some data. No significant differences can be found between IMP and nonIMP groups in this aspect, but the groups that made mistakes were all from School B.
On the ability to give clear explanations of the methods employed, four IMP groups versus two nonIMP groups gave unclear and/or incomplete explanations. On the other hand, only one IMP group from School A could match good verbal explanations and mathematical procedures in both methods. One more IMP group and one Algebra group were able to do it in one of the methods.
Regarding the ability to rely exclusively on a theoretical approach at least in one of the methods, two IMP groups and two nonIMP groups (one Algebra and one Geometry) were able to do it correctly. All of them employed the Pythagorean theorem and similar triangles. Two more groups, one IMP and one Geometry, attempted to rely on trigonometry alone as one approach to solve the problem. Both of these groups failed, although the IMP group was more able to relate theoretical procedures with the actual conditions of the problem. One more IMP group tried to use a theoretical method, in this case, the distance formula, but they made a conceptual error by confusing it with the rate formula.
Most groups relied on physical measure as a mean to find the solution of the problem, either alone or in combination with the Pythagorean theorem. Only one IMP group had a correct result by combining the Pythagorean theorem with physical measures. However, it is important to note that three IMP groups versus one Algebra group made rather inaccurate measurements or estimations. These students then used the incorrect numbers as the solution of the problem or employed them to get a solution. Moreover, two of the IMP groups tried to match results by using a very inaccurate physical measurement or estimation, instead of using the physical evidence to question the method employed.
Only one IMP group from School A was able to figure out two correct different strategies to solve the problem and to properly explain the differences between them. A common error, both to IMP and nonIMP groups, was the use of the general statement that physical measurements were less accurate than theoretical ones, to justify a lack of ability in finding accurate solutions.
In general, we can say that IMP students performed better than nonIMP students on the two tasks that were administered. In the first one, IMP groups showed a better understanding of combinatorics and probability. In the second one, the difference between IMP and nonIMP groups was less noticeable. Even though a few IMP groups were more successful in using theoretical methods and physical measures to determine the solution of the problem, and also in being coherent with the verbal explanations, a number of them had difficulty in giving clear and complete explanations, in making accurate measurements, and in properly using them to have some control over the theoretical result rather than trying to match the different outcomes. No other significant differences could be found. As a general feature, we would like to note that none of the groups could or would use trigonometry exclusively to solve the problem. Some of them made a good use of similar triangles, but none of them successfully employed trigonometric identities. The differences between schools clearly favored School A in the first task; in the second task, a slight superiority of IMP School A groups could be observed, but the pattern was not so clear in this case.