Posted on

Mark’s Response to Concerns About Local Norms

Here is a recent posting on a popular School Psychology Listserv that I responded to.

The Original Posting

IMHO I would be very cautious about using local norms of any kind, even if you are stratifying your sample. This can lead to errors in interpretation. For example, consider a sample in a group of students in school X that performs significantly poorer than students in school Y but the student is “Average or low Average” compared to peers. Than the student moves to a better performing school where there are language learners who score higher on your tests. When you use local norms, this student will look different than when compared to his/her previous group.  This scenario can also work inversely- a child moves from a school where most LEP students perform higher than he/she. Then at a lower performing school where LEP students perform low. Now the child’s performances doesn’t look so bad.

I suggest always using well developed national norms as these provide for better representation and are more defensible in the event of any due process.

Mark’s Response

Sorry, but I couldn’t disagree more.

The choice of norms depends on the group to which one would like to generalize and the decision made. In this instance, the question is “do we suspect disability?” If a sizable proportion of students in the student’s school are discrepant readers, why would one suspect disability? That’a general education problem, not a special education problem. Now if I wanted to make a statement about “how does this student compare to students in Anywhere USA–a very different question, then national norms would be more appropriate.

Here is what is really cool about assessment and decision making. We can always turn to our professional guidebooks. Here is one I use quite a lot and it provides us some guidance about norms. My emphasis in in BOLD.

American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological tests. Washington, DC: American Educational Research Association.

Standard 12.5.

Local norms should be developed to support test users’ intended interpretations.

Comment: Comparison of examinees’ scores to local as well as more broadly representative norms can be informative. Thus, sample size permitting, local norms are often used in conjunction with published norms, especially if the local population differs markedly from the population on which the published norms are based. In some cases, local norms may be used exclusively.

The validity of norm-referenced interpretations depends in part on the appropriateness of the reference group to which test scores are compared.

More than one reference population may be appropriate for the same test. For example, achievement test performance might be interpreted by reference to local norms based on sampling from a particular school district for use in making local instructional decisions, or to national norms for using in making comparisons to national groups. (p. 96)

 

In reading our professional standards, to me, it is clear that use of either national or local norms requires judgment about appropriateness and the generalizations one would like to make. The choice of norms is critical, especially in schools/districts that are atypical in achievement. In a high-performing community, few or no students might be identified as needing intervention. Well, that doesn’t seem to happen. In one high performing district I consulted with, the SE director said to be “if we only included students below the national norm as students with specific learning disabilities, we wouldn’t have any kids in SE.” Conversely, in a very low-performing school/district, any kid that teachers might refer might qualify because everyone is so low.

Here are my suggestions.

  • If Local Norms and National Norms Don’t Differ, Use the Norms that Work Best to Communicate.
  • IF They Differ, Use Local Norms as the PRIMARY Decision Making Metric. It’s How Teachers and Parents Think About Problems. It’s Straightforward. No Mental Gymnastics Required.
  • Local Norms Reflect a Real Distinction of What is a General Education Problem for Many Students and the Few Who May Require a More Intensive Intervention.

There is a pretty decent literature on this topic, with and without EL students.

Baker, S. K., Gersten, R., & Linan-Thompson, S. (2010). Early reading instruction and intervention with English learners: Key considerations in a multi-tiered approach. In M. R. Shiinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in a three-tier model, including RTI (pp. 501-526). Bethesda, MD: National Association of School Psychologists.

Baker, S. K., Plasencia-Peinado, J., & Lezcano-Lytle, V. (1998). The use of Curriculum-Based Measurement with language-minority students. In M. R. Shinn (Ed.), Advanced Applications of Curriculum-Based Measurement (pp. 175-213). New York: Guilford.

Peterson, K. M., & Shinn, M. R. (2002). Severe discrepancy models: Which best explains school identification practices for learning disabilities? School Psychology Review, 31, 459-476.

Shinn, M. R., & Baker, S. K. (1996). The use of Curriculum-Based Measurement with diverse learners. In L. A. Suzuki, P. J. Meller, & J. G. Ponterro (Eds.), Handbook of multicultural assessment (pp. 179-222). San Francisco, CA: Jossey-Bass.

Shinn, M. R., Collins, V., & Gallagher, S. (1998). Curriculum-Based Measurement and its use in a Problem-Solving model with students from minority backgrounds. In M. R. Shinn (Ed.), Advanced Applications of Curriculum-Based Measurement (pp. 143-174). New York: Guilford.