Tutoring News Post
Are the SAT and ACT "equated" beforehand, "scaled/curved" after the test, or both? Addressing one of the College Board's Most Persistent Lies
 Are the SAT and ACT exams truly "equated" beforehand only? Is the scoring curve for each test actually "predetermined," as so many seem to think, and the test makers would like us to believe? The answer is a resounding no — so long as the exam is a new test form that has never before been "nationally administered" — even though the College Board falsely and explicitly claims otherwise, and ACT Inc. is purposefully vague on the topic. That each edition of the SAT or ACT is almost perfectly equal in difficulty, and that every scoring curve is predetermined with complete accuracy, is certainly what the College Board and ACT Inc. would like us to believe, but historically, there has also been what's called a "scaling" a.k.a. "smoothing" a.k.a. "curve fitting" process (more popularly known as "curving") after the test is nationally administered for the first time, when a significantly higher amount of student performance information is collected than in the initial equating effort, and the College Board / ACT gains access to a much more sophisticated degree of data about the test. Thus, additional "fine-tuning" adjustments can be made post-test, and these adjustments are indeed made to the final scoring scale before student scores are released. Recent SATs and PSATs (which never re-use test forms) with extra-harsh curves such as the October 24, 2018 PSAT (-1 710 on Math!) and the May 4th, 2019 International SAT serve as further evidence that the College Board doesn't know exactly what the scoring curve will be until after the test is given to to a national sample of students, instead of only a small pretest sample group for equating purposes. Why would any reasonable person actually believe the College Board when it claims that ridiculous curves such as this one — where just one math mistake resulted in a 50-point score drop — are predetermined, when on most SATs and PSATs, a single mistake only costs about 10 points? That's not a "small" difference in difficulty, College Board: it's a huge one, and it was clearly unintentional. Please note that after the first "national adminstration" (in other words, not a "pretest" or "experimental" version, but the main version) of a test form, the College Board / ACT have historically used the exact same curve for any future administrations of that particular form, regardless of student performance on those future administrations. This means that if the test form is leaked after the first administration, and some students have an unfair advantage as a result, for example (yes, this has happened with the SAT multiple times), that the cheaters' test results should NOT unfairly affect the curve (aka "ruin the curve") for those who did not cheat, because the curve has already been set in stone and will no longer change due to student performance. As you can learn from reading the literature below, this type of intentional pre-test and post-test manipulation of SAT scores (whether it is called recentering, redistributing, equating, smoothing, scaling, or curving) has been going on for decades. In addition, today's vast computer processing power means that there is no longer a limit on the amount of test-day data that can be collected and analyzed in order to establish a fair scoring curve for each exam. Another term for a "fair scoring curve" is "equipercentile equating." https://web.archive.org/web/20211011080138/https://www.ets.org/Media/Research/pdf/RR-02-04-Dorans.pdf https://web.archive.org/web/20210414112633/https://www.ets.org/Media/Research/pdf/RR-10-29.pdf https://www.ets.org/Media/ https://files.eric.ed.gov/fulltext/ED544690.pdf https://www.act.org/content/dam/act/unsecured/documents/R1837-ACT-equating-stability-2020-11.pdf https://www.act.org/content/dam/act/unsecured/documents/ACT_Technical_Manual.pdf An excerpt: "If you want to do equipercentile equating, and you don’t have a good way to smooth the score distributions, there is an alternative. You can perform an equipercentile equating based on the observed distributions, and then smooth the equating relationship. (Some equating experts refer to this approach as “post-smoothing.")"  Source: Wikipedia I would also encourage you to read the Wikipedia definition of equating, which states clearly that equating is a pre-test process for raw scores, and that scaling/curving is a post-test process for scaled scores: "Unlike with item response theory, equating based on classical test theory is somewhat distinct from scaling. Equating is a raw-to-raw transformation in that it estimates a raw score on Form B that is equivalent to each raw score on the base Form A. Any scaling transformation used is then applied on top of, or with, the equating." To summarize: new SAT and ACT test forms are not only Equated Pre-Test using small samples of around 3,000 students (the content is controlled, compared to previous tests, and standardized to create a test of roughly average difficulty), but also Curved/Smoothed/Scaled Post-Test (raw to scaled score conversion tables adjusted) afterwards based upon a much larger, complete analysis of student results on that particular test: if it turns out that the test was harder than expected, then the curve might bump your score up by a few points, and vice-versa, so that the results of the test more strongly resemble a "smooth" bell-shaped curve. This curving process only happens once--after the first "national administration" of that particular exam. Can I prove definitively that the ACT and College Board engage in curve-fitting/post-smoothing for new test forms? No, because ACT and the College Board haven't yet explicitly admitted to this practice. However, we have strong historical evidence (see above), as well as evidence from both the SAT and ACT literature (sources: SAT Technical Manual) / ACT Technical Manual) that curving has been an unstated/largely ignored yet essential part of the "equating" umbrella all along, and that this post-test process is an entirely separate process from any pre-test calculations. If you read the SAT technical manual, for example, then you will see the term "operational administration," which refers to a national SAT test date when scaling data is collected from a sample group, vs. a "pretest," when equating data is collected prior to the national administration of the test. The ACT also freely admits to using post-test data from national test dates in its technical manual and elsewhere*, warning students that February and October ACT scores could take several extra weeks because of post-test "equating." Not coincidentally, you will notice that the February and October ACTs are never TIR (Test Information Release) exams, because if they were, then the exam form would be re-used, meaning that ACT Inc. would simply use the original scoring curve — which is why ACT scores from TIR test dates are returned much faster. Finally, it is also important to consider that curved tests have a bad reputation, so the College Board and ACT have reason to mislead us in this regard, and to instead lean on the much vaguer "equating" term instead. However, if changes to the scoring scale are made after the test, then it's not really equating: it's curving! As of the debut of the US Digital SAT in March 2024, the College Board and its many online minions continue to insist on playing semantic games by calling a pony a horse—but this doesn't change the fact that it's still a pony.  Source: https://www.youtube.com/watch?v=MDoc0qoKSHA Do you see the logical fallacy here? The College Board can't have it both ways, claiming at the same time that "even with careful planning there may be small differences in the difficulty of each of those tests," but also that other students' performance doesn't affect your score — because the only way to determine a test's difficulty is to measure overall student performance on that particular exam form. Yes, the College Board tries to make every test equal in difficulty — the exact reason for which the equating process is intended — but the above quotation is essentially a confession: after the exam, the Board often discovers that exams were easier or harder than anticipated, hence the need for a secondary, post-test curving process that further fine-tunes the scoring scale. In conclusion: yes, the SAT is curved—no matter how much the College Board has tried to mislead us on the matter. SAT and ACT scores are indeed curved after the test, based on other students' performance — at least during the initial "national" administration of each exam form — and yes, the College Board is straight-up lying when it repeatedly claims otherwise. And yes, the new, adaptive digital SAT aka DSAT is also curved after the exam—which is why exam scores still take weeks to be returned, despite the switch from paper to computer / tablet testing. -Brian *Source = https://www.act.org/content/act/en/products-and-services/the-act/scores.html
|
Subscribe to our RSS Feed
