The official process for the development of the North Carolina Writing Assessments included in the North Carolina Testing Program is depicted in the following flowchart and descriptions of steps attached to the flowchart.  This process is to be followed in the development and implementation of state writing assessments regardless of the grade level or mode of writing.

 

Questions regarding the Testing Development Process should be directed to:

 

            NC Department of Public Instruction

            Accountability Services Division

Test Development Section

6314 Mail Service Center

Raleigh, NC  27699-6314

 

(919) 807-3774

North Carolina Testing Program

North Carolina Writing Assessment at Grades 4, 7, and 10

Test Development Process

 

Adopted CurriculumAdopt Content Standards State Board of Education
Step 1 Develop Test Specs and Blueprint T		Step 8 Range Finding – Using Field Test Responses T		Step 15 Administer Test as Operational
â		â		â
Step 2 Establish or Review Grade Level Advisory Committees		Step 9 Statewide Pilot at Assessed Grades T		Step 16 Range Finding – Using Operational Responses T
â		â		â
Step 3 Develop/Refine Rubric T		Step 10 Range Finding – Using Pilot Responses T		Step 17 Select Anchor Papers T Develop Scorer Training/Qualifying Sets
â		â		â
Step 4 Create/Review/Revise Prompts T		Step 11 Select Anchor Papers T Develop Scorer Training/Qualifying Sets		Step 18 Score Operational Tests
â		â		â
Step 5 Field Test Prompts T		Step 12 Score Pilot Tests		Step 19 Reports of Test Results Compile/Release Anchor Papers
â		â		â
Step 6 Score Field Tests		Step 13 Compile Data – Conduct Data Analysis		Step 20 Technical Documentation
â		â		â
Step 7 Compile/Review Field Test Data – Select Pilot Prompts T		Step 14 Standard Setting – Professional Standard Setting Committee		Step 21 Conduct Ongoing Professional Development

                 ____________________________________________________________

                 Involvement of teachers in the process is denoted with a superscript T.

                 This process takes approximately 36 months to complete a cycle.

 

 

North Carolina Writing Assessments

Test Development Process

 

Introduction

North Carolina tests are curriculum-based tests designed to measure the objectives found in the North Carolina Standard Course of Study (NCSCS).  The responsibility of updating the Standard Course of Study falls to the North Carolina Department of Public Instruction (NCDPI) Curriculum and School Reform Division.  Curriculum specialists, teachers, administrators, university professors, and others assist in the process of updating curricula.  Once curricula are adopted or specific tested objectives are approved by the North Carolina State Board of Education, in areas where statewide tests are required, the test development process begins.

The Standard Course of Study is reviewed for possible revisions every five years; however, test development is continuous. The NCDPI Accountability Services/Testing Section test development staff members begin developing operational test forms for the North Carolina Testing Program when the State Board of Education determines that such tests are needed. The need for new tests may result from mandates from the federal government, the North Carolina General Assembly, or the State Board of Education.

An integral part of the test development process for the North Carolina Writing Assessment at Grades 4, 7, and 10 is the input and recommendations of the writing assessment advisory committees. 

Step 1:  Develop the Test Specifications (Blueprint)

Prior to developing test specifications, it is important to outline the purpose of a test and what types of inferences (e.g. diagnostic, curriculum mastery, student accountability, school accountability) are to be made from test scores. Millman and Greene (1993, in Robert Linn, ed)^[1] offer a rationale for delineating the purpose of the test. “A clear statement of the purpose provides the overall framework for test specification, item development, tryout, and review. A clear statement of test purpose also contributes significantly to appropriate test use in practical contexts.” Using a test’s purpose as the guiding framework, NCDPI curriculum specialists, teachers, NCDPI test development staff, and other content, curriculum, and testing experts establish the test specifications for each of the grade levels assessed. In general, test specifications include the following:

1.      Number of prompts to be administered

2.      Modes of writing, content to be measured (i.e., focusing on content solely vs. the application of conventions), and grade levels at which writing will be assessed

3.      Testing time allotted

4.      Scoring criteria and rubrics

5.      Methods of reporting

6.      Determinations about the level of contractor involvement, if any

Step 2: Establish or Review Grade Level Advisory Committees

The involvement of teachers, administrators, representatives in higher education, and other stakeholders is critical to the development of the North Carolina Writing Assessment at Grades 4, 7, and 10.  Committee members serve on a staggered term basis that is reviewed every year to ensure consistent, accurate representation of North Carolina students.

Writing assessment advisory committees, one for each of the three grade levels assessed, are comprised of current, practicing, traditional classroom teachers for each of the specific grades assessed, in addition to, grade-span teachers (e.g., for grade 4, grade-span teachers would be grade 3 and grade 5 teachers) whenever possible.  Along with traditional classroom teachers, teachers of exceptional children (EC) and English as Second Language (ESL) are also represented on the committees. Other committee members include curriculum supervisors, principals, and leaders in higher education (both two and four-year college faculty).

The primary purpose of the writing assessment advisory committees is to make recommendations based not on what students can do, but what students should be expected to do as outlined in the NC Standard Course of Study for English Language Arts for the assessed grade. 

Step 3:  Develop /Refine Rubric

The scoring model is collaboratively designed and developed by the NCDPI Accountability Services/Test Development Section, the NCDPI Curriculum and School Reform, English Language Arts (ELA) staff, and the writing assessment advisory committees. The State Board of Education approves the scoring model.

Step 4: Create/Review/Revise Prompts

The performance assessment test development staff is responsible for the development and maintenance of a prompt item bank, although contractors may be involved in the process.  Prompts are developed according to the North Carolina Standard Course of Study for English Language Arts and the modes of writing defined at each grade assessed. The prompt development process follows the principles of Universal Design, with the goal of accessibility for all students.

Once the prompt item bank is completed, prompts are given an initial internal review comprised of NCDPI curriculum staff and NCDPI test development staff.  The pool is then presented to the writing assessment advisory committees to decide if a prompt warrants acceptance, acceptance with revision or modification, or rejection.

Many considerations are examined during the prompt review process including, but not limited to, appropriateness for grade level, alignment with the current curriculum, amount of writing a prompt could elicit from students, and bias.  Bias review involves scrutinizing a prompt and determining if it is acceptable for a statewide administration. This close scrutiny includes looking at bias on the basis of race, ethnicity, cultural and socio-economic status, disability, limited English proficiency, gender, religion, physical/mental attributes, and geographic location. Procedures employed during prompt development are consistent with Standard 7.4 of the Standards for Educational and Psychological Testing (AERA, APA, NCME, 1999), which states^[2]:

Test developers should strive to identify and eliminate language, symbols, words, phrases, and content that are generally regarded as offensive by members of racial, ethnic, gender, or other groups, except when judged to be necessary for adequate representation of the domain.

Step 5:  Field Test Prompts

The writing assessment advisory committees convene for the purpose of reviewing prompts and evaluating the merits or inappropriateness of prompts for future use.  The committee selects prompts for potential use in field tests. 

Schools are randomly selected to participate in field testing on the basis of its demographics, considering some of the following characteristics:  rural schools, urban schools, schools in each region of the state (western, northwest, southwest, northeast, southeast, and central), student demographic makeup of the school, socioeconomic status of the school, as well as overall student performance at the school.

Data from the field test administration are compiled and analyzed by NCDPI Accountability/Test Development Section staff.  Reliability and validity statistics are analyzed to ensure that the student responses are scored consistently. Field test results, while not shared with the public, are used as the basis for the research and development activities related to the selection of prompts. Once all data are analyzed, staff from the NCDPI Accountability/Test Development Section, NCDPI Curriculum and School Reform, English Language Arts, and the writing assessment advisory committees work collaboratively to select the prompts to be used in a future pilot or operational assessment administrations.

Step 6:  Score Field Tests

A professional scoring contractor scores student responses from the field test writing assessment administration.  The scoring contractor is responsible for hiring and training scorers who must have at least a four-year college degree, pass a written exam, and maintain acceptable reliability and validity statistics.

Step 7:  Compile/Review Field Test Data – Select Pilot Prompts

Field testing and the resulting student responses are among the most critical steps in the writing assessment development process.  Accurate field test data are necessary to facilitate the development of a functional large-scale statewide writing assessment.  The compiled data help to determine how the department can best proceed to achieve a feasible, reliable, and technically-sound pilot or operational administration of the North Carolina Writing Assessment at Grades 4, 7, and 10.

Once all field test data are analyzed, staff from the NCDPI Accountability/Test Development Section, NCDPI Curriculum and School Reform, English Language Arts, and the writing assessment advisory committees work collaboratively to select the prompts to be used in future pilot or operational assessment administrations.

Step 8:  Range finding Using Field Test Responses

Range finding is the next step after field testing has occurred and the field test data, including the student responses, have been analyzed.  The range finding process involves the scoring contractor, writing assessment advisory committees, and staff from NCDPI Accountability/Test Development Section, NCSU-TOPS, and NDCPI Curriculum and School Reform, English Language Arts. 

The writing assessment advisory committees score student responses to field tested prompts. Anchor papers are then selected to represent examples of specific score points and are referenced by the scorers during the scoring process.  They are used in conjunction with the scoring rubrics to assign specific score points to student responses.  This ensures that established standards are consistently maintained and are applied to all student responses. 

Step 9:  Statewide Pilot Administration

A statewide pilot is administered to all students in each of the assessed grades.  A statewide pilot is a replication of an actual administration and is conducted for the purpose of collecting additional data about the assessment in the simulated testing environment.  Unlike field tests, where a random sample is drawn to select those students that will be assessed, pilot assessments are administered to all eligible students in the state on the same date the operational assessment would be administered.  The pilot strives to create, as nearly as possible, a duplicate assessment and testing environment so differences between the pilot assessment and the operational assessment can be reduced and potentially eliminated.  No alternate prompts are given during field testing or pilot administrations.  Test results are reported from the statewide pilot administration on a delayed schedule.

Step 10:  Range Finding and Selection of Anchor Papers Using Pilot Responses

Range finding is conducted again using student responses from the pilot administration. Anchor papers are selected and are used in the training and the process of qualifying scorers. 

Step 11:  Select Anchor Papers – Develop Scorer Training/Qualifying Sets

Student responses from the pilot administration are scored by the writing assessment advisory committees from which anchor papers are selected.  Anchor papers represent examples of particular score points and are referenced by the scorers during the scoring process.  They are used in conjunction with scoring rubrics to assign specific score points.  Scorers reference these anchor papers throughout the scoring process.  This ensures that established standards are consistently applied to all student responses. 

Step 12:  Score Pilot Test

A professional scoring contractor scores the statewide pilot writing assessment student responses. The scoring contractor is responsible for hiring and training scorers who must have at least a four-year college degree, pass a written exam, and maintain acceptable reliability and validity statistics.

Step 13:  Compile Data – Conduct Data Analysis

NCDPI Test Development staff and NCSU-TOPS staff electronically monitor the data that are collected throughout the course of the scoring process.  These data are compiled and analyzed for several purposes, including but not limited to, validity, reliability, and frequency (student score point) distributions.  Inter-rater reliability and validity are expected to be maintained at 70 percent, which represent industry standards.

Step 14:  Standard Setting by a Professional Standard Setting Process

A team of professionals that includes, but is not limited to, statisticians, psychometricians, practitioners, and testing experts make recommendations concerning the cut scores that define levels of proficiency.  Standards are recommended to the State Board of Education after thoroughly analyzing all relevant pilot administration data and considering a variety of student responses representing all score points on the score scales.

Once established, the standards do not change while the existing curriculum is in place unless directed by the State Board of Education, and are applicable across prompt variance and administration years within a curriculum cycle.

Step 15:  Administer Test As Operational

For an operational administration, the tests are administered statewide in grades 4, 7, and 10 following all policies of the State Board of Education, including the North Carolina Testing Code of Ethics. Standardized test administration procedures must be followed to ensure the validity and reliability of test results.  Eligible students with disabilities and students identified as Limited English Proficient may use approved accommodations when taking the tests.

Step 16:  Range Finding Using Operational Responses

Range finding is conducted again using student responses from the operational administration.  Anchor papers are selected and are used in the training and qualifying process of scorers. 

Step 17:  Select Anchor Papers – Develop Scorer Training/Qualifying Sets

Student responses from the operational prompt that was selected by the advisory committees are assembled and anchor papers are selected. Anchor papers represent examples of particular score points and are referenced by the scorers during the scoring process.  They are used in conjunction with scoring rubrics to assign specific score points to student responses.  Scorers reference these anchor papers throughout the scoring process.  This ensures that established standards are consistently maintained and applied to all student responses. 

Step 18:  Score Operational Test

A professional scoring contractor scores student responses to the statewide writing assessment. The scoring contractor is responsible for hiring and training scorers who must have at least a four-year college degree, pass a written exam, and maintain acceptable reliability and validity statistics.

Step 19:  Reports of Test Results -Compile/Release Anchor Papers

Writing reports that provide the statistical data are compiled and made available to the public on the North Carolina Department of Public Instruction website (www.ncpublicschools.org/accountability/testing).  Anchor papers from the current administration are released through a variety of media such as, statewide training, the NCDPI website, and videoconferencing. Reports of student performance are provided at the various levels: student, classroom, school, LEA, region, and state.  Disaggregated results by various subgroups are also made available to the public.

 Step 20:  Technical Documentation

The NCDPI also prepares a technical report that includes information on the characteristics of the assessments and the procedures used to develop the writing assessments, administer the assessments, collect data from the assessments, analyses of data from the assessment, as well as student participation and student performance information.

Step 21:  Conduct Ongoing Professional Development

Professional development is an integral part of the writing assessment test development process. Individuals assigned to conduct professional development provide North Carolina educators and the public with information regarding the scoring model, test development process, the scoring process, and the assessment timelines.