Skip to content

RAITRedundant Assessment Integrity Tool

A student’s score shouldn’t depend on which TA grades it.

When several TAs divide an assessment, each applies the rubric in their own way, and differences in strictness are hard to see. RAIT has every grader score the same small sample first, measures grader agreement, and shows the instructor where graders differ before the remaining responses are graded.

AI second review is optional and requires instructor approval for each question.

Open the demoSee how RAIT works

How RAIT works

The examples below follow one exam with 120 responses, an instructor and three TAs.

  • InstructorGrades the shared sample and adjudicates disagreements
  • Teaching assistantsGraders
  • RAITSelects the sample, compares scores and flags disagreements
  • AI (optional)Second review on approved questions

Without RAIT

Without RAIT: exam responses are divided among three TAs. Each TA grades 40 responses, and they do not assess any of the same work. The grades are then combined. Without shared responses, the instructor cannot directly compare how graders applied the rubric. In the example, TA 2’s average score is 2.8 out of 5, compared with 3.9 and 4.1 for the other TAs, and the averages alone cannot show whether TA 2 is stricter or received weaker responses.

With RAIT

Every grader first assesses the same small sample. RAIT compares their results, and the instructor resolves differences before the remaining work is graded.

With RAIT, a flowchart in seven steps. Step 1: the instructor adds questions and rubrics and imports student responses, and RAIT replaces student names with anonymous codes. Step 2: for each question, RAIT randomly selects a shared sample of responses. Step 3: the instructor and TA 1, TA 2 and TA 3 each score the same sample independently; TAs do not see other graders’ scores. Step 4: RAIT calculates grader agreement and flags large differences, and the instructor discusses them with the TAs, clarifies how the rubric applies and adjudicates disputed interpretations. In the example response, the instructor, TA 1 and TA 3 give 4 out of 5 and TA 2 gives 2, so TA 2’s score is marked as different. Step 5 is a decision: is the question ready for regular grading? If not, the instructor clarifies how the rubric applies and graders score a new shared sample, returning to step 2. If the instructor approves, step 6: RAIT divides the remaining responses among the TAs, one TA per response. Optional AI branch, for eligible questions only: AI scores the shared sample and RAIT compares its scores with the instructor’s. If AI meets the requirements and the instructor approves, AI provides a second review of TA-graded responses and significant differences are flagged. Otherwise the question is graded by people only. Step 7: RAIT flags significant disagreements, the instructor resolves each one and can adjust the score, then confirms final scores before export. AI scores never replace a grader’s score.

Step 1

Import the assessment

Who: Instructor and RAIT

The instructor adds the questions and rubrics and imports student responses. RAIT replaces student names with anonymous codes for grading.

Step 2

Select a shared sample

Who: RAIT

For each question, RAIT randomly selects responses that every participating grader will assess. The instructor sets the sample size.

Step 3

Grade independently

Who: Instructor and TAs

The instructor and all participating TAs score the same sample using the rubric. TAs do not see other graders' scores.

  • Instructorgives a Score
  • TA 1gives a Score
  • TA 2gives a Score
  • TA 3gives a Score

Step 4

Compare grader agreement

Who: RAIT and Instructor

RAIT calculates grader agreement on the sample and flags large differences. The instructor discusses the differences with the TAs, clarifies how the rubric applies and adjudicates disputed interpretations.

ExampleOne response from the sample, out of 5
  • Instructor4/5Reference score
  • TA 14/5 Same score
  • TA 22/5 Score differs
  • TA 34/5 Same score

One response shows a difference, not a pattern. RAIT reports grader agreement across the whole sample.

The instructor reviews differences and clarifies the grading standard.

Step 5Decision: Ready for regular grading?

Yes, instructor approves

No Not ready

Clarify and repeat calibration

The instructor clarifies how the rubric applies. Graders score a new shared sample.

Step 6

Grade the remaining responses

Who: TAs

RAIT divides the remaining responses among the graders. Each of these responses is scored by one TA.

OptionalEligible questions only

Test AI on the shared sample

AI scores the calibrated sample. RAIT compares its scores with the instructor’s and checks for consistent over- or under-scoring and frequent low confidence.

Passing on a small sample does not show that AI will score every response reliably. It only allows the instructor to approve a second review.

Decision: Meets AI requirements and instructor approves?

Yes Approved

AI second review

AI scores TA-graded responses to this question. Significant differences from the TA’s score are flagged for the instructor.

No Not approved

No AI review

The question is graded by people only.

Step 7

Review and finalize

Who: RAIT and Instructor

RAIT flags significant disagreements. The instructor resolves each one and can adjust the score, then confirms the final scores before export.

AI scores are stored separately. They never replace a grader’s score. Only the instructor can change a score during adjudication.

Why it matters

Divided grading hides differences between graders

In large courses, assessments are commonly divided among teaching assistants. Everyone receives the same rubric, but that doesn’t mean everyone interprets it the same way.

One grader may be consistently stricter. Another may give more credit for partially correct answers. A poorly defined rubric criterion may produce disagreement across the whole team. When no response is graded by more than one person, none of this can be measured.

Grading without and with RAIT
AspectWithout RAITWith RAIT
Shared responsesNone. Each response has one grader.A sample that every grader scores, for each question.
Grader agreementNot measured.Calculated for each grader against the instructor's scores on the sample.
Unclear rubric criteriaOften found after grades are released.Found during calibration, before regular grading starts.
Instructor reviewAd hoc.Directed to flagged disagreements.
RecordOne score per response.Every grader's score, flags and instructor adjustments, kept in the export.

Optional AI second review

After meeting question-specific calibration requirements and receiving instructor approval, AI can provide a second review of eligible responses. Significant differences are flagged for instructor review.

AI never overrides a human score. The TA’s score stands unless the instructor adjudicates the flag and adjusts it.

A calibration sample is small. Meeting the requirements on it does not show that AI will score every response reliably, which is why AI only provides a second review and why its disagreements go to the instructor. If AI does not meet the requirements, the question is graded by people only. Calibration and review work the same way with AI turned off.

Requirements for each question

  1. 1

    The question is eligible

    The instructor marks which questions AI may assess. RAIT only allows AI on questions rated as requiring moderate judgment.

  2. 2

    Human calibration is complete

    The instructor has approved calibration for the question and the human graders agree closely enough on the sample. With a solo instructor, the instructor must have scored the whole sample.

  3. 3

    AI matches the instructor

    AI scores the same sample of at least five responses. Its scores must be close to the instructor's on average, must not lean consistently higher or lower, must not miss badly on any one response, and must not report low confidence too often.

  4. 4

    The instructor approves

    Approval applies to one question and can be withdrawn at any time. It lapses if the rubric, the sample, the scores or the AI model change.

Use cases

Written answers graded against a rubric

RAIT is most useful when answers require interpretation and several graders apply the same rubric.

Courses with several TAs

Assessments where grading is divided among teaching assistants who share one rubric.

Accounting and business cases

Answers that involve explanation, recommendations, risk identification or professional judgment.

Short written answers

Questions where rubric criteria leave room for different interpretations.

Less useful for multiple-choice or exact numerical answers, where an answer key already keeps grading consistent.

Instructor oversight

Decisions about scores stay with the instructor

Adjudication: The instructor resolves flagged disagreements, can adjust a score with a recorded reason, and confirms the final scores before export.

Grader agreement: Measured on the shared sample for each question, not assumed.

AI by question: AI second review is approved one question at a time and never replaces a grader's score.

Records: Grading, AI and review actions are logged. The export keeps each grader's score next to the final score.

Live demo

Try the demo

Sign in as the demo professor or a demo TA. You can set up an assessment, grade the shared sample, compare grader agreement and resolve flagged disagreements.

The demo uses shared accounts with no passwords, and its AI is a simulated grader rather than a real model. Do not enter real student work or personal information.

Open the demo

RAIT is in early access for instructors, course coordinators and departments.

Choose a demo role

Demo accounts are pre-made, so no sign-up is needed.

The professor account manages sessions, calibration and AI settings. TA accounts open a grading queue.