For reviewers

Every promise is rated by two people working alone. This guide covers what reviewers do, start to finish.

Open the rating tool

Before you start

• Work alone. Don't discuss ratings with anyone until both reviewers have sent their files.
• Rate the evidence, not the person. The tool hides each official's name and party. Don't look them up.
• Read the rules. The How we rate page is the short version. The codebook is the full one and settles edge cases.
• Get a free GitHub account. You need one to send in your ratings. You don't need to know git.
• Check you were drawn. When there are more than two reviewers, a public random draw decides who rates which promise. The tool shows only the promises drawn for you.

1. Set up

1. Open the rating tool.
2. Type your reviewer ID: lowercase letters, numbers or underscores, such as awillems. Use the same ID every time.
3. Pick a task:
   • Check the promise list, before inauguration.
   • Rate what happened, after inauguration, at each review date: day 100, one year, after each legislative session and end of term. Enter the review date. Use only evidence dated on or before it.
4. The first time, the tool asks if you already have a ratings file. If you're new, choose This is my first time. If you rated before, choose Load my ratings file first, or your earlier ratings will be lost when you send.

2. Read the promise

Each card shows the promise ID, office, state and policy area, whether the official can act alone, the exact quote, and links to the saved copy and the live page. Some cards carry a flag, such as "check wording". Look at those closely.

The saved copy is the record. Judge the quote against it, not against the live page, which may have changed.

3a. Task: check the promise list

Answer three questions for each promise:

1. Keep, drop or merge? Drop it if the words aren't the candidate's own, were said after Election Day, or are too vague to test. Merge it if another promise says the same thing, and give that promise's ID. If you drop a promise, the note must say why.
2. Is it testable? Could someone later show it was kept or broken?
3. Does the quote match the saved copy word for word? Answer No if any word differs, or if an ellipsis removes an amount, date, condition or limit.

3b. Task: rate what happened

1. Outcome: Kept, Partially kept, In progress, Not yet rated or Broken. Judge against the candidate's own wording: amounts, dates and scope. A missed deadline in the promise makes it Broken, even if work continues. If you can't tell In progress from Not yet rated, choose Not yet rated.
2. Effort: No action, Talk only, Formal start, Spent capital, Every option or Undermined. Rate the highest level reached. Undermined overrides everything. For senators, effort is asked first, because a senator can rarely pass a law alone.
3. Main obstacle, if it wasn't kept: for example the opposing party, the official's own party, courts, the federal government, the budget or a change of position.
4. Evidence links, one per line, most authoritative first: enacted text, then official records, then government releases. Use news only for effort and obstacles. At least one link is required unless you choose Not yet rated.
5. Note, optional: if someone blocked the promise, name them and add a link. Add anything else the other reviewer should know.

4. Save as you go

• Save and next won't save until every required answer is filled in. It tells you what's missing.
• Shortcuts: number keys pick an option in the first unanswered question. Ctrl + Enter (Cmd + Enter on a Mac) saves.
• Your work is kept in this browser. To switch computers, select Download my ratings, then Load my ratings file on the new one.
• To change an answer, turn off Unfinished only and go back to it. The change is saved as a new rating. Old ratings are never deleted.

5. Send your ratings

1. When you've finished, select Download my ratings.
2. Select Open the send form. It opens on GitHub with your ID filled in.
3. Drag the downloaded file into the box and submit. Don't edit the file.

A bot checks the file within a few minutes and comments on your issue. If something is wrong, the comment says what. Fix it, edit the issue and attach the new file. The check runs again.

Your file is public once you send it, so send it only when you've finished the round.

What happens next

• Where you and the other reviewer agree, that rating is published.
• Where you disagree, a third person settles it with a written reason. They can't be either of you.
• One reviewer's rating is never published on its own. All ratings, including disagreements, are public in the code and data.