How to Calculate Overall Unit Agreement

The agreement column on the Data Page displays a score between 0 and 1, inclusive, indicating contributor consensus across all questions that aggregate responses for that unit. The calculation for the overall agreement is as follows, with an example scenario below and some important notes below.

For each question (field) with aggregation, the number of instances of the most common answer is divided by the number of total responses to that question, producing a score between 0 and 1 for that question. That score is added together with the scores for the other aggregated questions. The total score is then divided by the number of aggregated questions, producing the overall agreement score displayed.

Example

Let’s say a unit has 2 questions (fields), both with aggregated responses, and 4 contributors submit judgments for that unit. Here are their responses:

Question 1:

• Contributor 1: B
• Contributor 2: B
• Contributor 3: B
• Contributor 4: D

Question 2:

• Contributor 1: A
• Contributor 2: C
• Contributor 3: C
• Contributor 4: B

Question 1: the most common response was B, which 3 of the 4 contributors submitted. Total count of most common response (3) divided by the number of total responses (4) = .75

Question 2: the most common response was C, which 2 of the contributors submitted. Total count of most common response (2) divided by the number of total responses (4) = .5

(.75 + .5) divided by the total number of aggregated questions (2) = .63. As a result, this unit would have an agreement score of .63

Important notes 

Agreement factors in all questions (fields) for each unit that have aggregated responses, based on if the question is required and if it is logic-dependent, as explained below.

Essentially, unit agreement = the sum of field agreements / total number of fields

If the field is required:

• If all responses are null:
  • The field agreement = 0
• If at least one response is not null: 
  • The field agreement = count of top response / total number of responses
    • If the field is logic dependent:

      • The field agreement = count of top_response / total number of logic dependent responses

• Then, increment total fields 

However, if the field is not required:

• If all responses are null:
  • Ignore field (the field agreement is null) and do not increment total fields
• If at least one response is not null and the field is logic dependent:
  • The field agreement = count of top response / total number of logic dependent responses (does not include null responses)
• If at least one response is not null and the field is not logic dependent:
  • The field agreement = count of top response / total number of responses (includes null responses
• Then, increment total fields