Document Classification Metrics

The eds.doc_classification metric evaluates document-level classification by comparing the predicted and gold values of one or several Doc._ attributes. For each attribute it computes Precision, Recall, F1, number of true positives (tp), number of gold instances (support) and number of predicted instances (positives), per label and micro-averaged under micro_key. The per-label scores are also macro-averaged under macro_key.

The metric adapts to the type of each attribute. An attribute holding a single value (as predicted by a single-label head of eds.doc_classifier) contributes one item per document, so its micro scores amount to the accuracy of the head. An attribute holding a list or a set of values (as predicted by a multi-label head) contributes one item per predicted label, and is therefore scored as a multi-label task.

A document whose gold value is None for a given attribute is ignored for that attribute, which lets partially annotated corpora be scored as-is.

import edsnlp
from edsnlp.metrics.doc_classification import DocClassificationMetric
from spacy.tokens import Doc

for attr in ("dp", "das"):
    if not Doc.has_extension(attr):
        Doc.set_extension(attr, default=None)

nlp = edsnlp.blank("eds")
gold, pred = nlp("Compte rendu."), nlp("Compte rendu.")
gold._.dp, gold._.das = "C34", ["E11", "I10"]
pred._.dp, pred._.das = "C34", ["E11", "N18"]

metric = DocClassificationMetric(label_attr=["dp", "das"])
scores = metric([gold], [pred])
print(scores["dp"]["micro"]["f"])
# Out: 1.0
print(scores["das"]["micro"]["f"])
# Out: 0.5