SpaCy representations

EDS-NLP uses spaCy to represent documents and their annotations. You will need to familiarise yourself with some key spaCy concepts.

The Doc object

The doc object carries the result of the entire processing. It's the most important abstraction in spaCy, hence its use in EDS-NLP, and holds a token-based representation of the text along with the results of every pipeline components. It also keeps track of the input text in a non-destructive manner, meaning that doc.text == text is always true.

To obtain a doc, run the following code:

import edsnlp  # 

# Initialize a pipeline
nlp = edsnlp.blank("eds")  # 

text = "Michel est un penseur latéral."  # 

# Apply the pipeline
doc = nlp(text)  # 

doc.text
# Out: 'Michel est un penseur latéral.'

# If you do not want to run the pipeline but only tokenize the text
doc = nlp.make_doc(text)

# Text processing in spaCy is non-destructive
doc.text == text

# You can access a specific token
token = doc[2]  # 

# And create a Span using slices
span = doc[:3]  # 

# Entities are tracked in the ents attribute
doc.ents  # 
# Out: ()

We just created a pipeline and applied it to a sample text. It's that simple.

The Span objects

Span of text are represented by the Span object and represent slices of the Doc object. You can either create a span by slicing a Doc object, or by running a pipeline component that creates spans. There are different types of spans:

• doc.ents are non-overlapping spans that represent entities
• doc.sents are the sentences of the document
• doc.spans is dict of groups of spans (that can overlap)

import edsnlp, edsnlp.pipes as eds

nlp = edsnlp.blank("eds")

nlp.add_pipe(eds.sentences())  # 
nlp.add_pipe(eds.dates())  # 

text = "Le 5 mai 2005, Jimothé a été invité à une fête organisée par Michel."

doc = nlp(text)

The doc object just became more interesting!

# ↑ Omitted code above ↑

# We can split the document into sentences spans
list(doc.sents)  # 
# Out: [Le 5 mai 2005, Jimothé a été invité à une fête organisée par Michel.]

# And list dates spans
doc.spans["dates"]  # 
# Out: [5 mai 2005]

span = doc.spans["dates"][0]  # 

SpaCy extensions

We can add custom attributes (or "extensions") to spaCy objects via the _ attribute. For example, the eds.dates pipeline adds a Span._.date extension to the Span object. The attributes can be any Python object.

# ↑ Omitted code above ↑

span._.date.to_datetime()  # 
# Out: DateTime(2005, 5, 5, 0, 0, 0, tzinfo=Timezone('Europe/Paris'))