Skip to content

Transformer

The eds.transformer component is a wrapper around HuggingFace's transformers library. If you are not familiar with transformers, a good way to start is the Illustrated Transformer tutorial.

Compared to using the raw Huggingface model, we offer a simple mechanism to split long documents into strided windows before feeding them to the model.

Windowing

EDS-NLP's Transformer component splits long documents into smaller windows before feeding them to the model. This is done to avoid hitting the maximum number of tokens that can be processed by the model on a single device. The window size and stride can be configured using the window and stride parameters. The default values are 512 and 256 respectively, which means that the model will process windows of 512 tokens, each separated by 256 tokens. Whenever a token appears in multiple windows, the embedding of the "most contextualized" occurrence is used, i.e. the occurrence that is the closest to the center of its window.

Here is an overview how this works to produce embeddings (shown in red) for each word of the document :

Transformer windowing

Examples

Here is an example of how to define a pipeline with a Transformer component:

import edsnlp

nlp = edsnlp.blank("eds")
nlp.add_pipe(
    "eds.transformer",
    name="transformer",
    config=dict(
        model="prajjwal1/bert-tiny",
        window=128,
        stride=96,
    ),
)

You can then compose this embedding with a task specific component such as eds.ner_crf.

Parameters

PARAMETER DESCRIPTION
nlp

The pipeline instance

name

The component name

model

The Huggingface model name or path

window

The window size to use when splitting long documents into smaller windows before feeding them to the Transformer model (default: 512 = 512 - 2)

DEFAULT: 128

stride

The stride (distance between windows) to use when splitting long documents into smaller windows: (default: 96)

DEFAULT: 96

max_tokens_per_device

The maximum number of tokens that can be processed by the model on a single device. This does not affect the results but can be used to reduce the memory usage of the model, at the cost of a longer processing time.

DEFAULT: 16384

span_getter

Which spans of the document should be embedded. Defaults to the full document if None.

DEFAULT: None