Getting started

eds-scikit is a tool to assist data scientists working on the AP-HP’s Clinical Data Warehouse. It is specifically targeted for OMOP-standardized data to:

• Ease access and analysis of data

• Allow a better transfer of knowledge between projects

• Improve research reproduciblity

As an example, the following figure was obtained using various functionalities from eds-scikit.

Quick start

Installation

You can install eds-scikit by cloning the git repository:

$ pip install eds-scikit
---> 100%
color:green Successfully installed eds_scikit !

import eds_scikit
spark, sc, sql = eds_scikit.improve_performances()

A first example: Merging visits together

Let's tackle a common problem when dealing with clinical data: Merging close/consecutive visits into stays. As detailled in the dedicated section, eds-scikit is expecting to work with Pandas or Koalas DataFrames. We provide various connectors to facilitate data fetching, namely a Hive connector and a Postgres connector

from eds_scikit.io import HiveData

data = HiveData(DB_NAME)
visit_occurrence = data.visit_occurrence # (1)

from eds_scikit.io import PostgresData

DB_NAME = "my_db"
SCHEMA = "my_schema"
USER = "my_username"
data = PostgresData(DB_NAME, schema=SCHEMA, user=USER) # (1)
visit_occurrence = data.visit_occurrence # (2)

- It follows the OMOP format
- It is a Pandas or Koalas DataFrame

import pandas as pd
visit_occurrence = pd.read_csv("./data/visit_occurrence.csv")

# Importing the desired functions:

from eds_scikit.period.stays import merge_visits, get_stays_duration

# Calling the first function: computing stays

visit_occurrence = merge_visits(visit_occurrence)

As you can see, the function added a STAY_ID concept, grouping visits together

# Calling the second function: computing stays duration
stays = get_stays_duration(visit_occurrence, missing_end_date_handling="coerce")

Here, each stay duration was calculated, dealing with potential overlaps and inclusions.:

visit_occurrence = merge_visits(
   visit_occurrence,
   remove_deleted_visits=True,
   long_stay_threshold=timedelta(days=365),
   long_stay_filtering="all",
   max_timedelta=timedelta(hours=24),
   merge_different_hospitals=False,
   merge_different_source_values=["hospitalisés", "urgence"],
)

stays = get_stays_duration(
   visit_occurrence,
   algo="sum_of_visits_duration",
   missing_end_date_handling="coerce"
)

A word about AP-HP

Specifics of AP-HP CDW

eds-scikit was developped by AP-HP's Data Science team with the help of Inria's Soda team. As such, it is especially well fitted for AP-HP's Data Warehouse. In this doc, we use the following card to mention information that might be useful when using eds-scikit with AP-HP's data:

EDS-NLP

Also, a rule-based NLP library (EDS-NLP) designed to work on clinical texts was developped in parallel with eds-scikit. We decided not to include EDS-NLP as a dependency. Still, some functions might require an input à la note_nlp: For instance, the current function designed to extract consultation dates from a visit_occurrence car work either on structured data only or with dates extracted in text and compiled in a DataFrame.

You are free to use the method of your choice to get this DataFrame, as long as it contains the necessary columns as mentionned in the documentation. Note that we mention with the following card the availability of an EDS-NLP dedicated pipeline:

Contributing to eds-scikit

We welcome contributions! Fork the project and create a pull request. Take a look at the dedicated page for details.

Citation

If you use eds-scikit, please cite us as below.

@misc{eds-scikit,
    author = {Petit-Jean, Thomas and Remaki, Adam and Maladière, Vincent and Varoquaux, Gaël and Bey, Romain},
    doi = {10.5281/zenodo.7401549},
    title = {eds-scikit: data analysis on OMOP databases},
    url = {https://github.com/aphp/eds-scikit}
}