Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

2 watching

Forks

Repository files navigation

MeasuringFairnessWithBiasedData

This repository contains the code and some artifacts to reproduce the paper:

Sarah Schröder, Alexander Schulz, Ivan Tarakanov, Robert Feldhans and Barbara Hammer. Measuring fairness with biased data: A case study on the effects of unsupervised data in fairness evaluation. Accepted at IWANN 2023 and published in the LNCS proceedings.

Requirements

The following packages are necessary to run the notebooks:

  • numpy
  • pandas
  • datasets
  • scipy
  • plotly
  • kaleido
  • torch
  • scikit-learn
  • jupyter

In addition we used a custom wrapper for huggingface embeddings.

Dataset

The paper investigates the quality of the BIOS dataset. The dataset is not published due to copyright and data protection reasons. However, a crawler to download the dataset can be found here. The results from our review can be found in data/BIOS_LABELS_v1.csv and data/BIOS_LABELS_v2.csv. It contains no sensitive information such as the biographies themselves or names, only the original ids, labels and paths as well as our review results that can be merged with the BIOS dataset.

Version differences:

  • v1 is the version used in the published paper (10 of the original classes)
  • v2 adds the software enginner and model class (12 of the original classes)

Columns from the original dataset:

  • path
  • gender
  • start_pos
  • URI
  • auto_raw_title (renamed from raw_title)
  • auto_title (renamed from title)

Columns from our review:

  • review (1 if the sample was reviewed, otherwise 0)
  • raw_titles (a list of raw_titles determined by the annotator)
  • titles (a list of titles derived from the raw titles and the title_lookup.json, including only occupations used in the paper)
  • valid (1 if the annotator decided the sample was a valid biography (see paper) otherwise 0 (or -1 for samples not reviewed))
  • style_valid (1 if the annotator decided the text style matched a biography, in addition to being valid, otherwise 0 (or -1 if not reviewed/ invalid))

Reproducing the paper

Reproducing the review process

We used filter_dataset.ipynb to obtain a selection of approx. 20.000 candidate samples for the review. The code for merging the reviewed artifacts and creating the dataset patches can be found in merge_review_artifacts.ipynb. Note that the later one used artifacts from the review (containing names and bios) and the crawled dataset which are not published, but we are willing to share them with fellow researchers.

Patch review labels to dataset

If you downloaded the BIOS dataset yourself, use add_changes_to_dataset.ipynb to add our labels to the dataset. Move the downloaded BIOS.pkl to the data/ directory or adapt the path in the notebook.
If you prefer the huggingface version of the dataset instead, use huggingface_patch.ipynb.

Experiments from the paper

To reproduce our experiments, you first need to download the dataset yourself and merge our labels (see above). Then you can use data_review.ipynb to run our experiments. During the experiments we trained 30 BERT models, so it may take a while to run.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages