Skip to content

HPA Subcellular dataset

Do you want to explore/work with the HPA Subcellular atlas images? Tired to get one-liners as answers when you ask for it? Looking for some more comprehensive information that what you get when you are redirected to a github URL? Don't worry: some of us got your back!

This growing document is going to guide you through all the aspects of the HPA Subcellular dataset: what it is, where you can get/access it and how can you use it or generate your own data from it.

Subcellular atlas

You can find and play with the HPA Subcellular atlas by entering the "Subcellular" section at the HPA website: https://www.proteinatlas.org/ . There you are going to find a lot of fascinating information (particularly if you have biology knowledge) that we are not going to address here. What we are interested in is the dataset that feeds that website: IF images and a huge csv collecting all the information gathered about them.

How to access the HPA Subcellular data from the HPA

IMPORTANT: you don't need to access directly the HPA data, all relevant files have been already downloaded and are placed in our lab storage. The following information is more to understand where it all comes from.

To directly access the data from the HPA servers you need to request access to: - https://lims.proteinatlas.org/: the HPA annotation software. You have to ask permission to Emma Lundberg and contact the HPA personnel (Kalle von Feilitzen). You also NEED to be connected to the SciLifeLab VPN to access LIMS. - https://if.proteinatlas.org: you need to ask separate permission to access the files. Again, contact the HPA personnel (Kalle von Feilitzen). You DON'T need to be connected to SciLifeLab VPN to access this data.

You can check your access directly via browser at https://if.proteinatlas.org . You will find there a bunch of folders with images and the IF-image.csv with all the meta-data regarding them.

Now, normally you will not need to access the HPA servers on your own: we have already downloaded most of the dataset in our lab storage and you can work with it faster and more efficiently there. You can request access to the lab storage by following the instructions in the file storage_information.pdf: the project/dataset name is HPA

IF-image.csv file

This file contains all relevant information about the dataset images in CSV format. You will always find the IF-image.csv file in the base folder of the dataset. The most regularly used columns are:

  • filename: the location of the file in the dataset (replace the /archive/ part by /images/). This is the prefix for the blue (nuclei), red (microtubules), yellow (ER) and green (protein) channels of each image (Note: check Channel_scheme below)
  • status: the status of the image according to the annotation process in LIMS. Check the file Overview_data_and_indentifiers_HPA.pdf in the util folder for a lot of interesting information about how this works.
  • locations: in which subcellular compartments the protein is localized. This has been annotated by humans. Check the file SOP _annotation _2019.pdf in the util folder to understand the different categories.
  • unspecific: there are several reasons why an image might be flagged as unspecific. Check the files Overview_data_and_indentifiers_HPA.pdf and SOP _annotation _2019.pdf in the util folder to learn more.
  • antibody, ensembl_ids: related antibody and Ensembl gene ids for the image. Check the file Overview_data_and_indentifiers_HPA.pdf in the util folder for a more detailed explanation.
  • atlas_name: cell line. Check the file SOP _annotation _2019.pdf for the list of cell lines.
  • versions, earliest_version, latest_version: the HPA is a growing initiative that constantly adds new images to the dataset. They add a controlled batch of new images periodically, increasing the dataset version. At the time I'm writing this information, the current dataset version is v25. You can see when an image was added or re-annotated by playing with these 3 columns.
  • x_pos, y_pos, z_pos: coordinates of the image in the well plate.
  • Cell count: number of cells segmented by the HPA cell segmentation model ( https://github.com/CellProfiling/HPA-Cell-Segmentation ).
  • Channel scheme: channel scheme of the images. Not all images follow the same scheme as the classical subcellular images (blue -nuclei, red - microtubules, yellow -ER and green - protein); "Cilia" scheme, for example, is different.

This file is big (>600MB), so if you want to work with a subset of its images you can use a python library like pandas to filter what you want. The filtering can be complex: feel free to ask advice from your colleagues or the HPA personnel (Kalle von Feilitzen)

IF-image_v25.csv file

Next to IF-image.csv you will find IF-image_v25.csv, a much smaller file (about 50MB) with the same columns. This one is already filtered down to the current v25 release: only the images that are publicly released (state 35) and still present in v25, one row per FOV. It is the file you normally want to work with: IF-image.csv contains the whole HPA history (over 1.6 million rows, including images that were never released or have been removed), while IF-image_v25.csv matches what you actually find in the images folder.

When a new HPA version comes out, this file is regenerated for that version.

Public image files

Under the folder images you will find downloaded all the HPA v25 publicly released images (state 35, check Overview_data_and_indentifiers_HPA.pdf in the util folder). We always keep only the latest version of the dataset here: when a new HPA version is released, this folder is replaced by that version's image set.

The (original) images in the HPA servers are stored in TIFF format, gzip compressed. We have converted them to PNG, keeping the original pixel values untouched (the conversion is a straight dump of the TIFF, no rescaling or resizing). Note that PNG is itself a compressed format.

Most of the images are 16 bit, but 188 plates are 8 bit: the 184 oldest plates (the 1728 x 1728 ones) plus 4 more at 2048 x 2048. Don't assume a bit depth, read it from the file.

Resolutions vary more than you might expect, so account for all of them:

resolution bit depth plates
2048 x 2048 16 bit 1381
1728 x 1728 8 bit 184
3072 x 3072 16 bit 98
2160 x 2160 16 bit 33
2048 x 2048 8 bit 4

The folder currently holds 95,545 FOVs over 1,700 plates, which is the whole v25 release. By channel scheme: 86,825 Normal, 5,924 Cilia, 1,562 Sperm and 1,234 Stem cells. (One single FOV, 2252_B6_9, is incomplete: the HPA does not serve its yellow channel.)

For each image prefix you will find a blue (nuclei), red (microtubules), yellow (ER) and green (protein) image channel, for the "Normal" scheme.

The "Cilia" scheme adds a fifth white channel on top of those four, so those FOVs have 5 files instead of 4 (5907 of the 5924 Cilia FOVs currently on disk). Only the Cilia scheme uses it. If you glob for images, don't assume exactly 4 channels per prefix.

Finally, each HPA new released version adds new images, but also deletes some images from past versions for many reasons. For consistency sake, we have taken the decision to keep all images from version 23 on, even if they have been deleted in a more modern release. Those deleted images are not in the images folder (which only holds the current version): you will find them in the removedFromV23 folder, which mirrors the same images / segmentations / crops structure. It currently holds 2642 FOVs: 2336 that were dropped at v23 and 306 dropped at v24.1.

Additional image works

There are other folders where you can find several items produced from the original images:

  • segmentations: this folder contains all nuclei and cell segmentations masks created using the HPACellSegmentator software.

  • crops: this folder contains each single cell crop from the above mentioned cell segmentations, under crops/cell_crops/<plate>/. You get 10 files per cell, all of them 1024 x 1024:

    • <FOV>_cell<N>_cellmask.png: the cell mask crop (just the mask)
    • <FOV>_cell<N>_nucleimask.png: the nuclei mask crop (just the mask)
    • <FOV>_cell<N>_crop_[blue|red|yellow|green].png: 4 files, the original image cropped with the cell at its center
    • <FOV>_cell<N>_crop_masked_[blue|red|yellow|green].png: 4 files, the same crop AND masked by the cell shape
    • A general crop_info.csv in the crops base folder where you can find the cell position in the crop, for all cells. The y1,x1,y2,x2 columns are the cell bounding box inside the 1024 x 1024 crop (the cell is centered, so the box is centered on 512,512), not its position in the original FOV.

    Be aware that the number of cells per FOV is not uniform across the dataset. Most of these crops were generated discarding some cells (mainly those whose nucleus lies on the border of the FOV, plus some size criteria), while FOVs regenerated later keep every cell the segmentation found. So do not assume that crops contains the same cells as the segmentations masks: always take the cell list from crop_info.csv or from the files themselves, and expect some FOVs to have more cells than their neighbours.

  • v25: the images/segmentations/crops newly added/modified in the v25 version. All its contents have been merged with the aforementioned general folders, it's just a way to have direct access to the latest version of the HPA in case you want to compare with older versions you might have downloaded already.

  • micronuclei: this folder contains micronuclei detection, segmentation and quantification for all the v24 HPA images. In detail:
    • The segmentations folder contains all micronuclei segmentations for each FOV of the HPA v24 dataset. If the FOV name is not there, there were no micronuclei segmented
    • A general micronuclei_info.csv in the micronuclei base folder where you can find the data of each micronuclei. This data contains positional, morphological and intensity information of the micronuclei and related nuclei.
  • removedFromV23: images (plus their segmentations and crops) that were publicly released in v23 or later but have since been deleted by the HPA. Same <plate>/<FOV>_<channel>.png layout as images. Keep it in mind when you assemble a dataset: images + removedFromV23 is everything we hold from v23 on.
  • subcell-exps: the experiment archive of the SubCell work (Lundberg lab), including pre-computed single cell embeddings of the HPA images for several models. This is the place to go if you need protein/cell embeddings instead of computing them yourself. It is big (~4.7 TB) and it has its own documentation: read subcell-exps/SubcellExpsInfo.md before using it, as the covered image set is the v23 HPA and the cell numbering does not match our crops.
  • sequences: one protein sequence per antibody, as a FASTA file <antibody>.faa, for the 15,727 antibodies of the v25 Normal scheme crops (other schemes and removedFromV23 are not covered yet). Each is the longest of the transcripts that the antibody's antigen matches on the HPA antibody page, taken from Ensembl release 109 (the one HPA v25 links to); antibodies without an antigen match, like the CAB ones, get the longest isoform of the gene. Antibodies targeting several genes have one record per gene. sequences_provenance.csv records every choice and its candidates, and FASTA_sequence_generation.md explains how to reproduce it.
  • other: a catchall folder with information that was deemed useful by some people before organizing the things properly. Go there an explore at your own risk!

Utility scripts

  • util/download_simple.py: this utility script allows you to download the IF-image.csv, do some filtering over it and download all resulting images, all directly from the HPA servers. It's mainly used to update the dataset to the latest version when it's published.

WebDAV (lab storage) scripts

Although they work on their own, most of these scripts are guidelines in case you want to integrate direct connection to the lab storage into your own scripts.

  • util/webdav_download_file_list.py: this script allows you to download all paths included in a file list from the lab storage. Take it as an example, you can modify it as you see fit to your own needs.

  • util/webdav_download_folder.py: this script allows you to download a folder (and all it's contents) from the lab storage. Take it as an example, you can modify it as you see fit to your own needs.

  • util/webdav_stream_file_list.py: this script serves as an example of how you could do "streaming" file by file from the lab storage. You can of course run multiple streaming list in parallel, keep a certain number of files before removing them, etc, etc...

  • util/webdav_upload_folder.py: this script allows you to upload the contents of a folder (does not create the base folder itself) to a pre-existing folder in the lab storage. Take it as an example, you can modify it as you see fit to your own needs.

Additional resources

You might want to check this other resources related to several produced items in the HPA dataset: