Skip to content

Storage design study

This is a brief study undergone at 2023_07_14 to select a proper storage option for the lab. These are the gathered requirements:

Hard requirements

  • H1) Needs to be accessible through Linux, Windows and MacOS
  • H2) Needs to be accessible by any lab member regardless they are in Stanford or SciLifeLab (required steps may vary but they need to be feasible by all lab members)
  • H3) Needs to grant an "automated" access possibility (mapped network drive, ssh access, API, etc...) so we can setup scripts/programs to directly store instrument/services outputs
  • H4) Scalable to at least a maximum limit of 1000TB. We are most probably not going to need it, but it's a good upper limit
  • H5) Solid backups

Soft requirements

  • S1) User space granularity: different limit per user and/or a way to control user usage to limit misuse/abuse
  • S2) Project space granularity: option to create accounts/limits per project, separated from the lab members personal files.
  • S3) Good transfer speed
  • S4) Possibility to share files with other lab members and guests (non-authenticated)
  • S5) Allow simple access from any computer (SSH, web application, etc...)
  • S6) Low cost

The current WebDAV over HTTPS solution was chosen due to the high-level of compliance with the requirements:

Current setup at Stanford

  • H1) Mostly compliant (might need third party software like winscp or Cyberduck)
  • H2) Compliant
  • H3) Compliant 100% in Linux, most of the time 100% in Windows too
  • H4) Compliant with some minor logistics/infrastructure issues (that's a lot of hard drives)
  • H5) Compliant
  • S1) Mostly compliant (script monitoring)
  • S2) Compliant
  • S3) Good speed in Stanford, regular internet speed from SciLifeLab
  • S4) Compliant
  • S5) Mostly compliant (no direct SSH, but can be mounted as a network drive)
  • S6) Lowest possible price option

Storage solution

Hardware

The current solution Hardware consists in:

  • A regular workstation:

    • i7 12700 processor
    • 128GB RAM
    • NVIDIA RTX2000 ADA GPU
    • 1TB NVMe OS partition
    • 20TB local SATA HDD for working space
    • Ubuntu server 24.04 as OS

  • 2 Sabrent USB 10x HDD enclosures with 20x 20TB HDD disks

    • /data partition with a 200TB virtual group (10 disks)
    • /deep partition with a 200TB virtual group (10 disks)

  • A UPS battery for surge and power off protection

The current enclosures are full, but there's enough high-speed USB connector left to add 2 more enclosures, reaching 800TB in the future (if required)

Software

Software wise, the solution is a regular Nginx installation with WebDAV support. Basically:

  • Every lab member has a space under /data/dav/{username} with www-data:www-data permissions (protected by http basic authentication)
  • Every lab member has a /data/dav/{username}/www directory that is open to everyone (provided exact URL)
  • Every lab member has a /data/dav/{username}/web symlink to their /data/dav/{username}/www directory (protected by http basic authentication)

Refer to /etc/nginx/sites-enabled/services file to check the webserver specific configuration

Each lab member space is their own personal storage, which is usually soft capped to 1TB (see Backup & storage checks section below). They can also work in specific projects (shared with other lab members or not) that might require more storage. The projects are stored under /data/dav/projects/ directory, each one in their own subdirectory, and the lab members involved in it get a symlink to it in their own personal storage.

Finally, there's several symlinks redirected to /deep/external/ that are used for different purposes:

  • To share complete and constantly changing directories with external collaborators (like the Beacon project)
  • To be used as upload directories for third parties (like the COMAS dataset ingestion, etc...).
  • In case of the /data/dav/public directory, that is a symlink to the space in deep partition that stores all link for published papers links Important! this data, as being physically allocated in the deep partition, is not part of the /data/dav weekly backed-up data.

Then there's the /deep partition, used for the following datas:

  • Lab readily available datasets, located at /deep/datasets/. Lab members can have access seamlessly to the datasets by creating a symlink in their /data/dav/{username}/ account. Datasets should have root ownership and be read-only most of the time, inhibiting people to modify/delete them by mistake.
  • A separate /deep/external/ tree to handle uploads/transfers from external collaborations. This separation allows to do more fine-grain access control depending on needs.
    • There's a specific /deep/external/public directory that should be the place to host data specifically package for papers and other published resources.
  • A temporary /deep/archived storage zone to keep the data from former lab members until needed.

Backup & storage checks

A few simple processes keep the storage safe and fairly shared:

  • Weekly backup: the personal and project spaces of /data are backed up every week to ell-backup, which keeps the weekly copies for a year. Its goal is to recover lab members' own work after a mistake or a disk failure (H5).
  • Yearly cold storage: once a year, a script stores the personal and project spaces, the lab datasets and the archived data in Stanford's Elm cold storage (S3-compatible), including the /deep partition that the weekly backup does not cover. As there is no room to compress a whole disk, it works one conceptual chunk at a time (a dataset, a user space, a project...), compressing and splitting it into parts on the fly. A chunk's previous cold copy is only replaced once the new one is complete, so there is always one full copy in Elm.
  • No RAID: the disks are not in a RAID array. A 400TB RAID would double the hardware cost (S6), so the backups are the protection against disk failures.
  • Storage usage checks: instead of hard quotas, a weekly cron job checks each lab member's usage against their limit (1TB by default) and posts a warning to Slack when someone goes over it. Nobody gets blocked, since transfers sometimes need extra room for a while, but overuse is noticed and can be sorted out (S1).