Skip to content

Data storage

As a member of the Euclid group at CC-IN2P3, you have access to several storage spaces. More details on the different storage areas are available here.

THRONG

The Euclid THRONG directory is a 100 GB storage space available in /pbs/throng/euclid, or through the environment variable $THRONG_DIR. It it backed-up and therefore suitable for small set of data that are precious, like scripts or softwares. More details concerning this space can be found here. If you need a personal directory accessible for other Euclid members, you can create one in $THRONG_DIR/Users/. Projects directories can be created directly in $THRONG_DIR (example: $THRONG_DIR/OU-VIS). By default, Euclid members have a full read access to all directories. Common tools can also be installed in $THRONG_DIR/tools/.

SPS

The Euclid SPS (Semi-Permanent Storage) is a multi-petabytes shared file-system available in /sps/euclid. More details concerning SPS can be found here. There is no backup of this storage space, and files that are not intended to be used in the following year should be deleted or moved somewhere else. In general large data set, or software that does not need to be backup can be stored in SPS. You can create your own personal data storage directory inside /sps/euclid/Users (use your login for the directory name). Please contact us if you need a new project directory (example: /sps/euclid/OU-VIS/). By default all your files stored in SPS are readable by the other Euclid members.

iRODS

iRODS is a middleware providing a uniform access to different type of storage technologies (disk, tapes...). The general CC-IN2P3 iRODS documentation is here. The Euclid group has a dedicated iRODS area that can be accessed from CC-IN2P3 nodes, but also from outside, either using the iRODS CLI, or through a web interface, or using the WebDAV protocol. It is used to store several types of data:

  • Test data for Processing Function, accessed by CODEEN
  • Data that needs to be shared outside of CC-IN2P3
  • Data that needs to be archived, either on disk or on tapes

More details can be found on the Euclid Redmine dedicated documentation. Generic accounts with corresponding directories exist or can be created for every PF (see the dedicated documentation). If needed personal accounts can also be created. Set the following configuration in your configuration file $HOME/.irods/.irodsEnv to access the Euclid iRODS space using the iRODS client:

irodsHost 'ccirods.in2p3.fr'
irodsPort 5515
irodsUserName 'your_login'
irodsZone 'euclid-fr'

Note that the version 3.3.1 of the CLI must be used.

HPSS

HPSS is the magnetic tape storage system at CC-IN2P3. The Euclid group has a dedicated space /hpss/in2p3.fr/group/euclid that can be used to store large volume of data that do not need to be used in the next months. Be careful, storing small files should be avoided since reading them can be very inefficient. It is recommended to store files larger than 1 GB, by creating tarballs if necessary. More details about HPSS can be found here. Note also that data stored on iRODS can be transparently migrated to HPSS. In case you need to move some of your data to HPSS, please contact us: we can enable automatic migration for a specific iRODS directory.