Cloud Object Storage - Community Hub Guide

This section gives an overview of storing data in the cloud, as well as links to how-to guides for using specific tools to manage your cloud data:

Overview¶

Your hub lives in the cloud, therefore the preferred way to store data is using object storage, such as Amazon S3 or Google Cloud Storage. Cloud object storage is essentially a key/value storage system. The keys are strings and the values are bytes of data. Data is read and written using HTTP calls.

The performance of object storage is very different from file storage. On one hand each individual read / write to object storage has a high overhead (10-100 milliseconds) since it has to go over the network, while on the other hand object storage “scales out” nearly infinitely, meaning that we can make hundreds, thousands, or millions of concurrent read / write requests. This makes object storage well suited for distributed data analytics. However, data analysis software must be adapted to take advantage of these properties.

Scratch versus persistent buckets on a 2i2c hub¶

Bucket: A bucket is a container for objects.
Object: An object is a file and any metadata that describes that file.

Scratch buckets¶

Scratch buckets are designed for storage of temporary files, e.g. intermediate results.

Check the name of your scratch bucket by opening a Terminal in your hub and running the command

$ echo $SCRATCH_BUCKET
s3://2i2c-aws-us-scratch-showcase/<username>

Persistent buckets¶

Persistent buckets are designed for storing data that is consistently used throughout the lifetime of a project and the data is not purged after a set number of days.

Check the name of your persistent bucket by opening a Terminal in your hub and running the command

$ echo $PERSISTENT_BUCKET
s3://2i2c-aws-us-persistent-showcase/<username>

Storage costs¶

See 2i2c Infrastructure Guide – What exactly do cloud providers charge us for? for a detailed overview of cloud object storage costs.

Access permissions¶

A common set of credentials is used for accessing storage buckets.

It is possible to configure read-only access for objects stored in cloud storage on your hub, though this is not a standard feature of our hubs. Please consult 2i2c support to discuss enabling this feature.