Data and Filesystem

This section describes how files and data storage are handled in 2i2c Hubs.

Quick recommendations¶

Some quick recommendations for how to handle data:

• Use your home directory to store code, notebooks, and small data files (<1 GB) for personal use

• Use cloud object storage to store larger datasets and to share data across your team

• Consider whether your project would benefit from other cloud-native data storage solutions such as a database, data warehouse, or data lake

For more information, see the sections below.

• Filesystem and user directory
• Move code in and out of the hub with GitHub
• Share data files with your users
• Cloud Object Storage
  • How-to work with object storage in Python
  • How-to manage S3 cloud object storage with AWS CLI
  • How-to manage GCP cloud object storage with Google Cloud SDK

References and attribution¶

Much of this material was adapted from the Pangeo documentation.