Setup AWS CLI

AWS CLI is a command line tool developed by Amazon. Many developers and IT system administrators make use of this, making it one of the most widely used CLI tools. You can use it with any S3-compatible API object storage service, including Cubbit, to manage your storage objects and buckets.

Prerequisites

First, you need to get your access key and secret key, from the Cubbit web app. Please follow these instructions on how to get started with a Cubbit account and generate these keys.

Installation

Then, you have to install the AWS CLI by looking at the AWS instructions.

Configuration

To configure the AWS CLI, type:

aws configure

Using a different profile

If you want to set a different profile (e.g. cubbit), simply type:

aws configure --profile cubbit

This command will generate a series of prompts, which should be filled out with the previously generated keys:

AWS Access Key ID [None]:       <your Cubbit access key>
AWS Secret Access Key [None]:   <your Cubbit secret key>
Default region name [None]:     eu-west-1
Default output format [None]:   ENTER

You can also be more specific, like setting the maximum number of concurrent requests:

aws configure set default.s3.max_concurrent_requests 20

Using a different profile

Once again, if you want to edit a different profile (e.g. cubbit):

aws configure set cubbit.s3.max_concurrent_requests 20

Usage

Now you have to include --endpoint https://s3.cubbit.eu in each of the usual AWS CLI commands so that it points to Cubbit's Object Storage. But apart from that, the command syntax stays the same.

Bucket commands

Create a bucket

aws s3 --endpoint https://s3.cubbit.eu mb s3://my-cubbit-bucket

List all buckets

aws s3 --endpoint https://s3.cubbit.eu ls

List the content of a bucket

aws s3 --endpoint https://s3.cubbit.eu ls s3://my-cubbit-bucket

Delete a bucket

aws s3 --endpoint https://s3.cubbit.eu rb s3://my-cubbit-bucket

Object commands

Upload a single file

aws s3 --endpoint https://s3.cubbit.eu cp sergio.jpg s3://my-cubbit-bucket

Upload a folder

aws s3 --endpoint https://s3.cubbit.eu sync test_folder s3://my-cubbit-bucket

Multipart Upload

S3-compatible object storage services support uploading large files in separate chunks of data and uploading them in parallel when the file size is above a certain threshold.

By default, the multipart threshold for AWS CLI is 8MB. This means that any file larger than 8MB will be automatically split into chunks and uploaded together in parallel. To use this feature, simply upload a file that is larger than 8MB in size and AWS CLI takes care of the rest automatically.

For more info, you can refer to the detailed section about Multipart Upload.

Download a single file

aws s3 --endpoint https://s3.cubbit.eu cp s3://my-cubbit-bucket/sergio.jpg ~/Downloads/sergio.jpg

Download a folder

aws cp --recursive s3://my-cubbit-bucket/test_folder ~/Downloads/new_folder

Delete a single file

aws s3 --endpoint https://s3.cubbit.eu rm s3://my-cubbit-bucket/sergio.jpg

Delete all files in a bucket

aws s3 --endpoint https://s3.cubbit.eu rm --recursive s3://my-cubbit-bucket/

Using a different profile

If you want to use a different profile (e.g. cubbit), add --profile to the commands, for instance:

aws --endpoint https://s3.cubbit.eu --profile cubbit s3 ls

Testing single S3 APIs

Until now we've only mentioned aws s3 CLI, but it's worth mentioning that Amazon provided also a slightly different CLI called aws s3api.

s3api is a very low-level utility and provides access to single AWS S3 APIs, while the previous commands were more focused on the overall operation.

For instance, let's pick the upload operation. The command:

aws s3 --endpoint https://s3.cubbit.eu cp sergio.jpg s3://my-cubbit-bucket

behind the hood calls a few S3 APIs, like the HeadObject and the PutObject. On the other hand, calling:

aws s3api --endpoint https://s3.cubbit.eu put-object --bucket my-cubbit-bucket --key sergio.jpg --body ~/sergio.jpg

performs a very specific PutObject and nothing more.