Rclone

Intro

Rclone is a command-line tool for managing files on cloud storage, offering an alternative to web storage interfaces. It supports over 40 cloud storage products, including S3 object stores and various file storage services, and provides cloud equivalents to popular Unix commands. It offers shell pipeline support, --dry-run protection, and can be used at the command line, in scripts, or via its API.

Read along to learn how to configure Rclone to perform a backup on your Cubbit Account.

tip

This guide refers to Rclone v1.64.2.

Please note that the Rclone options listed below may vary depending on the version you use.

Prerequisites

First, you need to get your access key and secret key, from the Cubbit Web Console or https://console.[your-tenant].cubbit.eu. Please follow these instructions on how to get started with a Cubbit account and generate these keys.

Installation

There are multiple ways of installing Rclone on your machine; please follow the instructions on the official website.

Configuration

caution

The following instruction should work on any Unix machine capable of running Rclone.

For any doubts, you can check the official configuration guide.

Once installed, you should be able to run the command to add a configuration.

rclone config

It will start an interactive session to guide you through the configuration process of a remote.

Press n to create a new remote:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> cubbit

Then choose which storage the remote will connect to.

Since Cubbit uses the S3 protocol, enter 5 to select the S3 generic storage option. Then, select the line "Any other S3 compatible provider"

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 1 / 1Fichier
   \ (fichier)
 2 / Akamai NetStorage
   \ (netstorage)
storage> 5

Next, configure the authentication method. We will enter the credentials, so press 1.

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 1

Insert your access key here.

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> ******************

and, the secret key.

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> ******************

As a region, set eu-west-1, and press enter.

Option region.
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Use this if unsure.
 1 | Will use v4 signatures and an empty region.
   \ ()
   / Use this only if v4 signatures don't work.
 2 | E.g. pre Jewel/v10 CEPH.
   \ (other-v2-signature)
region> eu-west-1

Configure the Cubbit endpoint.

danger

Remember to replace the endpoint https://s3.cubbit.eu with https://s3.[your-tenant].cubbit.eu if you have a custom tenant.

Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a value. Press Enter to leave empty.
endpoint> https://s3.cubbit.eu

Do not set the location_constraint, press enter.

Option location_constraint.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a value. Press Enter to leave empty.
location_constraint>

Then you can configure the default ACLs. We pick 1.

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
acl> 1

Unless you want to see the advanced configuration, press n.

Edit advanced config?
y) Yes
n) No (default)
y/n> n

...then y, and we are done!

danger

Remember to replace the endpoint https://s3.cubbit.eu with https://s3.[your-tenant].cubbit.eu if you have a custom tenant.

Configuration complete.
Options:
- type: s3
- provider: Other
- access_key_id: <access_key_id>
- secret_access_key: <secret_access_key>
- region: eu-west-1
- endpoint: https://s3.cubbit.eu
- acl: private
Keep this "cubbit" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

How to

The following operation assumes you have already created a bucket called bucket-name on your account.

List files

rclone ls cubbit:bucket-name

Sync folder

rclone sync -P ./backup-folder cubbit:bucket-name

Delete a bucket either with enabled or disabled S3 Versioning and its contents

rclone purge -P -vvv cubbit:bucket-name

For more information on Rclone, please refer to the official documentation.