Vai al contenuto principale

Smart Data Placement

Overview​

Smart Data Placement lets you define rules that automatically route objects uploaded by your customers to different redundancy classes in your swarm. As a DS3 Gateway owner, you control where each object is stored based on the storage class selected by the customer and the object's properties.

In Composer, Smart Data Placement is configured from the gateway settings under Routing rules, also called Gateway rules. The page edits a single routing rule as a JSON document in the Live JSON editor.

Customers select a storage class (Standard, Hot, Cold, or a custom class defined for the gateway) when uploading files. The gateway evaluates its routing rules and assigns the final redundancy class where the data is stored.

Why Smart Data Placement?​

This feature is intended for resellers and gateway owners who want to:

  • Optimize costs: Route large or infrequently accessed files to more economical redundancy classes
  • Ensure compliance: Automatically enforce data locality and retention policies
  • Improve performance: Direct time-sensitive data to high-performance redundancy classes
  • Offer differentiated services: Create service tiers with different storage characteristics
  • Automate management: Remove manual intervention from data routing
info

Smart Data Placement is available for private gateways that you own and manage. As a gateway owner, you configure the routing rules, while your customers select storage classes when uploading data.

How It Works​

Smart Data Placement operates at two levels.

1. Client-Level Selection​

Customers uploading files through the S3 API specify a storage class with the standard S3 header x-amz-storage-class. The gateway accepts the built-in Standard, Hot, and Cold classes, plus any custom class defined in Settings → Storage class. Names are matched case-insensitively, so STANDARD, HOT, and COLD are equivalent header values. If the header is absent, the upload uses STANDARD.

You define the pricing and Service Level Agreements (SLAs) for each storage class tier.

2. Gateway-Level Routing​

The gateway evaluates its routing rules against the upload's storage class and object metadata (key, size, content type, and tenant name) and selects the final redundancy class.

Customer Upload → Storage Class Selection → Rule Evaluation → Redundancy Class Assignment → Data Storage

Complete Flow​

┌─────────────┐
│ Customer │ Uploads file with storage class = "STANDARD"
└──────┬──────┘
│
▼
┌──────────────────┐
│ DS3 Gateway │ Evaluates configured routing rules
│ (Smart Data │ - File name: *.mp4
│ Placement) │ - Size: >100MB
└──────┬───────────┘
│
▼
┌──────────────────┐
│ Matched Rule │ "video-to-cold-storage"
│ (First Match) │
└──────┬───────────┘
│
▼
┌──────────────────┐
│ Redundancy Class │ rc-cold-storage
└──────────────────┘

Configure Routing Rules in Composer​

The Routing Rules page edits the routing rule as a JSON document. It does not use a form: the field names described on this page are the JSON properties of the document.

The page describes the behavior as: "Routing rules decide where uploaded data is stored. Each rule matches the upload's storage class and file properties. The first match wins, and the fallback redundancy class handles everything else."

  1. Open Composer and select the DS3 Gateway.
  2. Go to Settings → Routing rules. The section is also called Gateway rules.
  3. Edit the rule in the Live JSON editor:
    • Edit the document directly. The editor validates JSON syntax while you type.
    • Click Import to load a .json file up to 10 MB. The dialog previews the file; use the dialog's Upload action to replace the editor content. Imported JSON is not applied until you click Apply changes.
    • Click Export to download the current editor content as a .json file.
  4. Review the document and click Apply changes.
  5. A successful apply replaces the rule and shows a confirmation notification.

The page edits a single routing rule and has no list view. If the gateway returns more than one rule, the page edits the first rule in evaluation order.

The Live JSON header shows Unsaved changes while the editor differs from the applied rule. The page checks JSON syntax and that the document is a JSON object; the gateway validates the routing policy semantics and displays any server-side validation errors below the editor.

Empty State​

If the gateway has no routing rule, the page shows No routing rule found. A rule must be configured for that gateway before it can be edited from this page; check the gateway connections or contact support.

Storage Classes​

Storage classes are managed in Settings → Storage class. They define the values tenants use when creating routing rules and are the values accepted in inputStorageClass. The built-in Standard, Hot, and Cold classes are provisioned with the gateway, and custom classes can be created for it.

An upload that specifies a storage class not defined for the gateway is rejected with InvalidStorageClass. A defined storage class with no matching rule is accepted and uses the fallback redundancy class.

Routing Rule Document​

The editor holds one routing rule as a JSON object, not an array.

{
"id": "rule-1",
"name": "Cold data routing",
"description": "Route COLD storage class objects to cold storage",
"enabled": true,
"inputStorageClass": "COLD",
"conditions": null,
"action": {
"type": "route",
"target": {
"type": "fixed",
"redundancyClassId": "rc-cold-storage"
}
}
}
FieldDescription
idIdentifier of the rule. The page applies changes to the rule it opened, so editing this value does not create or rename a rule.
nameName of the rule. Required; maximum 100 characters.
descriptionOptional description of the rule.
enabledWhether the rule is evaluated. Set it to true for the rule to take effect.
inputStorageClassStorage class that selects the uploads this rule applies to. It must be defined for the gateway in Settings → Storage class.
conditionsCondition tree that an upload must match. null or omitted matches every upload that uses inputStorageClass.
actionAction to apply when the rule matches: route or drop.
nota

The JSON document must be an object. Arrays, scalar values, and malformed JSON are rejected before the request is sent to the gateway.

Rule Conditions​

A rule can match uploads on the following fields.

Available Fields​

FieldDescriptionExample
object.keyObject key, including the file name and pathvideos/movie.mp4
object.sizeObject size in bytes104857600 (100 MB)
object.content_typeObject's Multipurpose Internet Mail Extensions (MIME) typevideo/mp4
tenant.nameTenant namecubbit
nota

Operator availability depends on the field. String fields (object.key, object.content_type, tenant.name) accept the string operators plus equals, not_equals, in, not_in, exists, and not_exists. object.size accepts gt, gte, lt, lte plus the same common operators.

Supported Operators​

OperatorDescriptionValue
equalsExact matchString or number
not_equalsNot equalString or number
gtGreater thanNumber
gteGreater than or equalNumber
ltLess thanNumber
lteLess than or equalNumber
starts_withPrefix matchString
ends_withSuffix matchString
containsContains a substringString
not_containsDoes not contain a substringString
matches_regexMatches a regular expressionString (valid regular expression)
not_matches_regexDoes not match a regular expressionString (valid regular expression)
existsField existsNo value
not_existsField does not existNo value
inValue is in a listArray
not_inValue is not in a listArray

Each condition can set caseSensitive (default false) to control case sensitivity for string comparisons, including regular expressions and in lists.

Logical Groups​

Conditions are combined with logical groups:

  • and: all nested conditions must match.
  • or: at least one nested condition must match.

A logical group uses operator with and or or, omits field, and lists the nested conditions in conditions. Groups can be nested.

{
"operator": "and",
"conditions": [
{
"field": "object.key",
"operator": "matches_regex",
"value": "\\.(mp4|avi|mov)$",
"caseSensitive": false
},
{
"operator": "or",
"conditions": [
{
"field": "object.size",
"operator": "gt",
"value": 104857600
},
{
"field": "object.content_type",
"operator": "equals",
"value": "video/mp4"
}
]
}
]
}

This object is the value of the conditions field of a routing rule.

Action Types​

Route to a Fixed Redundancy Class​

A route action stores matching uploads in one redundancy class. The target uses type: "fixed" and the UUID of a redundancy class connected to the gateway.

{
"type": "route",
"target": {
"type": "fixed",
"redundancyClassId": "rc-cold-storage"
}
}

Drop Upload​

A drop action rejects matching uploads. The S3 request fails and no object is stored. A drop action cannot define a target.

{
"type": "drop"
}
attenzione

The redundancyClassId values in this page are placeholders. Replace them with the identifiers of redundancy classes connected to the gateway. Apply changes fails if the rule targets a redundancy class that is not associated with the gateway.

Complete Examples​

Each example is a complete single-rule document for the Live JSON editor. The storage class in inputStorageClass must exist for the gateway; the redundancy class identifiers are placeholders.

Example 1: Route Large Video Files to Cold Storage​

Route video files larger than 100 MB uploaded with the STANDARD storage class:

{
"id": "video-to-cold-storage",
"name": "Large Video Files to Cold Storage",
"description": "Route video files larger than 100MB to cold storage",
"enabled": true,
"inputStorageClass": "STANDARD",
"conditions": {
"operator": "and",
"conditions": [
{
"field": "object.key",
"operator": "matches_regex",
"value": "\\.(mp4|avi|mov|mkv|webm|flv)$",
"caseSensitive": false
},
{
"field": "object.size",
"operator": "gt",
"value": 104857600
}
]
},
"action": {
"type": "route",
"target": {
"type": "fixed",
"redundancyClassId": "rc-cold-storage"
}
}
}

Example 2: Block Executable Files​

Reject potentially dangerous executable files uploaded with the STANDARD storage class:

{
"id": "block-executables",
"name": "Block Executable Files",
"description": "Drop potentially dangerous executable files",
"enabled": true,
"inputStorageClass": "STANDARD",
"conditions": {
"operator": "or",
"conditions": [
{
"field": "object.key",
"operator": "matches_regex",
"value": "\\.(exe|bat|sh|com|scr|dll)$",
"caseSensitive": false
},
{
"field": "object.content_type",
"operator": "starts_with",
"value": "application/x-executable"
}
]
},
"action": {
"type": "drop"
}
}

Example 3: Route Backup Files to Archive​

Route backup files uploaded with the COLD storage class to long-term archive storage:

{
"id": "backup-to-archive",
"name": "Backup Files to Archive",
"description": "Route backup folders to archive storage",
"enabled": true,
"inputStorageClass": "COLD",
"conditions": {
"operator": "or",
"conditions": [
{
"field": "object.key",
"operator": "contains",
"value": "/backup/"
},
{
"field": "object.key",
"operator": "starts_with",
"value": "backups/"
}
]
},
"action": {
"type": "route",
"target": {
"type": "fixed",
"redundancyClassId": "rc-archive"
}
}
}

Example 4: Optimize Small Files​

Route small files uploaded with the HOT storage class to a fast-access redundancy class:

{
"id": "small-files-optimization",
"name": "Optimize Small Files",
"description": "Route files smaller than 10MB to fast access redundancy class",
"enabled": true,
"inputStorageClass": "HOT",
"conditions": {
"operator": "and",
"conditions": [
{
"field": "object.size",
"operator": "lt",
"value": 10485760
},
{
"field": "object.key",
"operator": "not_matches_regex",
"value": "\\.(tmp|log|cache)$"
}
]
},
"action": {
"type": "route",
"target": {
"type": "fixed",
"redundancyClassId": "rc-fast-access"
}
}
}

Rule Evaluation​

When the gateway processes an upload:

  • Only rules with enabled: true are evaluated.
  • The gateway groups rules by inputStorageClass and considers only rules whose storage class matches the upload. Storage class matching is case-insensitive. An upload without x-amz-storage-class uses STANDARD.
  • Within a storage class, rules are evaluated in order. The first rule whose conditions match determines the action.
  • A rule with conditions set to null or omitted matches every upload that uses its storage class.
  • A route action stores the object in the target redundancy class.
  • A drop action rejects the upload immediately; no object is stored.
  • If no rule matches, the gateway uses the fallback redundancy class from the gateway's default connection.

Testing Your Configuration​

Test routing with normal S3 uploads after applying the rule.

The following examples use the AWS CLI; other S3 clients set the same x-amz-storage-class header.

# Matching Example 1: a video file larger than 100MB with storage class STANDARD
aws s3 cp large-video.mp4 s3://your-bucket/videos/ \
--storage-class STANDARD \
--endpoint-url https://your-gateway.example.com

# Matching Example 4: a small file with storage class HOT
aws s3 cp document.pdf s3://your-bucket/docs/ \
--storage-class HOT \
--endpoint-url https://your-gateway.example.com

# Matching Example 2: an executable with storage class STANDARD
aws s3 cp program.exe s3://your-bucket/apps/ \
--storage-class STANDARD \
--endpoint-url https://your-gateway.example.com
Test uploadStorage classExpected result
large-video.mp4 (larger than 100 MB)STANDARDRouted to rc-cold-storage when Example 1 is applied
document.pdf (small)HOTRouted to rc-fast-access when Example 4 is applied
program.exeSTANDARDRejected when Example 2 is applied

Uploads that match no rule are accepted and use the fallback redundancy class. Test both matching and non-matching objects before relying on a rule in production.

Best Practices​

1. Start with One Condition​

Begin with a single condition, apply it, and verify the result with real uploads. Add logical groups and additional conditions only after the basic behavior is correct.

2. Use Descriptive Names and Descriptions​

The name and description fields identify the rule in the editor and in exported files. Describe the business purpose of the rule, not only its technical conditions.

3. Scope Rules by Storage Class​

inputStorageClass narrows a rule before any object condition is evaluated. Use the storage class that reflects the tenant's service tier, and create the class in Settings → Storage class before referencing it in a rule.

4. Test with Representative Objects​

Test boundary values for object.size, the file extensions and content types used by your tenants, and keys that should not match. Include at least one object per storage class.

5. Export Before Large Edits​

Export downloads the current document. Keep the exported file in version control, and use Import to restore or move the document between gateways. Imported content remains unsaved until you click Apply changes.

6. Document Your Rules​

Maintain documentation of the routing logic for team reference and compliance:

  • What business requirement does the rule address?
  • What are the expected outcomes?
  • Who owns the rule?

Troubleshooting​

Apply Changes Is Disabled​

The Apply changes button stays disabled when:

  • The document has no unsaved changes.
  • The document is not valid JSON or is not a JSON object.
  • An apply is already in progress.

If the page shows the empty state instead of the editor, there is no routing rule to edit.

Server-Side Validation Fails​

The gateway validates routing policy semantics after Apply changes. Validation errors are displayed below the editor. Common causes:

  • inputStorageClass is not defined for the gateway. Create the class in Settings → Storage class or use an existing class.
  • redundancyClassId is not a valid UUID or is not associated with the gateway through a connection.
  • A condition uses an operator that is not supported by the selected field, or a regular expression that does not compile.
  • A required field is missing: name, inputStorageClass, and action.
  • name is longer than 100 characters.
  • A drop action defines a target.

Import Fails​

The import dialog accepts only .json files up to 10 MB and rejects files that are not valid JSON. After a successful import, the document is loaded into the editor as unsaved changes; click Apply changes to send it to the gateway.

The Page Shows "No routing rule found"​

The gateway has no routing rule to edit. Routing rules are configured with the gateway; check the gateway connections or contact support.

Rules Are Not Being Applied​

Verify that:

  1. The rule has "enabled": true.
  2. The upload uses the storage class in inputStorageClass.
  3. The object metadata matches the conditions, including the caseSensitive setting.
  4. Another rule for the same storage class is not matching first. The first match wins.
  5. The rule is applied. The Live JSON header must not show Unsaved changes after the apply.

If no rule matches, the upload is accepted and stored with the fallback redundancy class.

Uploads Are Rejected With InvalidStorageClass​

The upload specifies a storage class that is not defined for the gateway. Create the class in Settings → Storage class or use one of the existing classes. An upload without x-amz-storage-class uses STANDARD.

Unmatched Uploads Use an Unexpected Redundancy Class​

Unmatched uploads use the fallback redundancy class from the gateway's default connection. Check Settings → Connections to confirm which connection is marked as default. The default connection cannot be deleted.