Smart Data Placement

Overview

Smart Data Placement is an advanced feature that allows you to configure intelligent rules for automatically routing objects uploaded by your customers to different redundancy classes in your swarm. As a DS3 Gateway owner, you can define rule-based policies that determine where each piece of data is stored based on customizable criteria.

With Smart Data Placement, your customers select a storage class (HOT, STANDARD, COLD) when uploading files, and the gateway automatically applies your placement rules to determine the most appropriate final redundancy class.

Why Smart Data Placement?

This feature is essential for resellers and gateway owners who want to:

Optimize costs: Route large or infrequently accessed files to more economical redundancy classes
Ensure compliance: Automatically enforce data locality and retention policies
Improve performance: Direct time-sensitive data to high-performance redundancy classes
Offer differentiated services: Create service tiers with different storage characteristics
Automate management: Eliminate the need for manual intervention in data routing

info

Smart Data Placement is available for private gateways that you own and manage. As a gateway owner, you configure the routing policies, while your customers select storage classes when uploading data.

How It Works

The Smart Data Placement workflow operates at two levels:

1. Client-Level Selection

Your customers, when uploading files via S3 API, specify a storage class in the upload request using the standard S3 header x-amz-storage-class. Common storage classes include:

HOT - For frequently accessed, performance-critical data
STANDARD - For general-purpose storage
COLD - For archival and infrequently accessed data

You define the pricing and SLAs for each storage class tier.

2. Gateway-Level Routing

The gateway evaluates your configured placement rules against the upload request metadata (storage class, file name, size, content type) and determines the final redundancy class where the data will be stored.

Customer Upload → Storage Class Selection → Rule Evaluation → Redundancy Class Assignment → Data Storage

Complete Flow

┌─────────────┐
│  Customer   │ Uploads file with storage class = "STANDARD"
└──────┬──────┘
       │
       ▼
┌──────────────────┐
│  DS3 Gateway     │ Evaluates configured placement rules
│  (Smart Data     │ - File name: *.mp4
│   Placement)     │ - Size: >100MB
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│ Matched Rule     │ "video-files-to-cold-storage"
│ (First in Array) │
└──────┬───────────┘
       │
       ▼
┌──────────────────┐
│ Redundancy Class │ rc-cold-storage-uuid
│ rc-cold-storage  │
└──────────────────┘

Prerequisites

Before configuring Smart Data Placement, ensure you have:

A private DS3 Gateway deployed and operational
At least one redundancy class configured in your swarm
Composer CLI installed and configured (available on GitHub)
Gateway owner permissions to configure placement policies
Customer ID with access to the gateway

Configuration Steps

Step 1: Configure Default Redundancy Class

Smart Data Placement is automatically enabled when you configure placement policies. You only need to specify a default redundancy class that will be used as fallback when no rules match.

# Configure your gateway with a default redundancy class
cubbit --profile <profile> gateway edit \
  --tenant-id <customer-id> \
  --gateway-id <gateway-id> \
  --default-redundancy-class-id <default-redundancy-class-uuid>

Example:

cubbit --profile dev gateway edit \
  --tenant-id 00000000-0000-0000-0000-000000000000 \
  --gateway-id 550e8400-e29b-41d4-a716-446655440000 \
  --default-redundancy-class-id 12345678-1234-1234-1234-123456789abc

Example:

cubbit --profile dev gateway edit \
  --tenant-id 00000000-0000-0000-0000-000000000000 \
  --gateway-id 550e8400-e29b-41d4-a716-446655440000 \
  --default-redundancy-class-id 12345678-1234-1234-1234-123456789abc

note

The default-redundancy-class-id is the fallback redundancy class used when no placement rules match an upload request. Smart Data Placement becomes active automatically once you apply placement policies in Step 3.

Step 2: Define Placement Rules

Placement rules are defined as a JSON array. Each rule specifies:

Conditions: When the rule applies (file patterns, sizes, content types, etc.)
Action: What to do when the rule matches (route to a redundancy class or reject)
Input Storage Class: Which storage class triggers this rule (STANDARD, HOT, or COLD)

When multiple placement policies are configured, they are evaluated against the upload's storage class and object metadata. The first matching policy determines the action.

Rule Configuration Structure

[
  {
    "id": "unique-rule-id",
    "name": "Rule Display Name",
    "description": "Human-readable description",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {
      "operator": "AND",
      "conditions": [...]
    },
    "action": {
      "type": "route",
      "target": {
        "type": "fixed",
        "redundancyClassId": "<target-rc-uuid>"
      }
    }
  }
]

Step 3: Apply Placement Policies

Upload your placement rules configuration to the gateway using the Composer CLI:

# Create policies.json file with your rules
# Then apply them to your gateway
cubbit --profile <profile> gateway edit \
  --tenant-id <customer-id> \
  --gateway-id <gateway-id> \
  --smart-data-placement-policies "$(cat policies.json)"

Example:

cubbit --profile dev gateway edit \
  --tenant-id 00000000-0000-0000-0000-000000000000 \
  --gateway-id 550e8400-e29b-41d4-a716-446655440000 \
  --smart-data-placement-policies "$(cat policies.json)"

tip

For complex policies, always use a JSON file instead of inline JSON. This makes it easier to:

Edit and maintain rules
Version control with git
Reuse across multiple gateways
Validate JSON structure before applying

Rule Conditions

You can create rules based on various object metadata fields:

Available Fields

Field	Description	Example
`object.key`	File name/path	`videos/movie.mp4`
`object.size`	Object size in bytes	`104857600` (100MB)
`object.content_type`	MIME type	`video/mp4`
`tenant.name`	The tenant name	`cubbit`

note

Some advanced metadata fields may still be unsupported and will be introduced in future versions.

Supported Operators

Operator	Description	Value Type	Example
`equals`	Exact match	string/number	`"value": "exact"`
`not_equals`	Not equal	string/number	`"value": "exact"`
`matches_regex`	Regular expression	string (regex)	`"value": "\\.mp4$"`
`not_matches_regex`	Not matching regex	string (regex)	`"value": "\\.log$"`
`starts_with`	Prefix match	string	`"value": "backup/"`
`ends_with`	Suffix match	string	`"value": ".mp4"`
`contains`	Contains substring	string	`"value": "/temp/"`
`not_contains`	Does not contain	string	`"value": "/temp/"`
`gt`	Greater than (numbers)	number	`"value": 104857600`
`lt`	Less than (numbers)	number	`"value": 1048576`
`gte`	Greater than or equal	number	`"value": 100`
`lte`	Less than or equal	number	`"value": 1000`
`in`	Value in list	array	`"value": ["mp4", "avi", "mov"]`
`not_in`	Value not in list	array	`"value": ["tmp", "log"]`
`exists`	Field exists	N/A	No value needed
`not_exists`	Field does not exist	N/A	No value needed

Logical Operators

Conditions are grouped using logical operators:

AND: All conditions in the group must match
OR: At least one condition in the group must match

Conditions can be nested recursively to create complex decision trees.

{
  "operator": "AND",
  "conditions": [
    {
      "field": "object.key",
      "operator": "matches_regex",
      "value": "\\.(mp4|avi|mov)$"
    },
    {
      "operator": "OR",
      "conditions": [
        {
          "field": "object.size",
          "operator": "gt",
          "value": 104857600
        },
        {
          "field": "object.content_type",
          "operator": "equals",
          "value": "video/mp4"
        }
      ]
    }
  ]
}

Action Types

Route to Fixed Redundancy Class

Route matched uploads to a specific redundancy class:

{
  "action": {
    "type": "route",
    "target": {
      "type": "fixed",
      "redundancyClassId": "rc-cold-storage-uuid"
    }
  }
}

Drop Upload

Reject uploads that match certain criteria:

{
  "action": {
    "type": "drop"
  }
}

note

Advanced target types like fieldMapping and loadBalancing are planned for future implementation but are not currently available.

Complete Examples

Example 1: Route Video Files to Cold Storage

Route all video files larger than 100MB to cold storage:

[
  {
    "id": "video-to-cold-storage",
    "name": "Large Video Files to Cold Storage",
    "description": "Route video files larger than 100MB to cold storage",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {
      "operator": "AND",
      "conditions": [
        {
          "field": "object.key",
          "operator": "matches_regex",
          "value": "\\.(mp4|avi|mov|mkv|webm|flv)$",
          "caseSensitive": false
        },
        {
          "field": "object.size",
          "operator": "gt",
          "value": 104857600
        }
      ]
    },
    "action": {
      "type": "route",
      "target": {
        "type": "fixed",
        "redundancyClassId": "rc-cold-storage-uuid"
      }
    }
  }
]

Example 2: Block Executable Files

Prevent upload of potentially dangerous executable files:

[
  {
    "id": "block-executables",
    "name": "Block Executable Files",
    "description": "Drop potentially dangerous executable files",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {
      "operator": "OR",
      "conditions": [
        {
          "field": "object.key",
          "operator": "matches_regex",
          "value": "\\.(exe|bat|sh|com|scr|dll)$",
          "caseSensitive": false
        },
        {
          "field": "object.content_type",
          "operator": "starts_with",
          "value": "application/x-executable"
        }
      ]
    },
    "action": {
      "type": "drop"
    }
  }
]

Example 3: Backup Folder to Archive

Route backup files to long-term archive storage:

[
  {
    "id": "backup-to-archive",
    "name": "Backup Files to Archive",
    "description": "Route backup folders to archive storage",
    "enabled": true,
    "inputStorageClass": "COLD",
    "conditions": {
      "operator": "OR",
      "conditions": [
        {
          "field": "object.key",
          "operator": "contains",
          "value": "/backup/"
        },
        {
          "field": "object.key",
          "operator": "starts_with",
          "value": "backups/"
        }
      ]
    },
    "action": {
      "type": "route",
      "target": {
        "type": "fixed",
        "redundancyClassId": "rc-archive-uuid"
      }
    }
  }
]

Example 4: Small Files Optimization

Route small files to fast access storage:

[
  {
    "id": "small-files-optimization",
    "name": "Optimize Small Files",
    "description": "Route files smaller than 10MB to fast access RC",
    "enabled": true,
    "inputStorageClass": "HOT",
    "conditions": {
      "operator": "AND",
      "conditions": [
        {
          "field": "object.size",
          "operator": "lt",
          "value": 10485760
        },
        {
          "field": "object.key",
          "operator": "not_matches_regex",
          "value": "\\.(tmp|log|cache)$"
        }
      ]
    },
    "action": {
      "type": "route",
      "target": {
        "type": "fixed",
        "redundancyClassId": "rc-fast-access-uuid"
      }
    }
  }
]

Example 5: Multiple Rules in One Policy

Combine multiple rules in a single policy file:

[
  {
    "id": "block-executables",
    "name": "Block Executable Files",
    "description": "Drop executable files for security",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {
      "operator": "OR",
      "conditions": [
        {
          "field": "object.key",
          "operator": "matches_regex",
          "value": "\\.(exe|bat|sh)$",
          "caseSensitive": false
        }
      ]
    },
    "action": {
      "type": "drop"
    }
  },
  {
    "id": "video-to-cold-storage",
    "name": "Large Video Files to Cold Storage",
    "description": "Route large video files to cold storage",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {
      "operator": "AND",
      "conditions": [
        {
          "field": "object.key",
          "operator": "matches_regex",
          "value": "\\.(mp4|avi|mov)$",
          "caseSensitive": false
        },
        {
          "field": "object.size",
          "operator": "gt",
          "value": 104857600
        }
      ]
    },
    "action": {
      "type": "route",
      "target": {
        "type": "fixed",
        "redundancyClassId": "rc-cold-storage-uuid"
      }
    }
  }
]

Rule Evaluation

When multiple placement policies are configured:

Only policies with enabled: true are evaluated
Policies are evaluated against the upload's storage class and object metadata
The first matching policy determines the action (route or drop)
If no policies match, the default_redundancy_class_id is used as fallback
Policies with action.type: "drop" reject the upload immediately

Testing Your Configuration

Test Upload Scenarios

After configuring your rules, test them with sample uploads:

# Upload a video file (should route to cold storage if rule matches)
aws s3 cp large-video.mp4 s3://your-bucket/videos/ \
  --storage-class STANDARD \
  --endpoint-url https://your-gateway.example.com

# Upload a small document (should use default RC)
aws s3 cp document.pdf s3://your-bucket/docs/ \
  --storage-class HOT \
  --endpoint-url https://your-gateway.example.com

# Upload an executable (should be rejected if block rule is active)
aws s3 cp program.exe s3://your-bucket/apps/ \
  --storage-class COLD \
  --endpoint-url https://your-gateway.example.com

Verify Configuration

Check that your policies are correctly applied:

cubbit --profile <profile> gateway describe \
  --tenant-id <tenant-id> \
  --gateway-id <gateway-id> \
  --output json

Best Practices

1. Start Simple

Begin with a few basic rules and gradually add complexity as needed:

[
  {
    "id": "block-large-uploads",
    "name": "Block Large Uploads",
    "description": "Prevent uploads larger than 1GB",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": [
      {
        "field": "object.size",
        "operator": "gt",
        "value": 1073741824
      },
      "action": {
        "type": "drop"
      }
    ]
  }
]

2. Use Descriptive IDs and Names

Make your rules easy to identify and maintain:

{
  "id": "video-files-large-than-100mb-to-cold-storage",
  "name": "Large Video Files → Cold Storage",
  "description": "Redirects video files (mp4, avi, mov) larger than 100MB to cold storage redundancy class for cost optimization"
}

3. Order Rules by Specificity

Place more specific rules earlier in the array:

[
  {
    "id": "security-block-executables",
    "name": "Block Executables",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {...},
    "action": {"type": "drop"}
  },
  {
    "id": "general-video-routing",
    "name": "Video Files to Cold Storage",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {...},
    "action": {"type": "route"}
  }
]

4. Test Before Production

Always test your rules in a staging environment before deploying to production:

# First apply to the test gateway
cubbit --profile dev gateway edit \
  --tenant-id <test-customer-id> \
  --gateway-id <test-gateway-id> \
  --smart-data-placement-policies "$(cat test-rules.json)"

# Run test uploads
# Verify the results
# Adjust the rules as needed

5. Document Your Rules

Maintain documentation of your routing logic for team reference and compliance:

What business requirement does each rule address?
What are the expected outcomes?
Who owns each rule?

Troubleshooting

Rules Not Being Applied

Problem: Uploads are not being routed as expected

Solutions:

Check rule order - earlier rules in the array may be matching first
Verify condition syntax - ensure operators and values are correct
Check storage class - rules are scoped to specific input storage classes
Ensure rules are not explicitly disabled (set enabled: false to disable a rule)

Default Fallback Not Working

Problem: Uploads fail when no rules match

Solutions:

Verify default_redundancy_class_id is set correctly on the gateway
Ensure the default redundancy class exists and is active
Verify that placement policies have been applied to the gateway

Invalid Rule Configuration

Problem: Policies fail to apply via CLI

Solutions:

Validate JSON syntax using a JSON validator
Ensure all required fields are present (id, name, inputStorageClass, action). The conditions field is optional and, when omitted or empty, the rule matches all uploads.
Verify redundancy class IDs are valid UUIDs
Check that rule IDs are unique within the policy array
Use a JSON file instead of inline JSON for complex policies

CLI JSON Syntax Errors

Problem: Composer CLI returns JSON parsing errors

Solutions:

Use a JSON file and pipe content instead of inline JSON:

# Create policy.json file
cat > policy.json << 'EOF'
[
  {
    "id": "my-rule",
    "name": "My Rule",
    "enabled": true,
    "inputStorageClass": "STANDARD",
    "conditions": {...},
    "action": {...}
  }
]
EOF

# Use with CLI
cubbit --profile dev gateway edit \
  --tenant-id YOUR-CUSTOMER-ID \
  --gateway-id YOUR-GATEWAY-ID \
  --smart-data-placement-policies "$(cat policy.json)"

API Reference

Update Gateway with SDP Policies

Endpoint: PATCH /v1/tenants/{tenant_id}/gateways/{id}

Request Body:

{
  "default_redundancy_class_id": "uuid",
  "placement_policies": [
    { /* Policy object */ }
  ]
}

Smart Data Placement is automatically activated when placement_policies are provided.

Get Gateway Configurations

Endpoint: GET /v1/tenants/{tenant_id}/gateways/{id}/configurations

Response: Includes placement_policies array and default_redundancy_class_id

Overview​

Why Smart Data Placement?​

How It Works​

1. Client-Level Selection​

2. Gateway-Level Routing​

Complete Flow​

Prerequisites​

Configuration Steps​

Step 1: Configure Default Redundancy Class​

Step 2: Define Placement Rules​

Rule Configuration Structure​

Step 3: Apply Placement Policies​

Rule Conditions​

Available Fields​

Supported Operators​

Logical Operators​

Action Types​

Route to Fixed Redundancy Class​

Drop Upload​

Complete Examples​

Example 1: Route Video Files to Cold Storage​

Example 2: Block Executable Files​

Example 3: Backup Folder to Archive​

Example 4: Small Files Optimization​

Example 5: Multiple Rules in One Policy​

Rule Evaluation​

Testing Your Configuration​

Test Upload Scenarios​

Verify Configuration​

Best Practices​

1. Start Simple​

2. Use Descriptive IDs and Names​

3. Order Rules by Specificity​

4. Test Before Production​

5. Document Your Rules​

Troubleshooting​

Rules Not Being Applied​

Default Fallback Not Working​

Invalid Rule Configuration​

CLI JSON Syntax Errors​

API Reference​

Update Gateway with SDP Policies​

Get Gateway Configurations​

Related Documentation​

Overview

Why Smart Data Placement?

How It Works

1. Client-Level Selection

2. Gateway-Level Routing

Complete Flow

Prerequisites

Configuration Steps

Step 1: Configure Default Redundancy Class

Step 2: Define Placement Rules

Rule Configuration Structure

Step 3: Apply Placement Policies

Rule Conditions

Available Fields

Supported Operators

Logical Operators

Action Types

Route to Fixed Redundancy Class

Drop Upload

Complete Examples

Example 1: Route Video Files to Cold Storage

Example 2: Block Executable Files

Example 3: Backup Folder to Archive

Example 4: Small Files Optimization

Example 5: Multiple Rules in One Policy

Rule Evaluation

Testing Your Configuration

Test Upload Scenarios

Verify Configuration

Best Practices

1. Start Simple

2. Use Descriptive IDs and Names

3. Order Rules by Specificity

4. Test Before Production

5. Document Your Rules

Troubleshooting

Rules Not Being Applied

Default Fallback Not Working

Invalid Rule Configuration

CLI JSON Syntax Errors

API Reference

Update Gateway with SDP Policies

Get Gateway Configurations

Related Documentation