Setting Up S3 Credentials for Ingest Cluster

In order to take advantage of cluster-based ingest, the cluster must be able to read the contents of the bucket. If the source bucket is encrypted with KMS, then the cluster will also need the ability to decrypt objects using that key.

Before you start, you will need to request the ARN of the ingest worker role from Flywheel Support.

Instruction Steps

Add Read Access to the Source S3 Bucket

1. Log in to the AWS console and find your source bucket in the Amazon S3 services.

2. Go to the Permissions tab:

   

3. Click the Edit button next to Bucket Policy:

   

4. Add read access for the ingest worker role.

   In this example, we'll use a role ARN of arn:aws:iam::784277622827:role/example-fwapp-ingest-role

   {  
    "Version": "2012-10-17",  
    "Id": "Policy1580769485782",  
    "Statement": [  
      {  
        "Sid": "ListBucket",  
        "Effect": "Allow",  
        "Principal": {  
          "AWS": [  
            "arn:aws:iam::784277622827:role/example-fwapp-ingest-role"  
          ]  
        },  
        "Action": [  
          "s3:GetBucketLocation",  
          "s3:ListBucket"  
        ],  
        "Resource": "arn:aws:s3:::dev-import-testing"  
      },  
      {  
        "Sid": "GetObject",  
        "Effect": "Allow",  
        "Principal": {  
          "AWS": [  
            "arn:aws:iam::784277622827:role/example-fwapp-ingest-role"  
          ]  
        },  
        "Action": "s3:GetObject",  
        "Resource": "arn:aws:s3:::dev-import-testing/\*"  
      }  
    ]  
   }  
   

5. Save the bucket policy. If your bucket has a default encryption policy, move on to the next section.

Add Decrypt Access to the KMS Key

If your bucket has a default encryption policy, then you'll also need to add decrypt access to the associated KMS key.

1. Navigate to the bucket settings to view the default encryption policy

   

2. If encryption is enabled, click on the encryption key ARN to go to the settings for that key.

3. Click Edit on the key policy:

   

   Here we're adding a new statement that allows decryption using the KMS key:

   (NOTE: Just the second statement was added)

   {  
      "Version": "2012-10-17",  
      "Id": "key-default-1",  
      "Statement": [  
          {  
              "Sid": "Enable IAM User Permissions",  
              "Effect": "Allow",  
              "Principal": {  
                  "AWS": "arn:aws:iam::784277622827:root"  
              },  
              "Action": "kms:\*",  
              "Resource": "\*"  
          },  
          {  
              "Sid": "Enable Decrypt for Flywheel Ingest",  
              "Effect": "Allow",  
              "Principal": {  
                  "AWS": "arn:aws:iam::784277622827:role/example-fwapp-ingest-role"  
              },  
              "Action": "kms:Decrypt",  
              "Resource": "\*"              
          }  
      ]  
   }
   

4. Click Save

This completes the changes needed to enable cluster-based ingest for your source bucket. Repeat these steps for any additional buckets you'd like to import data from.