In order to take advantage of cluster based ingest, the cluster will need to have the ability to read the contents of the bucket. If the source bucket is encrypted with KMS, then the cluster will also need the ability to decrypt objects using that key.
Before you can start, you will need to request the ARN of the ingest worker role from Flywheel Support.
Add read access to the source S3 bucket:
Step 1 - Login to the AWS console and find your source bucket in the Amazon S3 services and go to the "Permissions" tab:
Step 2 - Click the Edit button next to Bucket Policy:
Step 3 - Add read access for the ingest worker role.
In this example, we'll use a role ARN of arn:aws:iam::784277622827:role/example-fwapp-ingest-role
{
"Version": "2012-10-17",
"Id": "Policy1580769485782",
"Statement": [
{
"Sid": "ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::784277622827:role/example-fwapp-ingest-role"
]
},
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::dev-import-testing"
},
{
"Sid": "GetObject",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::784277622827:role/example-fwapp-ingest-role"
]
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::dev-import-testing/*"
}
]
}
Save the bucket policy. If your bucket has a default encryption policy, move on to the next section.
Add decrypt access to the KMS key:
If your bucket has a default encryption policy, then you'll also need to add decrypt access to the associated KMS key. You can see this policy the "Settings" of the bucket:
If encryption is turned on, click on the encryption key ARN to go to the settings for that key, then click Edit on the key policy:
Here we're adding a new statement that allows decryption using the KMS key:
(NOTE: Just the second statement was added)
{
"Version": "2012-10-17",
"Id": "key-default-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::784277622827:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Enable Decrypt for Flywheel Ingest",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::784277622827:role/example-fwapp-ingest-role"
},
"Action": "kms:Decrypt",
"Resource": "*"
}
]
}
Then click save.
This completes the changes needed to enable cluster based ingest for your source bucket. You will need to repeat these steps for any additional buckets you'd like to import data from.