Skip to main content

Search

How can we help you?

How do I configure AWS Glue Credentials?

How to Connect to AWS Glue

This guide outlines the process of creating an IAM user, configuring the necessary permissions, and adding the credentials to the ThinkData Platform to connect to an AWS Glue data catalog.

Step-by-Step Guide:

Step 1: Create an IAM User

To create an IAM user for AWS Glue access, follow these steps:

  1. Log in to the AWS Management Console and navigate to the IAM service.

  2. Navigate to Users > Create user.

  3. Enter a user name (e.g., ThinkDataGlueUser) and click Next.

    Important: This user name will be used to identify the credentials in AWS. You will use the Access Key ID and Secret Access Key (created in Step 3) when adding credentials to the ThinkData Platform.

  4. Select "Attach policies directly" on the permissions page.

  5. Search for and attach the AWS managed policy AWSGlueServiceRole. This provides the necessary permissions for AWS Glue operations.

  6. Click Next and then Create user to complete the user creation.

Step 2: Create and Attach Customer Managed S3 Policy

In addition to the AWSGlueServiceRole, create a customer managed policy to grant S3 bucket access:

  1. Navigate to IAM > Policies > Create Policy and select the JSON tab.

  2. Enter the following policy document: 

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3GlobalPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:PutAccountPublicAccessBlock",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListJobs",
                "s3:CreateJob",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        },
        {
            "Sid": "S3BucketPermissions",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::your-bucket",
                "arn:aws:s3:::your-bucket/*"
            ]
        }
    ]
}

    Replace your-bucket with the name of your S3 bucket.

    Important: The first statement grants account-level S3 permissions required for Glue operations. The second statement grants full S3 access to your specific bucket and its contents.

  3. Name the policy (e.g., ThinkDataGlueS3Policy) and click Create Policy.

  4. Attach the policy to the IAM user:

    • Navigate to IAM > Users and select the user created in Step 1.
    • Click Add permissions > Attach policies directly.
    • Search for and select the policy you just created (ThinkDataGlueS3Policy).
    • Click Add permissions.

The IAM user should now have two policies attached:

  • AWSGlueServiceRole (AWS managed)
  • ThinkDataGlueS3Policy (Customer managed)

Step 3: Create Access Keys

Generate access keys for programmatic access to AWS Glue:

  1. Navigate to IAM > Users and select the user created in Step 1.

  2. Go to the Security credentials tab and click Create access key.

  3. Select "Application running outside AWS" as the use case and click Next.

  4. Add a description tag (optional) and click Create access key.

  5. Save the Access Key ID and Secret Access Key securely. You will need these when adding credentials to the ThinkData Platform.

    Important: The Secret Access Key is only shown once. Store it in a secure location immediately after creation.

Step 4: Add AWS Glue Credentials to the ThinkData Platform

After creating the IAM user and permissions in AWS, add the credentials to the ThinkData Platform:

  1. Open the platform and navigate to the Sources page:
    • Data > Sources > Create Connection
  2. Enter a name for the source (e.g., "AWS Glue Production") in the Source Name field.
  3. Configure the following Credential Details:
    External Warehouse Type Iceberg
    File Format Select one: parquet, avro, or orc
    Compression Codec Specify the compression codec (e.g., snappy, gzip, zstd)
    Glue Metastore Region Specify the AWS region (e.g., us-east-1)
    Glue Metastore Endpoint This is typically: https://glue.{region}.amazonaws.com
    Default Warehouse Directory This may be left blank. Enter your S3 bucket name (e.g., s3://your-bucket/warehouse/)
    Catalog ID Enter the ID of your Glue Metastore Catalog (typically your AWS Account ID)
    S3 File System Endpoint https://s3.{region}.amazonaws.com (e.g., https://s3.us-east-1.amazonaws.com)
    Region Specify the AWS region (e.g., us-east-1)
    Application ID Leave blank unless you need to override
    External Warehouse Schema Enter the database name from the Glue Metastore
  4. Under Credential, click Create New and enter:
    • Access Key ID: The access key ID created in Step 3
    • Secret Access Key: The secret access key created in Step 3
  5. Click Save to store the credentials.

Step 5: Using the Credentials

Now, you can use these credentials with a: