How to connect RDS with AWS IAM authentication

Leave a reply

A quick post listing step by step procedure to connect RDS database configured with IAM authentication.

RDS with IAM authentication

We are considering RDS running MySQL and configured with AWS IAM authentication option throughout this post. However, if you are using a different database engine, consider editing commands/arguments whenever necessary.

Basically, we will be creating a local database user that leverages AWS IAM for authentication. Then EC2 will be configured with an IAM role or aws configure that has appropriate policies defined. And lastly, the user will generate an authentication token and log into the RDS database.

Why should we use IAM authentications for RDS?

Here is a list of reasons that are helpful to understand the benefits of the IAM authentications option for RDS.

  1. IAM tokens used to log into the RDS database are valid for 15 minutes only. So they are more secure than permanent username/password pairs, and administrators don’t need to enforce/manage password reset policies.
  2. IAM tokens are generated by making API calls to the AWS IAM service whenever needed. As a result, storing tokens is useless, and even if someone does, that does not pose a security threat due to its short life.
  3. Applications can use EC2 instance profiles for generating tokens, so there is no need to store authentication information anywhere for applications to consume.

What you need?

You should be equipped with below inventory before hand –

  1. RDS instance up and running configured with IAM DB authentication.
  2. EC2 instance configured with AWS CLI (make sure SG allows the connectivity between EC2 and RDS on database port)
  3. Master user login to RDS datatase
  4. Access to AWS IAM service.

Creating user on database for the RDS access

For this step, you need to log in to the RDS database with the master user and create a new user. If you are on windows, you can use a lightweight tool like Sqlectron, or if you are already on EC2, you can use CLI as well –

Create DB user

For SQL CLI :

[root@kerneltalks ~]# mysql -h kerneltalks-rds.cn8uwrapea4b.us-east-1.rds.amazonaws.com -P 3306 -u admin -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 26
Server version: 8.0.20 Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> CREATE USER 'kt_iam_user' IDENTIFIED WITH AWSAuthenticationPlugin as 'RDS' REQUIRE SSL;
Query OK, 0 rows affected (0.01 sec)

Make necessary changes to username and RDS endpoint!

Create IAM role for RDS access

  • Navigate to AWS IAM
  • Create the IAM policy (sample policy below)
  • If you intend to use the EC2 instance profile, create an IAM role for AWS service EC2 and attach the IAM policy to it.
  • If you intend to use only IAM users, then make sure you configure your CLI with aws configure command by supplying access key ID and secret access key.
  • Make necessary changes to the resource section!
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:us-east-1:986532147:dbuser:db-kerneltalks-rds/kt_iam_user"
         ]
      }
   ]
}

Download the SSL root certificate available for all regions from S3 bucket. This certificate required while making RDS connection since we enforced SSL on the database user. This ensures data is encrypted in flight.

[root@kerneltalks ~]#  wget https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem
--2021-07-03 14:44:12--  https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.205.189
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.205.189|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1456 (1.4K) [binary/octet-stream]
Saving to: ‘rds-ca-2019-root.pem’

100%[===================================================================================================================================>] 1,456       --.-K/s   in 0s      

2021-07-03 14:44:12 (81.2 MB/s) - ‘rds-ca-2019-root.pem’ saved [1456/1456]

Now your EC2 or IAM user is ready to access RDS.

Connect to RDS using IAM token

It’s time for you to generate an IAM token and connect to RDS. We will save the token in the Shell variable for easy management and pass that shell variable token into RDS connect command. If the downloaded certificate is not in the PWD then use the absolute path for the pem file.

[root@kerneltalks ~]# token=`aws rds generate-db-auth-token --hostname kerneltalks-rds.cn8uwrapea4b.us-east-1.rds.amazonaws.com  --port 3306 --region us-east-1 --usernam
e kt_iam_user`
[root@kerneltalks ~]# mysql -h kerneltalks-rds.cn8uwrapea4b.us-east-1.rds.amazonaws.com -P 3306 --ssl-ca=rds-ca-2019-root.pem -u kt_iam_user --password=$token --protocol
=tcp
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 59
Server version: 8.0.20 Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>

And you are connected!

Here we have used IAM token as password to connect to RDS database! Make a note that, these generated tokens are good only for 15 minutes. If you are using connect command after 15 minutes you need to generate the token again.

Just for anote, token looks like this –

[root@kerneltalks ~]# echo $token
kerneltalks-rds.cn8uwrapea4b.us-east-1.rds.amazonaws.com:3306/
NXZG4BNZMCVY%2F20210703%2Fus-east-1%2Frds-db%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210703T153110Z&X-Amz-Signature=04605698c01f3de6b2d7a7a48ddf28008d290795e11e
e4ac89587cbb3cdd9661

Understanding the authentication flow

For your understanding, lets see how authentication flows under the hood –

  1. User runs the token generation command with database name, port, region and username for which token to be generated.
  2. RDS sends back API token with 15 min lifespan. It requires your EC2 Instance role/IAM user to have rds connect permission. We covered this by defining IAM policy.
  3. The user attempt to connect to RDS using the token acquired in the previous step.
  4. A secure connection establishes, and the user logs in only if
    • The root certificate is valid
    • IAM permissions are in place and valid for db:connect
    • Supplied username has RDS authentication set in the database
    • The supplied token is generated for the same username, and currently, it’s not expired.
  5. A user granted access and logs in. SQL prompt will be presented.

How to connect AWS RDS database from Windows

Leave a reply

A quick article for AWS beginners on connecting the AWS RDS database from Windows.

A step by step RDS database login procedure using the lightweight SQL client. This article is intended for AWS beginners who wanted to learn about RDS service with a little bit of database hands-on. Being said that, there is always a question for the non-DB guys; “How did I connect with this RDS database?”. So here we will walk you through step by step procedure for doing that.

For connecting to RDS database you should have –

  1. RDS database up and running
  2. RDS database endpoint
  3. A SQL client
  4. Connectivity between your machine and RDS database

Throughout this article, I am considering the MySQL database on RDS. If you are using a different database engine on RDS, then the connection port may vary. Also, we are considering ‘Password authentication’ of authentications options here –

  1. Password authentication: RDS configured with this option has users managed at the database level.
  2. AWS IAM database authentication: RDS configured with this option authenticates users by leveraging AWS IAM service. As a result, users are managed outside the database.

Let’s get started!

Identify RDS Endpoint

  • Navigate to the RDS console
  • Go to Databases and select your database
  • Database details screen should open up where you can copy your RDS endpoint.
RDS endpoint

Prepare SQL client

  • Download lightweight SQL client Sqlectron
  • Install and open the Sqlectron client.
Sqlectron client!

Click on Add button to enter RDS connection details. We are testing RDS with ‘Password authentication’ here hence user and password needs to be supplied.

RDS connection details in SQL Client

Click on Test to verify the connectivity.

If you encounter the below error, try to connect from a machine with direct connectivity to RDS. You can analyse RDS security groups to determine allowed subnets.

DB_CONNECT Error

Verify below things –

  • There are no firewall rules between your machine and RDS blocking the port 3306 traffic.
  • RDS database is publicly accessible. (Obv. It applies to testing POC databases). If not, you can configure RDS to be publicly accessible.
  • If you don’t want your RDS to be publicly accessible, you need to connect RDS from the allowed subnets. That means your machine needs to be in the same VPC as RDS (e.g. Windows EC2 in the same VPC)

How to make RDS publicly accessible

Not recommended for production RDS instances or RDS carrying sensitive data.

  • Navigate to RDS console and respective database
  • Select Modify from the action menu for that particular database
  • Goto Connectivity section and expand Additional configuration
  • Here choose the radio button against Publicly accessible and Apply the changes
RDS Public Access

RDS connection using ‘Password authentication’ via SQL client

Once you sort out RDS connectivity issues, go back to the SQL client and try Test again. Now you should be able to succeed.

Now, click on Save to save this configuration in SQL client. And then hit Connect to connect to your RDS database.

Connect RDS using Client

After connecting, you can see schema on the left-hand side and a command box on the right-hand side to execute commands on the database.

Running SQL commands on RDS

Command outputs or messages will be shown in the blank space below the command text area. You will be able to download outputs in JSON, CSV format or even copy them directly.

Running SQL commands with Sqlectron!

That’s all! You can use this light weight SQL client to get started with RDS immediately.

Preparing for Hashicorp Certified Terraform Associate Exam

Leave a reply

A quick article that helps you preparing for Hashicorp Certified Terraform Associate Exam

Terraform Associate exam!

In this quick post, I would like to share some of the resources that help you clear the terraform associate exam. Feel free to add resources you know in the comments section, which may help fellow readers.

The terraform associate exam is designed to test the candidate’s readiness towards IaC, i.e. Infrastructure as code. IaC concepts, terraform CLI hands-on (a lot of it) and knowledge on terraform’s paid offerings through Cloud or Enterprise should get you through this exam. It’s a practitioner level exam, so it shouldn’t be hard to beat if you have IaC and cloud background.

You must have researched already about the exam on its official page, but here are quick facts for your reference.

Topics to study

I suggest you have good hands-on with terraform CLI before taking this exam. It will help you cover the majority of topics, and you don’t have to learn them during preparation. That leaves you with minimal topics to prepare for actual certification.

Hashicorp’s study guide is the best resource to follow along for preparation. Let me quickly list down a couple of topics you should not miss during preparation –

  • IaC concepts
    • Traditional infra provisioning v/s IaC
  • Terraform basic workflow
    • Write, plan and apply.
  • Different types of blocks in terraform code
  • Terraform CLI commands (a huge list of them!)
  • Terraform Modules, functions, state files
    • At least go through all functions once.
    • Lots of hands-on to understand how modules works
    • State management (a big topic!)
  • Debugging and variables
    • Different ways to handle variables
    • Debugging levels, ways to set them, logging in files
  • Detailed understanding of Terraform cloud and enterprise
    • Free and paid offerings in each type
    • Sentinal, workspaces, remote runs etc. understanding
    • Clustering, OS availability in each type

Resources for preparation

Assorted list of online resources you can leverage to follow along your preparation journey.

I am linking here my own last day revision notes as well that I prepared during my certification preparation.

Practice tests

Here is a list of practice tests you can take online before going in for an actual exam. It will test the understanding of your topic and concretes your decision for exam booking.

That’s all I have to share. All the best!

How to create atomic counter in AWS DynamoDB with AWS CLI

Leave a reply

A step by step procedure to create and update atomic counter in AWS DynamoDB table.

Creating counter in DDB!

First of all, we will see what is atomic counter and why do we need it. We will also check why DybamoDB is chosen in this use case.

What is atomic counter?

Often it would help if you tracked some numerical like website visits, which are incremental in nature. Such counters need to be stored in a centralized place, and the update should be atomic. Atomic means one request executes without interfering with another request. i.e. concurrent updates do not clash with each other and so no data lost in the process. Since everyone is leveraging temporary infrastructure like EC2 getting replaced by Auto Scaling groups or containers, storing such counter locally is not a good idea. So to get central storage, DDB is the best choice since it’s an ultra-fast, single-digit milliseconds latency, NoSQL database. Perfect for the atomic operation of scaling infra/traffic.

Now let’s get into the process of creating this counter and updating it.

Creating DynamoDB table for the counter

  1. Login to DDB console
  2. Click on Create table
  3. Enter Table Name
  4. Enter Primary Key (Partition Key)
  5. We don’t need a sort key here since our table will carry only one counter value. Keep the rest of the settings default and click Create
Create DynamoDB table

You can use below Cloudformation resource block to create a DDB table:

Type: AWS::DynamoDB::Table
    Properties:
      TableName: kerneltalks-counter
      AttributeDefinitions: 
        - AttributeName: ID
          AttributeType: S
      KeySchema: 
        - AttributeName: ID
          KeyType: HASH

Make sure you change the TableName to your choice. You can also explorer other properties supported by DDB in CloudFormation.

Preparing DynamoDB table for counter updates

Now, you will be presented with newly created table details.

Goto Items tab and click Create item

Create DynamoDB Item

Add attributes in item as mentioned below :

Click on Save. Now your DDB table is ready for updating the counter

DDB table creation can be done via AWS CLI as well using the below command:

$ aws dynamodb put-item --table-name kerneltalks-counter --item '{"ID": { "S": "Counter" }, "TotalCount": { "N": "0" }}'

Updating counter in DDB table

Now you can use below AWS CLI command below to update the counter! Every time you run the command, it will update the counter by 1. You can code it in your application at the appropriate place to run this command/API call to update the counter.

$ aws dynamodb update-item --table-name kerneltalks-counter --key '{"ID": { "S": "Counter" }}' --update-expression "SET TotalCount = TotalCount + :incr" --expression-attribute-values '{":incr":{"N":"1"}}' --return-values UPDATED_NEW
{
    "Attributes": {
        "TotalCount": {
            "N": "1"
        }
    }
}

The command should return with the updated attribute in JSON format. You can format it to text for easy usability in code.

$ aws dynamodb update-item --table-name kerneltalks-counter --key '{"ID": { "S": "Counter" }}' --update-expression "SET TotalCount = TotalCount + :incr" --expression-attribute-values '{":incr":{"N":"1"}}' --return-values UPDATED_NEW --output text
TOTALCOUNT      2

The same can be verified in the console.

Counter in DDB.

That’s all! Now you have a counter in DDB which can be updated from different sources using API calls.

How to add Capacity Providers in the existing ECS Cluster?

Leave a reply

A quick post on how to add Capacity Providers in ECS clusters.

In our last article, we walked you through the basics of Capacity Providers. In this article, let’s create them to test their functionalities. You can create Capacity Providers and add them to the running ECS cluster without touching anything in the cluster.

Creating EC2 type Capacity Providers

Steps:

  1. Make sure you are running EC2 backed ECS Cluster.
  2. Create new ASG using same launch template/configuration as existing ASG
  3. Create Capacity Provider
  4. Update cluster with newly created Capacity Provider.

For creating EC2 type Capacity Providers, you should use the new ASG. AWS does not recommend re-using the same ASG which is configured in ECS Cluster. You can use the same Launch template or Launch configuration and create a new ASG to be used by Capacity Providers. It makes it easy for Capacity Providers to keep track of EC2 instances instantiated by them.

While creating new ASG make sure –

  • Desired and minimum capacity is set to zero.
  • Maximum capacity is set to a non-zero number.
  • Enable instance scale-in protection. So that scale in action does not terminate EC2s running tasks.

Once ASG is ready, head back to the ECS cluster dashboard and click on Cluster name. On a cluster detail page, you should see a tab named Capacity Providers. Click on it and then click on Create button.

Creating Capacity Providers!

You should be presented with the Capacity Provider configuration page. Here you can specify –

  • Capacity provider name: Identifier
  • Auto Scaling group: Select newly created ASG from the dropdown.
  • Managed scaling: Enabled so that capacity providers can manage both scales in and scale-out.
  • Target capacity %: It defines how much capacity you want to be used for the task’s compute requirements. I chose 100% that means I don’t want the capacity providers to maintain any spare capacity for upcoming tasks. For example, if you specify 70%, then the capacity provider maintains 30% capacity spare so that any new upcoming tasks can directly be placed here and don’t need to wait in a PROVISIONING state for long. For optimum billing experience, 100% should be good enough.
  • Managed termination protection: Enabled. It prevents ASG from terminating any instance running at least one task on it during scale-in action. Your ASG should have instance scale-in protection enabled for it.
EC2 capacity provider settings

Click on Create button to create your capacity provider. You should be able to see the capacity provider on the cluster details page now.

Capacity provider information

Now, it’s time to add this capacity provider to the ECS cluster.

You can create multiple capacity providers with different ASGs so that you can leverage different instance types. And they all should be listed here as well. For the sake of this article, I am moving ahead with a single capacity provider.

Click on the Update cluster button on the Cluster details page. On the update page, capacity providers can be added/removed to the cluster.

Adding capacity provider to ECS cluster

Select a capacity provider from the dropdown and click on the Update button. If you have multiple capacity providers, then add them by clicking Add, another provider.

When adding more than one capacity provider, you need to specify Base and Weight values for each capacity provider.

You should see the above message confirming capacity provider addition is successful!

Creating FARGATE capacity providers

Well, you cant create them technically! FARGATE capacity providers are readily available in FARGATE clusters. You need to associate them with the cluster.

When you create an ECS cluster backed by FARGATE, you can see FARGATE and FARGE_SPOT types of capacity providers already available in the cluster.

FARGATE capacity providers

You can add them to the cluster by the same Update cluster process explained above.

Adding FARGATE capacity provider to cluster
Adding FARGATE capacity providers to existing cluster

If you have an existing FARGATE ECS Cluster, you need to use AWS CLI or API calls to update it with the FARGATE capacity providers.

You need to use the put-cluster-capacity-providers switch—more details on AWS documentation here. Since I don’t have any existing FARGATE cluster, I can not demonstrate it here.

Now, your cluster is updated with capacity providers and ready to run tasks on a complete auto-scaled ECS cluster!

Amazon ECS Capacity Providers Overview

Leave a reply

A quick rundown at Amazon ECS Capacity Providers

ECS Capacity Providers basics!

We have seen the Amazon ECS service in detail under a couple of previous articles. In this article, we will be discussing the Capacity Provider feature of the ECS service.

What are the Capacity Providers?

As you must be aware, Amazon ECS Clusters leverage EC2 or Fargate compute in the backend to run the container tasks. Before CPs, ECS was conceptualized as infrastructure first! That means you need to have infra ready before you can run the tasks. If you see the ECS’s functioning, you have to create ASG, run the desired count of instances in ASG, and then run the tasks in ECS.

ECS as infrastructure first!

With the concept of Application first, Capacity Providers came into existence. CPs are managers of infrastructure. Basically, they are providing and managing the compute capacity for running tasks in the cluster. So, you need not run instances before running tasks. You can run the tasks even if there is no infra provisioned in the cluster and when CPs detects tasks are awaiting compute to run, they will provision compute (EC2/Fargate). They work in both ways: scale out and scale in.

CPs responds to the service/application’s requirement and mange compute capacity automatically.

Types of Capacity Providers

There are two types of capacity providers offered –

  1. For ECS using EC2:
    • A capacity provider will be a logical entity grouping ASG of EC2 and setting for managed scaling, termination protection.
    • It is recommended to create a new ASG to use in this capacity provider. You may use the same launch config/template from the existing ECS ASG.
  2. For ECS using FARGATE:
    1. Amazon offers FARGATE and FARGATE_SPOT capacity providers.
    2. There are reserved and can not be created or deleted. You can directly associate them with your cluster.

Why should I move from Launch type to Capacity Providers?

All this capacity provider hype leaves many of us to the very question. Why? So let me jot down few reasons why you should consider moving to capacity providers than using launch types.

  1. Out of the box autoscaling for both scale-in and scale-out actions. So, you don’t have to engineer it using the CloudWatch matrix!
  2. You can start a cluster without provisioning any instances beforehand (desired capacity set to 0 in ASG)
  3. Capacity will be made available to tasks when you run them by scaling out ASG. This makes sure AWS bills are light on your pocket.
  4. Scale-out compute when your application takes the load and automatically scales in when load flattens. Peace of mind from management and bills!
  5. You can make use of FARGATE_SPOT!
  6. You can migrate existing ECS services to Capacity Providers without any downtime.

Downsides of Capacity Providers

There are few shorts in the capacity providers as well for now. They may improve in the future. A quick list as below –

  1. CAP is not supported if you are using Classic Load Balancers for services in the ECS cluster.
  2. CAP is not supported for blue/green deployment types for services.
  3. The scale in duration is set to 15 minutes, and you can not customize it.
  4. This leads to another issue: you should have EC2 idle for 15 minutes straight to get it nominated in scale-in action. Hence, scale in action is not aggressive and might not even happen in a crowded/spiking environment.
  5. It also does not for fragmentation of tasks, i.e., grouping them in underutilized instances, which is another reason scale-in will not be aggressive. You can try binpack placement strategy, though, to mitigate this.
  6. Once you update the service to use capacity providers, you can not update it to use EC2/previous launch type. You have to delete and re-create the service for that (includes downtime).
  7. When using the capacity provider strategy, a maximum of 6 capacity providers can be provided.

That’s all for starters! We will see Capacity Providers in action in upcoming articles.

Creating first AWS Lambda function

Leave a reply

A quick article explaining all available options for creating a new AWS Lambda function and how to run it.

Learning AWS Lambda!

AWS Lambda is a compute service offered by AWS. It is treated as a serverless compute since the end-user did not have to worry about providing and maintaining the back-end infrastructure. Without further delay, let’s move ahead with creating the first AWS Lambda function.

  • Login to AWS Lambda console.
  • Click on the Create function button.
  • You should see wizard as below –
Lambda function wizard

There are 4 ways you can choose to create AWS Lambda function –

  1. Author from scratch: Select the programming language of your choice and start writing the function code
  2. Use a blueprint: Use readymade function pre-written for you by AWS.
  3. Container image: Use a docker container image to deploy function
  4. Browse serverless app repository: Use complete application pre-written in Lambda.
  • Basic information
    • Function name: For identification purpose
    • Runtime: Function’s coding lanuage. Select a supported language from the dropdown.
    • Permissions: Arrange the Lambda function IAM role to access AWS resources on your behalf.
AWS Lambda roles

You can either use an existing role or let the wizard create a default permission role. If your code requires special permissions, it’s recommended to create a new role using policy templates to ensure that function has sufficient privileges to access AWS resources.

Advanced settings
  • Advanced settings
    • Code signing: To ensure the integrity of the code and its source.
    • Network: If function expected to have network access, then select VPC here and hence in turns select subnet, security group which will be applicable to function when it runs.

Click the Create function button.

Function created!

AWS Lambda function should be created. Click on the Latest alias to edit your code and test it.

Edit lambda code

In latest alias, there are 5 tabs you can navigate.

  1. Overview: Overall details.
  2. Code: Edit code and code-related configurations.
  3. Test: Invoke test event for the Lambda function
  4. Monitor: Monitoring aspects in context to Cloudwatch
  5. Latest configuration: Various Lambda function related configurations.

Lets have a look at all these tabs one by one.

  • Overview
    • General configuration: Shows description, last modified, and Function ARN.
    • Function visualization: Visual representation of a function.
  • Code
    • Code Source: In browser code editor. You can upload the code in a zip file or reference it from the S3 location.
    • Runtime settings: Edit the code language.
    • Code signing details: Edit code signing for code integrity.
    • Layers: Edit or add layers to function.
  • Test
    • Test event: Create, format, or invoke test event.
  • Monitor
    • View CloudWatch metrics or CloudWatch logs insights.
  • Latest configuration
    • This is the biggest section of all. It has 5 subsections
      1. General
      2. Environment
      3. Triggers
      4. Permissions
      5. Destinations

Lets go through each section one by one:

  • General
    • Basic settings: Edit below configurations
      • Description
      • Memory (MB): Memory allocated to function. CPU will be allocated in proportion to memory.
      • Timeout: Max execution time. Limit: 15 mins.
      • Execution role: IAM role being used by the function.
    • Asynchronous invocations:
      • Maximum age of event: To keep the unprocessed event in the queue.
      • Retry attempt: Maximum number of retries after the function returns an error.
      • Dead letter queue: SQS queue for sending unprocessed events for asynchronous invocations.
    • Monitoring tools
      • Amazon CloudWatch: By default enabled for AWS Lambda
      • AWS X-ray: For analysis and debug purposes.
      • CloudWatch Lambda insights: Enhanced monitoring.
    • VPC
      • Edit VPC, subnet, security group for Lambda function if function required networking.
    • State machines
      • Step functions to orchestrate Lambda function.
    • Database proxy
      • Manage database connections if the function requires DB connectivity.
    • Filesystem
      • Make EFS available to Function.
  • Environment
    • Environment variables: Set of key-value pairs to use as environment variables in function during execution.
  • Triggers
    • Add triggers to invoke the lambda function.
  • Permissions
    • IAM permissions via role or resource-based policies
  • Destinations
    • Configure destination for invocation records once function executes.

Since its a test function, I move ahead with all the defaults. Once everything is configured you are ready to test the fuction.

Click on the Test tab and then click the Invoke button to invoke the Lambda function test event.

Invoking test event

Test invokation results presented on same screen. The important details to look for are –

  • Returned results by function
  • Duration: Actual execution time
  • Billed Duration: Billed execution time. Rounded to nearest 1ms.
  • Max memory used

The log output at the end can also be viewed in CloudWatch log group.

CloudWatch log group for Lambda

The bill amount is calculated using the amount of memory allocated and billed duration.

So that’s how you configure and execute simple basic Lambda functions for understanding the foundation of AWS Lambda!

Replication in Amazon S3

Leave a reply

An article to overview SRR and CRR in Amazon S3. Why, when, and how to use it.

Replication in amazon S3.

Amazon S3 is a Simple Storage Service offered by AWS. It’s object storage with virtually unlimited in size. S3 also offered the feature of replication to replicate objects from one bucket to another bucket. The bucket is a placeholder for arranging objects in S3. It can be visualized as a root folder in the file structure.

What is replication in Amazon S3?

Replication is a process of copying objects and their metadata from a source bucket to a destination bucket in an asynchronous way in the same or different region.

Why S3 replication is needed?

There are couple of use cases where S3 replication can prove helpful –

  1. Compliance: Some customers need to keep a copy of data in a particular territory for compliance purposes. In such cases, S3 replication can be set to copy data into the destined region.
  2. Data copy under different regions and accounts: S3 data replication enables users to copy data in different regions and under different accounts.
  3. Latency requirement: Keeping a copy in a region close to consumers helps to achieve low latency in accessing data stored in S3.
  4. Objects aggregation: Data from different buckets can be aggregated into a single bucket using replication.
  5. Storage class requirement: S3 replication can directly put objects in Glacier 9diff storage class) in the destination bucket.

Types of S3 replications

  • Same-Region Replication (SRR)
    • Source and destination buckets lie within the same region.
    • Used in case data compliance requires the same copy of data to be maintained in different accounts.
    • Log aggregation can be achieved using SRR by setting up SRR from different log buckets to the same destination bucket.
  • Cross-Region Replication (CRR)
    • Source and destination bucket lies in different regions.
    • Used in case data compliance requires the same copy of data to be maintained in different regions.
    • Useful in latency requirements.

Requirements and limitations of Amazon S3 replication

  • Versioning must be enabled at source and destination buckets.
  • Existing files are not replicated. Only new uploaded objects (after enabling replication) are replicated.
  • Delete markers created by lifecycle rules are not replicated.
  • You can’t replicate objects which are encrypted using SSE-C.
  • Can set up across accounts.
  • It’s one-way replication, not bidirectional. It means source bucket to destination and not vice versa.
  • Lifecycle policy events are not replicated.
  • Amazon S3 should have access to read/write on both buckets by assuming your identity.
  • The destination bucket can not be configured with requestor pays mode.

How to configure S3 replication?

Let’s dive into the practical aspects of it. We will be configuring replication between 2 buckets hosted in different regions under the same account for this exercise. Read here how to create an S3 bucket. Below are 2 buckets in consideration –

Source and destination bucket
  • Log in to the S3 console.
  • Click on Buckets in the left navigation panel.
  • On the buckets page, click on the bucket name you want to create replication (source bucket).
Creating replication rule for source bucket
  • Click on the Management tab under bucket details.
  • And scroll down to the Replication rules section.
  • Click on the Create replication rule button.

Make sure you have versioning enabled on both source and destination buckets, or else you will see a warning like the one below –

Bucker versioning warning

In replication rule configuration wizard –

Replication rule wizard
  • Replication rule configuration
    • Replication rule name: Name
    • Status: Enable/disable the rule on creation
    • Priority: In the case of multiple replication rules, the object follows priority.
Source and destination configurations
  • Source bucket
    • Source bucket name: Prepopulated.
    • Source region: Region of source bucket. Prepopulated.
    • Choose a rule scope: Replication can be limited to selected objects by using a prefix filter. Or apply it for all objects in the bucket.
  • Destination
    • Bucket name: Choose bucket from the same account or a different account here as a destination.
    • Destination Region: Region of destination bucket.
Few other configs
  • IAM role
    • Select the IAM role to be used by S3 for performing the copy operation. Make sure you have carefully added proper permissions to it. I choose to create new for this exercise.
  • Encryption
    • If data is encrypted using AWS KMS, it can be replicated.
  • Destination storage class
    • Change the storage class of replicated objects or keep it the same as the source.
Additional options
  • Additional replication options
    • RTC: Premium service with guaranteed 15 mins ETA for replication of new objects
    • Replication metrics and notifications: Cloudwatch for replication!
    • Delete marker replication: Enable to replicate S3 delete operation created delete markers. Remember lifecycle rule created delete markers can not be replicated.

Click the Save button. Replication rule should be created and you should be presented with rule details like below –

Replication rule details

You can even see the IAM role created by AWS while creating a rule since we opt to create one.

Now, it’s time for the test. I had one file already in the source bucket, and it’s not replicated after creating the rule since it’s only applied to new objects uploaded to the source bucket after rule creation.

Source bucket objects

File1.text was existing before rule creation and File2.txt uploaded after rule creation. So the destination bucket should be having File2.txt replicated from the source bucket.

Destination bucket objects

And its there! So replication rule worked perfectly!

How to check if object is replicated?

To verify if the object is replicated or not. Click on the object and check its replication status.

Replication status as REPLICA

For replicated object replication status should show it as REPLICA. Whereas, for source object i.e., who uploaded to source object and then got replicated status is COMPLETED

Replication status as COMPLETED

And it would be blank for non-replicated objects. Since File1.txt was uploaded before replication rule creation, the rule was not applied to it.

No replication status

And hence we see its replication status as - i.e., no replication rules applied to this object.