On the left navigation panel, click on the Users link.
On the right-hand side Users page click on the Add user button.
Add user screen should come up –
IAM Add user wizard.
Fill in details –
User name: User id
Access type:
Programmatic access: No access to AWS console. Use of keys for authentication. Console access can be enabled later.
AWS management console access: Access to AWS console Use of userid/password authentication. Programmatic access can be enabled later.
Click on the Next: Permissions button.
Setting user permissions
Set permissions
Add user to group
An efficient way to manage user permissions by making them members of the group. Apply policies/permissions to the group!
Copy permission from an existing user
If you want to have user same permission as another existing user in the same account.
Attach existing policies directly
Attach permission policy to user either AWS managed policy from the list given or by writing your own policy by clicking Create policy button.
For this exercise, I will choose an easy way by marking AWS managed Administrator access policy to users.
Set permission boundary
Define the maximum permissions this user can have. User’s permissions can not breach the boundary defined here. Again you can create your own or use AWS managed policy here.
Click on Next: Tags button
User tags
Add user tags for identification purpose and click on Next: Review button.
Review
Review all the configurations and click on Create user button.
IAM user created
User should be created and you should be seeing above screen.
From this screen, you can copy or download the keys required for AWS programmatic access.
Click on the Download .csv button to download the key pair. Or click on Show link under the Secret access key and then copy/save both the Access key ID and Secret access key. Once you navigate away from this page, you will not be able to retrieve the secret access key from anywhere. You need to recreate the pair for this user then.
You have successfully created an IAM user with programmatic access to AWS. You have access keys with you!
How to re-create IAM secret access keys?
As I mentioned above, if you lose the secret access key there is no way to retrieve it unless you saved it somewhere. But in such unfortunate incidents, you can re-create them using an AWS root account.
On the left navigation panel, click on the Users link.
On the right-hand side users page, click on the user name whose keys needs to be regenerated
On users, summary page click on the Security credentials tab
Access keys in IAM console
It’s good practice to keep only one key pair active at a time so click on the Make inactive link for the existing key. You can keep it unless there is such a requirement. Click the Deactivate button on the pop-up. You can even delete this key 9if it does not have any dependency) by clicking a small x next to it.
Click on Create access key button to generate new key pair.
Create new access keys
New key pair will be generated and you will have a chance to download/copy save the secret access key again!
The step by step procedure to create an Amazon SQS queue and later test it with Amazon SNS.
Testing Amazon SQS queue!
What is an Amazon SQS?
Amazon SQS (Simple Queue Service) is the very first AWS service. It’s a message queueing service that helps with decoupling the architecture. It’s widely used in microservices designs. It’s a poll based service. So basically, it receives messages from some source, maintains messages in the queue for consumers to poll for it, consume it, and delete it after processing.
Using SQS, one need not keep consumer compute/infrastructure up and running all the time. SQS takes care of buffering and queueing messages, and consumers can poll them once a while, processes them, and exists/shuts them till the next poll. It also holds messages if consumers are not available or busy and scales accordingly, so you don’t lose any messages.
Amazon SQS offers two types on queue and their differences are as below –
Standard Queue
FIFO Queue
Default SQS queue
First in first out queue
Messages order is not preserved
Preserved messages order which is first in first out
message can be processed twice
Each messages processed exactly once
It can scale indefinitely
It can processes 3000 messages per seconds only
Best fit for scaling and high throughput requirement
Best fit where order is important and duplicates cant be tolerated
Click on the Create queue button on the introduction page.
Create a queue wizard should open up –
Create queue screen
Details
Choose the type of queue
Standard
FIFO
Name: Enter the queue name. For the FIFO queue, the name should have a .fifo suffix.
SQS queue configuration
Configuration
Visibility timeout: Time for which message won’t be visible to other consumers when picked up by consumer for processing. Range 0 secs to 12 hours
Messages retention period: Duration for which messages live in the queue if not deleted by any consumers. Range 1 min to 14 days
Delivery delay: Time for which message won’t be visible to consumers to pick once the queue receives it. Range 0 sec to 15 min. 0 means the message will be available immediately for processing once received by the queue.
Maximum message size: Message size. Range 1KB to 256KB
Receive message wait time: Max time for polling to wait for messages to be available. 0 means short polling (increases empty responses), and 20 means long polling—range 0 to 20 secs.
SQS queue access policy
Access policy
Basic
Simple bullet selections for access to send and receive messages to/from the queue. JSON will be generated accordingly.
Advanced
Use your own JSON to define policies
SQS optional settings
Three optional configurations for SQS queue.
Encryption: Server-side encryption offered by AWS.
Dead-letter queue: Home for undeliverable messages.
Tags: For identification purposes.
Click on Create queue button.
SQS should create a queue and present you with queue details.
SQS queue created!
You can click on the Send and receive messages button to manually send, receive, and delete messages in this queue.
Send message to SQS queue manually
Enter the message body and click on the Send message button. The message will be sent to the SQS queue.
If you scroll down on the same screen, you can click the Poll for messages button and receive the messages from the SQS queue you sent in the earlier step.
Receive message from SQS queue manually
Click on message ID to view message details. Under the message body tab, you can see the message body.
Message details
Once messages are processed (i.e., read in our case), you can delete messages from the queue by choosing a message and clicking the Delete button.
Sending messages to SQS queue from SNS
We walked through sending and receiving messages manually to and from the SQS queue. Now we will do it using SNS. SNS is a Simple notification Service offered by AWS. SNS topics are the channels where you publish messages for recipients. It’s a push-based service.
I already created an SNS topic named sns-topic-001 to which the SQS queue will be subscribed.
On an SQS queue details page click on the Subscribe to Amazon SNS topic button.
Subscribing SQS to SNS Topic
You should be able to select the existing SNS topic from the drop-down on the next page.
In the Topic details screen, click on the Publish message button.
Make sure you see the SQS queue under Subscriptions.
SNS Topic details
Fill in message details like Subject and body and click the Publish message button.
Publish message to SNS Topic
Now the message is published to the SNS topic. We have an SQS queue that is subscribed to SNS Topic. That means this published message should appear in the SQS queue.
Lets go back to SQS queue and check for it.
SNs message received in SQS
You should see 1 message available to receive in SQS. And when you click the Poll for messages button, you will see that message in the messages details section below.
If you check message body you will see all SNS formatted message details.
Message details
In this way, you can check the SQS queue’s functionality by sending and receiving messages manually or using SNS Topic. You can even receive messages automatically using the Lambda function and then publish it to another SNS Topic, which can mail it to you for testing! We might cover this in another article. But this is it for now!
A step by step procedure to create an application load balancer for a web application.
Application Load Balancer creation in AWS!
This article will walk you through the steps to create an application load balancer and then testing. ELB can be used in Amazon ECS as well, but for this exercise, we will be using the below architecture, which is running webservers on EC2, not in containers. Its a subset of our custom VPC –
ALB design
We have 2 EC2 instances running Apache webserver in 2 different public subnets. Application Load Balancer will receive traffic from the internet and forward it to the back-end EC2 instances.
Gateway load balancer: For load balancing virtual appliances traffic over GENEVE.
Classic load balancer: Old ELB tech.
Click on the Create button under the Application load balancer to proceed. Load balancer configuration wizard should open up.
ALB basic config
Fill in details –
Name: Name for ALB.
Scheme: Choose internet-facing since we are configuring the web load balancer.
IP address type: Select the addressing type.
Listeners: Choose HTTP with port 80. If your application is on HTTPS, then select accordingly.
ALB AZ config
VPC: Select VPC under which ALB will be deployed.
Availability zones: Select minimum 2 for HA. If you are creating internet-facing ALB, then subnets should have a route to the internet gateway, i.e., public subnets. You can select only one subnet per AZ.
AWS Global accelerator: For performance. This is part of integrated service and can be modified later as well.
Tags: Tagging.
Click on the Next: Configure Security Settings button at the end.
Since HTTP was selected in the basic configuration, a security notice should appear.
HTTP notice
Click again on the Next: Configure Security Settings button in the end to proceed.
ALB SG
Create new or select an existing security group for ALB. We are selecting here existing SG, which allows HTTP traffic.
Click on Next: Configure Routing button.
ALB routing
In the routing section, we are configuring the destination for ALB. Here, ALB will come to know where it needs to direct traffic once it receives the traffic. As per our design, we are going to direct traffic to 2 EC2 instances.
Target group
Target Group: Create new or use existing. It’s a collection of resources acting as targets for ALB.
Name: for identification.
Target type: In our case, its instance.
Protocol: HTTP or HTTPs
Port: Depends on your web application listening port. I am using the default web server listening on port 80
Protocol version: Again depends on the web application.
Health checks
Protocol: To be used by ALB to perform health checks on the target type.
Path: ALB will reach out to this path using the mentioned protocol to determine health or target.
Advanced health check settings
Port: Communication port for a health check.
Healthy threshold: Number of consecutive successful health check before marking any unhealthy target as healthy. Range 2-10.
Unhealthy threshold: Number of consecutive failed health checks to mark a target as unhealthy. Range 2-10.
Timeout: If no response is received within this timeframe, mark the health check as failed. Range (2-120 secs)
Interval: Time between health checks (5-300 secs)
Success codes: HTTP code to be received back for marking health check as a success.
Click on Next: Register Targets button.
Registering targets
As you can see, we have 2 EC2 instances running in different zones that are available to register as a target for ALB. Meanwhile, also verify that both instances are serving the webpage properly.
Verifying webservers
Both our webservers are serving different webpages (to test on ALB later). We verified it by using the public Ip of the EC2 instances.
Now, select instances serving web traffic and click on the Add to registered button.
Target registered
Both targets should be registered and list under the registered target list. Verify and then click the Next: Review button.
ALB config review
Review all configurations and click on Create button.
ALB creation completed!
You should be seeing success message like one above.
Click the Close button, and it will take you to the load balancers page. In here, newly created application load balancers should be listed, and mostly it’s in provisioning state. After a couple of minutes, it should go into an Active state once at least one target passes the health check.
ALB details
Once ALB is active, grab the DNS name from the ALB details screen and load it in the browser. It should populate the webpages from either of the EC2 targets.
Testing ALB
The above small GIF shows the same ALB DNS loads web page from different backend targets. I created distinct web pages to identify the difference and functionality of ALB. In the real world, it should be serving the same page as any of the backend targets.
That’s it! We created an Application load balancer that serves the webpage from different backend servers!
A quick rundown on how to create an Amazon ECR repository and push container image to it.
ECR HowTos!
What is Amazon ECR?
Amazon ECR, i.e., Elastic Container Registry, is a fully managed container image registry service provided by AWS. It is integrated with Amazon ECS so that developers can have a fully managed container platform by AWS. You can visualize it as your own docker hub.
Browse through our Amazon ECS related articles here.
How to create ECR repository?
Amazon ECR has its own home under Amazon ECS dashboard.
Click on Repositories in the left navigation panel
Click on the Create repository button on the repository page.
You should be presented with the below screen –
ECR repository create screen
Different configurations are –
Repository name: It will follow the below format
account-id.dkr.ecr.region.amazonaws.com/name
Tag immutability: Prevents overwriting images with the same tags in the subsequent push.
Image scan settings: Enable it to scan images as soon as they are pushed to ECR for vulnerabilities.
Encryption settings: Use KMS or let ECR use default encryption for images once pushed to ECR.
Click Create repository button.
A repository should be created, and the ECR dashboard should enlist the newly created repository. Please select it and click on the View push commands button to get the authentication token, login, and push commands. These are ready-made commands for you to ease your tasks.
Getting ECR commands from AWS console
ECR will provide you with 4 commands pre-populated with the correct repository name. The explanation is given with each command is pretty self-explanatory.
ECR push commands
How to push container image to Amazon ECR?
Now, we will use these commands to push the test container image to Amazon ECR. I will be using the Amazon Linux EC2 instance for this exercise. Make sure you have docker and aws-cli installed, configured, and running fine on the EC2 instance. Also, make sure EC2 is having at least an AmazonEC2ContainerRegistryPowerUser role attached so that ECR push works fine if you are using EC2 roles for authentication.
Step 1.
Get the authentication token and log in using the docker command. Its taken care of by the first command we saw in the previous screenshot. It has a two-part. First, aws command gets the authentication details and passes them to the second docker command for login.
[root@ip-10-0-0-226 ~]# aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 3xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Step 2.
Tag container image. I am assuming you already have an image created using Dockerfile. For testing, I pulled the Nginx image from DockerHub and tagged it.
[root@ip-10-0-0-226 ~]# docker tag nginx:latest 3xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/webserver:latest
Verify image tags.
[root@ip-10-0-0-226 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
3xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/webserver latest daee903b4e43 9 hours ago 133MB
nginx latest daee903b4e43 9 hours ago 133MB
If you come back to the AWS console and check under the repository, you should be seeing this latest pushed image there.
As you can see, since the Image scan was enabled, ECR scanned the image for vulnerabilities as soon as it was pushed. And added a report in the same view. Click on the image tag for checking image details.
Image details
If you don’t enable the Image scan setting, then the manual scan can be done by clicking the Scan button on the Images details screen.
Another point to note here is ECR showing image size as 53.61MB, whereas it was reported as 133MB on EC2 command outputs. This is because docker compresses the image layers when pushing the image to the repository. Note about the same can be found here in AWS documentation.
How to pull container image from ECR?
It’s a simple docker pull command. Provided you have EC2 instance and docker daemon properly setup for authentication (as mentioned above).
[root@ip-10-0-0-226 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
[root@ip-10-0-0-226 ~]# docker pull 3xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/webserver
Using default tag: latest
latest: Pulling from webserver
852e50cd189d: Pull complete
a29b129f4109: Pull complete
b3ddf1fa5595: Pull complete
c5df295936d3: Pull complete
232bf38931fc: Pull complete
Digest: sha256:bb84ff0786cd1dbde780d84f6bf76bfdef36fe8ffa658f7f5c48e39363b4d500
Status: Downloaded newer image for 3xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/webserver:latest
3xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/webserver:latest
[root@ip-10-0-0-226 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
3xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/webserver latest daee903b4e43 10 hours ago 133MB
Assorted list of resources to help you with your work in AWS!
Swiss knife for AWS tasks!
In this post, I will quickly run down the assorted list of different software, tools, or online resources that will help you with your AWS journey. So without further delay, let’s jump into it.
AWS Simple Monthly Calculator: Estimate the monthly running cost of your infra on AWS. It is going to be deprecated soon and will be replaced by the former one.
AWS Architecture Icons: List of supported drawing and diagramming tools and download links for Icons assets and toolkits. It’s very much helpful in designing presentable diagrams with updated, latest AWS icons. Every architecture’s must-have resource!
Draw.io: Create architecture diagrams with draw.io
Visual subnet calculator: Easy way to slice up your CIDR block into the required number of subnets.
SQLectron: An open-source lightweight utility for connecting to RDS.
Infrastructure coding
Visual Studio Code: Most favoured and loved software for all coding you do on your computer. Supports almost all languages and clouds. Allows you to connect to your cloud and run your code from the software window itself.
cfn-lint plugin: Cloudformation Linter plugin for Microsoft Visual Studio Code.
InfraCost: cloud cost estimates on the terraform pull requests!
The list can grow on and on. I just collected a few of them here to start with. Let me know your additions in the comments down below!
A quick post on advanced container configurations in Amazon ECS.
ECS container advanced configurations.
Container definitions are part of Task Definitions in Amazon ECS. It’s the configuration where you can customize the container’s infrastructure aspects. In this article, we will walk you through advanced configurations of containers.
In our last article about Task Definitions, we walked you through standard container configurations. Now, we will check all the parameters available in Advanced container definitions.
Command: It will be run within containers to determine if the container is healthy. Since I am spinning up a webserver I used the curl command. It depends on what kind of container is and how you can determine its health.
Interval: Duration of two consecutive health checks. (Range: 5-300, default: 30)
Timeout: Duration to wait to check health check once it’s executed. (Range: 2-60, default: 5)
Start period: Grace period for the container to recover before it can be marked unhealthy after max health check retries. (Range: 0-300)
Retries: Max number of failed health checks to mark containers as unhealthy and terminate. (Range: 1-10, default: 3)
Container environment
Environment
CPU Units: 1 CPU core of ECS instances = 1024 CPU units. These are units of CPUs allocated for the container.
GPUs: Number of GPU units reserved for containers. 1 GPU = 1 unit. ECS instances must be GPU supported.
Essential: If this is checked, the task will be marked as failed on the failing of this container. If unchecked, the task will continue to run even if this container is failed.
Environment Files: Source container environments saved in S3.
Environment variables: Key-value pairs of variables to be used by the container.
Container timeout and network settings
Container timeouts
Start timeout: Duration to wait for the container to resolve all dependencies to become fully operational
Stop timeout: Duration to wait for the container to exit normally or kill it after this timeout.
Network settings
Disable networking: No communication outside of the container. The container will be assigned with a loopback address.
Links: To communicate with other containers.
Hostname: Hostname for the container.
DNS servers: To be used by the container
DNS search domains: To be used by containers.
Extra hosts: Any entry not resolvable by the above two options can be added here.
Container storage and logging
Storage and logging
Read only root file system: RO for root FS in the container. If mounted it will be able to write on data volumes.
Mount points: Data volumes to be mounted inside the container
Volumes from: Data volumes from other containers
Log configuration: Loggings container logs in AWS CloudWatch
Rest of the configs
Security
Privileged: Container gets elevated privileges on container instances
User: To be used inside the container
Docker security options: SELinux and AppArmor security settings to be passed to the container
Resource Limits
Ulimits: Those are Linux kernel ulimit values.
CORE: Limites the core file size (KB)
CPU: Max CPU time (MIN)
FSIZE: Maximum filesize (KB)
LOCKS: Max file locks user can hold
MEMLOCK: Max locked-in-memory space (KB)
MSGQUEUE: Max memory used by POSIX messages queue (bytes)
Docker labels
Key value pairs: Tags
Once I run the Task containing the above container definition, it ran successfully. All the custom configurations can be seen in the AWS console. under Task details.
Container details under running task in Amazon ECS
Alternatively, we can log into the ECS instance and then a container to verify stuff.
Checking container on ECS instance
First verify if the container is running.
[ec2-user@ip-10-0-0-122 ~]$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2c2267e6ce85 nginx:latest "/docker-entrypoint.…" About a minute ago Up About a minute (health: starting) 0.0.0.0:32768->80/tcp ecs-webserver-nginx-8-nginx-d28beae194c4eada5b00
9bb8f8b0b6ea amazon/amazon-ecs-agent:latest "/agent" 2 minutes ago Up 2 minutes (healthy)
Log in to container and verify if custom configurations are applied.
You can see container hostname is set, DNS nameservers are set, extra IP-hostname pair has been added to /etc/hosts, logged in user is the root and working directory is set to /usr/share/nginx/html! Everything is accommodated.
Last thing to verify if the container is sending logs to the CloudWatch service. Click on the link View Logs in CloudWatch under container details on the Tasks page. (can be seen in the above screenshot)
ECS container logs in CloudWatch
And logs are being populated in CloudWatch!
That’s all! All advanced container configuration which one can configure under Amazon ECS Task Definition.
An article about service in Amazon ECS. What is the service? How to configure it? and how to run it?
Services in Amazon ECS
Amazon ECS helps you to spin containers on the cloud. ECS is a complete managed orchestration service offered by AWS. Container instantiation in ECS is taken care of by ECS Tasks. And to manage the ECS Tasks we have Services.
What are Services in Amazon ECS?
Amazon ECS Services enables you to run and maintain the desired number of instances of a task definition on the Amazon ECS cluster. It spins up a new instance of tasks in case existing running tasks die/fail and maintain the desired count.
How to configure Service in Amazon ECS?
I will be using the same Task Definition created in the previous article here to configure in Service.
Load balancer name: Select existing LB from the dropdown. If not then create a new LB from the EC2 console and then select here. You don’t need to register targets while creating a load balancer. It will be managed by the ECS cluster.
Container to load balance: Your container port will be listed which is fetched from Task Definition. Click on Add to loan balancer button
Container load balancing
Production listener port: ALB listening port
Production listener protocol
Target group name: If you haven’t created it with ALB then you can create here or select from the dropdown.
Target group protocol
Target type: ECS instances
Path pattern: For path-based routing. If you want to route traffic depending on different paths then list it here along with the order.
Health check path: for ALB, to determine the health of the target. If this path is reachable then ALB considers the target as healthy.
ECS service discovery.
Lastly, App Mesh as defined in Task Definition and Service discovery for using it with DNS.
Click Next step button.
ECS service auto scaling
On the next screen, you should be able to configure service auto-scaling to handle the high and low demands.
Click Next step button
Review complete configuration once and then click on the Create Service button.
Service will be created and you can click on the View service button and you should be seeing service launching tasks in a couple of minutes.
Running ECS service.
As configured, 2 tasks are launched by service and both are running. Now to verify if service is working fine and containers are serving the purpose, ALB needs to be checked.
I took the ALB DNS name and opened it in the browser. It should go to target groups which are ECS instances and hence to the containers where Apache is running. And then it should display the Apache default page.
ALB test
And it does! Test succesful! ECS service is runnign as expected.
If it does not, then do check if the security group attached to ALB allows incoming HTTP port 80 traffic from the world. Secondly, check the security group of ECS instances that allows HTTP port 80 traffic from ALB’s security group/subnet.
Now, to check if the service maintains the desired count of tasks let’s run a small test. Click on any one task ID and click the Stop button to stop that Task. This should kick in service to start a new task in order to maintain the desired count of 2 Tasks. I killed one task and it did start another task on its own.
Service test
The new task is in pending status and went into RUNNING in a couple of seconds since containers are pretty much fast to insatiate and start serving.
That’s all. We did not cover auto-scaling in this article. That will be something similar to having ASG of EC2 but here for containers!
A step by step guide on how to create task definition in Amazon ECS and how to run tasks in Amazon ECS Cluster
All about Tasks in Amazon ECS
In this article, we will walk you through defining ECS tasks and running them on ECS Cluster. To begin with, let’s understand the basics of ECS tasks.
What is ECS task?
ECS task is responsible to instantiate docker containers in ECS instances or Fargate. Tasks are defined using Task definitions. Each task definition is a collection of parameters like docker image to use, CPU, memory limits, networking mode, etc. When the task is run in the ECS Cluster, it reads Task definitions and accordingly spins up docker containers.
In the left navigation panel, click on Task Definitions
Under the task definitions page, click on the Create new Task Definition button.
Task definitions start with defining the launch type. Choose launch type and click the Next step button.
ECS Task Launch types
ECS offers 2 launch types –
Fargate
Tasks will be launched on infra managed by AWS. (serverless)
Tasks will be billed on resources being used and usage duration.
EC2
TAsks will be launched on ECS instances registered to ECS Cluster
No separate bills. You will be paying for ECS instances as per normal EC2 instance bills.
For this exercise, I am using the ECS launch type since I have an ECS cluster running with 2 ECS instances registered to it.
After clicking the Next step button, the task and container definition screen should appear. Lots of things to be defined on this screen. Let’s go one by one –
Task definitions
Task Definition Name: For identification purpose
Task Role: If containers being used designed to access some AWS services then you can specify the IAM role here which to be used by containers while accessing AWS services.
Network Mode: There are 4 modes available here –
<default> which is bridge mode
Bridge: Traffic forwards between host and container by bridge (kernel-level software)
Host: Container network mapped directly to host network
awsvpc: Each container assigned with ENI (and hence SG too) Hence each container’s networking can behave like EC2’s ENI.
None: No networking for containers. Containers spin up with the loopback IP address assigned.
Task sizing
Task execution IAM role: Needed for pulling container images and sending container logs to Cloudwatch.
Task size: Resource allocation for the task. Limiting resources to be consumed by task container. Should be defined properly when using the FARGATE launch type. For EC2 launch type, this should be calculated depending on ECS instances resources available to the task. Since I am launching a small Apache container on EC2 I left it unfilled.
Container definitions
Container Definitions: Under this section, all container-related settings can be defined. Click on the Add container button and it should take you to the container definitions screen.
Image: Container image. repository-url/image:tag. If you want to use an image from Dockerhub then simply specify image:tag. And for that, you should be having internet access on ECS instances to pull images either via NAT or Internet gateway.
Private repository authentication: If you are using private container repo like ECR use this option.
Memory limits: Its memory reserved or allowed for containers during execution.
Hard limit: Max allowed memory for containers to use.
Soft limit: Memory reserved for the container.
Port mappings: Host port to container port mapping. It’s always advisable to use dynamic host port mapping by defining the host port as zero.
Click Add button to add this image definition in task definition.
More task definitions
Elastic Inference: Allows you to attach low-cost GPU powered acceleration to tasks. The inference is the process of making predictions using the trained model. It requires processing power. If your container is into such stuff it makes sense to use Elastic Inference which can save you up to 75% cost.
Constraint: Helps you to decide the placement of containers on ECS instances. Not applicable for the FARGATE launch type. Define conditions to filter and select ECS instances. Once the constraint is applied and instances are selected for task deployment, further placement strategy (Explained in the Run Task section below) will be applied and finally, tasks will be launched on the final instance.
Service Integration: It’s a facility by AWS to manage your microservices easily. This configures proxy to communicate between microservices for better visibility and HA for services.
Proxy configuration: Should be auto-configured once you select App Mesh and fill out the required details in that section.
Log Router Integration: Enables routing of container logs to other AWS services or APN services for storage and analysis. It will spin up the respective container using AWS provided image.
Volumes: Volumes to be mounted inside containers. It supports 3 types
Bind mount: Mounts file or directory on the host inside the container. More info.
Docker: Managed by docker and creates /var/lib/docker/volumes on the container where volume data resides. Drivers can be selected local or third-party. Can persist on task completion if declared as shared.
EFS: Mount EFS volumes in containers!
Tags: For identification.
Click Create button to create task definition.
Task definition created!
Task definition should be created and it will be versioned as :1. Task definition can be edited using the Create new revision button and it will be versioned as :2 and so on.
How to run Amazon ECS Task?
From same page or going back to Task definitions page, select recently created task definition and click on Actions button.
Run ECS task
Run Task screen should appear where you can provide details on how tasks should be run.
Run Task screen!
Launch type: Fargate or EC2
Task Definition: choose which revision to be used
Cluster: On which cluster task should be run
Number of tasks: How many containers need to spin up. This count is for HA, FT, or for performance.
Task Group: Identification purpose.
VPC and security groups: Available only if the task definition mentions the use of awsvpc networking mode. Defines ENI level networking details for containers.
Task Placement: Supports 5 templates –
AZ Balanced Spread
Deploy containers so that they are evenly spread across AZ
Make use of available ECS instances in each AZ
AZ Balanced BinPack
Deploy containers by filling one host at a time.
Do not start with another host unless the current host capacity is full. Allows maintaining unused hosts.
Balanced it across AZ. So start with one host in each AZ and go on filling it till full capacity then move on to the next host in that AZ.
BinPack
Same as above except AZ balancing
One Task Per Host
Strictly one task per host.
If no free hosts available, the task will fail.
make sure you have enough available hosts for tasks you are running.
Custom
User-defined with a combination of Spread, BinPack or random
Configure the order in which it needs to be evaluated.
Run Task advanced options
Under advanced options, IAM roles can be overridden that are defined in Task Definitions.
Task role and Task Execution role, both the IAM role can be overridden with a new one under this section.
ECS Task tagging
Lastly, tagging settings to be done. Enable ECS managed tags as ECS tag tasks with cluster and service name which is pretty much easy for identification later.
Click on Run Task button.
Running tasks in ECS cluster
The task will be started and in a couple of minutes, you should be seeing them in RUNNING state. In the above screenshot, the Pending tasks count lists 2 EC2 since I captured the screen by refreshing only the Tasks tab below. I did not refresh whole cluster page 🙂
Now, the task is in a RUNNING state that means containers are instantiated on ECS instances and port 80 of container bound to host port.
Click on any single task ID and it should show task details like below –
Finding host port
Under container details, a host port can be obtained. In this case, the 32768 host port is bounded to port 80 of the container. To verify the functionality of the Apache container, the external link needs to be checked.
Since this cluster is running with ECS instances placed in a private subnet we need to use bastion host to open this external link. Also, since these are private instances you can see the external link is a private IP address, not the public one.
I curled to the external link from bastion host and it worked!
If it’s not working for you make sure security groups allow the respective traffic between hosts.
And with this, we are completing the creating and running of ECS tasks tutorials. ECS services are used to manage the ECS tasks. We will walk through it in an upcoming article.
A step by step guide to create VPC endpoint for Amazon ECS
ECS VPC endpoints!
Let’s start with some VPC endpoint basics and why we need VPC endpoint for Amazon ECS. Followed by step by step procedure to create the VPC endpoints for ECS along with screenshots.
What is VPC endpoint?
The VPC endpoint is your gateway for communicating with AWS services public endpoints from resources having no internet access at all. Services like S3, ECS, API Gateway has public endpoints. So when you access them, your request will route through the internet to those service endpoints.
In a secure environment, where instances or resources in the private subnet have absolutely no access to the internet not even via NAT gateway etc., they will not be able to communicate with public AWS endpoints. In such cases, we can leverage VPC endpoints to communicate with such services using Amazon’s internal network (Amazon PrivateLink).
Even with internet access, since traffic is going out to the internet and then reaching AWS services it will have some delay. Using VPC Endpoint makes your access pretty fast using Amazon PrivateLink!
On the endpoint page displayed on right, click Create Endpoint
3 endpoints need to be created for ECS.
com.amazonaws.region.ecs-agent
com.amazonaws.region.ecs-telemetry
com.amazonaws.region.ecs
where the region is a region where the ECS cluster is running. In my case its us-east-1
Creating VPC Endpoint for ECS
Here list of fields to be set –
Service category: AWS services
Service Name: All 3 provided above.
VPC: Select VPC where ECS cluster is running
Subnets: Select subnets to associate endpoints with. I selected private subnets only.
Enable DNS name: Recommended to enable so that ECS agents can communicate with ECS service without any trouble.
Security Group: Security group to be attached to the ENI of this gateway. Make sure port 443 inbound traffic is allowed from above subnets
Tags: For identification
Finally, click the Create endpoint button. Repeat the same process to create 3 endpoints for the services mentioned above.
3 Endpoints should goto available status from pending.
3 VPC Endpoints for Amazon ECS
It is clear that each endpoint is having 2 ENIs in 2 subnets. i.e. one interface in each subnet.
This completes VPC Endpoint creation for ECS service. Now, ECS instances can make use of these interfaces when they spun up. If instances are already running then you need to restart the ECS agent on them using the below command and it will start using VPC Endpoints.
For testing, I just terminated existing ECS instances and the ECS autoscaling group spun up new ECS instances in a private zone (which does not have a NAT gateway so no internet). Both got registered to the ECS cluster successfully via VPC endpoint!
Private instances in ECS cluster
Troubleshooting:
In case ECS instances are not getting registered to the ECS cluster using VPC endpoints then the below points needs to be validated –
The instance is running ECS agent version 1.25.1 or higher
Security group of endpoints is allowing 443 traffic from instances
Endpoints are created in the same region as the ECS cluster
ECS agents are restarted on ECS instances after endpoints creation.
If ECS instances are registered but Agent connected is being shown as False. In such scenario below points needs to be validated –
Docker and ECS agent services are running on the server. (systemctl status docker/ecs)
The proper instance role (ecsInstanceRole) is attached to ECS instances. (curl http://169.254.169.254/latest/meta-data/iam/info)
Inspect logfile at location : /var/log/ecs/ecs-agent.log on ECS instances.
