A quick post on how to resolve an issue with Gitbash that prevents MFA prompts while using AWS CLI.
MFA GitBsah issue.
Problem
GitBash under the hood uses winpty emulator for providing a bash experience on windows. Winpty does not work well with AWS CLI especially when dealing with MFA prompts. Hence you need to replace this with bash.exe and you should be good.
Procedure
Go to the Windows start menu and search for Git Bash. Click on Open file location.
Right click on the shortcut and select Properties
Under properties change the target from “C:\Program Files\Git\git-bash.exe” to “C:\Program Files\Git\bin\bash.exe“
The user is not able to register an MFA device. When a user tries to assign a new MFA, IAM throws an error –
This entity already exists. MFADevice entity at the same path and name already exists. Before you can add a new virtual MFA device, ask your administrator to delete the existing device using the CLI or API.
MFA assignment error
Whereas if you as admin or even user check the AWS console it shows Assigned MFA device as Not assigned for that user.
Resolution
As an administrator, you need to delete the MFA device (yes even if says not assigned) using AWS CLI. The performer needs to have IAM permission iam:DeleteVirtualMFADevice on to the given resource to update the IAM user’s MFA.
Run below command from AWS CLI –
# aws iam delete-virtual-mfa-device --serial-number arn:aws:iam::<AWS account number>:mfa/<username>
where –
AWS account number is account number where user exists
username is IAM username of that user
This should clear out the error message and the user should be able to register a new MFA device.
A quick reference to configure EC2 for Session Manager in AWS
EC2 session manager!
Ok this must be a very basic post for most of you and there is a readily available AWS doc for it, but I am just cutting it short to list down steps for achieving the objective quickly. You should go through the official AWS doc to understand all aspects of it but if you are on the clock then just follow along and get it set up in no time.
Checklist
Before you start, make sure you checked out these minimum configurations to get going.
Your EC2 is running supported Opertaing System. We are taking example of Linux here so all Linux versions that supports AWS Systems Manager supports session manager.
SSM agent 2.3+ installed on system. If not, we got it covered here.
Outbound 443 traffic should be allowed to below 3 endpoints. You must have this already covered since most of the setups has ALL traffic aalowed in outgoing security group rule. –
ec2messages.region.amazonaws.com
ssm.region.amazonaws.com
ssmmessages.region.amazonaws.com
In a nutshell, probably point 2 is the one you need to verify. If you are using AWS managed AMI then you got it covered for that too! But, if you are using custom-built, home-grown AMI then that might not be the case.
SSM agent installation
It’s a pretty basic RPM installation as you would do on any Linux platform. Download package relevant to your Linux version from here. Or global URLs for Linux agents –
If you do not have access to EC2 (Key lost or EC2 without keypair) then probably you need to re-launch the EC2. If your EC2 is part of an auto-scaling group (ASG) then it makes sense to add these commands in the user-data script for the launch template and launch a new EC2 from ASG.
Instance role permissions
Now the agent is up and running. The next step is to authorize the AWS Systems Manager service to perform actions on EC2. This is done via Instance Role. Create the IAM instance role with below IAM policy:
You can scope it down to a particular resource if you want. You can even add KMS encryption-related permissions in it if you are planning to encrypt session data using KMS encryption. An example can be found here.
Once done attach the role to EC2. If EC2 is already having a role attached to it then add the above policy to the existing role and you should be good.
In this post, I will be sharing various resources to help you prepare for the CKA exam. In addition, feel free to add resources you know in the comments section, which may help fellow readers.
Exam details
Offered by: The Cloud Native Computing Foundation (CNCF)
Duration: 2 hours
Type: Complete tasks on Linux CLI (Practical)
Number of questions/tasks: 15-20 (I had 17)
Mode: Online proctored
Cost: $375 (that includes one free retake). Watch out over LinkedIn or internet for coupons. They got good deals on black friday as well.
Result: It will be available in 24 hours from the exam completion.
You are allowed to open one additional browser tab to access K8s docs, K8s Github or K8s blog. You should not be clicking/opening any links other than these domains that includes K8s forum as well.
Study journey
Having Linux and container background helps picking up Kubernetes world quickly. No prior cloud experience is required.
If you are not familier with containerization, its good to look into it before starting with Kubernetes. Docker containers are the best place to get started with.
Practise course labs heavily. You may go through course quickly to understand the Kubernetes world but you need to spend more time on practising Kubernetes on CLI.
Once you are good with the theory and understood all aspects of Kubernetes world, Labs are the only places where you should spend all of your study time.
Once you are through all the scenarios/tasks provided by online courses, you can think of your own custom scenarios and try implementing them.
Tips
Practise! Practise!! Practise!!! The more you are familiar with the CLI and commands, the more time you will save during the exam. In addition, it helps to build your muscle memory for command arguments and gain those extra seconds during the exam.
CKA requires you to complete the given tasks in the Linux terminal (Ubuntu) CLI on a shared Kubernetes cluster setup. So, having good Linux background is added plus! Moreover, it helps you in navigating through CLI, editing files and save a lot of time.
Make use of -h frequently! If you are not sure about the command arguments, use a -h flag that lists arguments along with example commands. You can directly copy those example commands and edit them accordingly before executing. A quick way to get the job done rather than navigating through kubectl commands on Kubernetes documentation
Try to complete tasks using imperative commands rather than building spec files.
Read the question carefully and completely before creating any objects. Keep an eye on the namespaces mentioned in the questions. Assume default namespace when no specific namespace is mentioned.
Verify created objects to make sure they carry properties asked in questions. For pods, make sure they reach running state before proceeding.
Setting alias in Linux shell is one of the famous tips you will come across over the internet. Use it according to your comfort. I did not use it.
Always make sure you run the given context commands at the start of each task. It makes sure you are on the right cluster to perform the task.
Always make sure to return to the main terminal if you are doing ssh to other nodes for performing the tasks.
For Tasks mentioning sudo -i for root privileges, it’s good practice to switch to root as soon as you log in to the respective node rather than finding out you are not run after running some commands and investing time there!
If you are not familiar with Linux editors like vi, edit your spec files in the exam provided notepad and then copy the final version of the config directly on the terminal within the file rather than running around in Linux editors and wasting time.
Get familiar with copy, paste operations in the terminal. There are different key combinations depending on the operating system. Refer exam handbook for the same. Then, practise using those key combinations.
Use kubernetes.io/docs heavily during practice. If you are stuck at something, always try to search and get information from Kubernetes official documentation. This will make you comfortable navigating through the documentation site and hence saves some time during the exam. In addition, you will know exact keywords to search and exact links to click on topics you had a hard time studying.
It’s the student’s responsibility not to click/open any other sites than the allowed three. Search in K8s documentation may yield results with links to the K8s forum. You should not be clicking them. Make a habit of checking links before opening to avoid issues during the exams.
Please note that the exam simulator you get along with your exam booking has more challenging questions than the actual exam. They mentioned it explicitly there. So if your morale goes down pretty quickly, then it’s best not to check those questions just before the exam :P. They aim more at getting an in-depth understanding of how things run under the hood.
A short howto on configuring AWS CLI to switch roles
AWS CLI Switch Roles configuration
Requirement:
You have one AWS account that needs to switch roles before executing things on AWS. It’s an easy method on AWS console, but how to switch roles in AWS CLI.
Solution:
Let’s consider the below setup-
AWS IAM account with programmatic access – user101
Same IAM account having sts:AsumeRole permissions.
AWS IAM role for above said IAM user to assume (same or cross-account)- role101
Start with configuring the AWS CLI in a standard way.
$ aws configure --profile user101
AWS Access Key ID [None]: AKIAQX3SNXZGUQFOSK4T
AWS Secret Access Key [None]: 33hjtNbOq9otA/OjBgnAcawHQjxTKtpY465NrDxR
Default region name [us-east-1]: us-east-1
Default output format [None]: json
In some cases, your AWS environment must have MFA restrictions in place where the user user101 must have MFA enabled to switch to the role role101. In such a scenario, your role profile in credentials files should include MFA device ARN as well like below –
A quick rundown on how to hunt AWS resources that needs tagging
Scan AWS resources to tag
Tags are the most important and equally negligible AWS entity! As AWS spread grows in an organization they start to realize the importance of tags and then comes the projects for tagging existing resources!
At this stage, the first question on the table is how to search for AWS resources that need tagging? or How can we search non-tagged AWS resources?
It’s a very short process that can be summarised in a single picture!
On left hand side menu, select Tag Editor under Tagging.
Now you should have seelction on right hand side.
Select perticular region or All regions from Regions drop down.
Select specific resource or All supported resource types from Resource types drop down.
Tags – Optional: You can specify key, value details to search for specific tags. Since we are searching for resources that are not tagged lets keep it blank.
Finally, click on Search resources button and you are done!
You should be presented with list of AWS resources in specified regions that needs to be tagged like below.
List of AWS resources to tag
You can export the list to CSV as well for further data analytics.
A quick article on how to configure ALB Auth via Amazon Cognito for ConsoleMe webapp
Securing ConsoleMe with ALB Auth
In our last article, we looked at Netflix’s IAM management tool ConsoleMe. We installed it on the Ubuntu Linux machine with the Local install method, and we got it up and running with the default example configuration. In this article, we will walk you through the process of configuring ALB authentication for the ConsoleMe webapp.
With the default example configuration, ConsoleMe webapp opens up without any authentication. However, since ConsoleMe will manage your AWS account’s IAM, it’s not safe to keep your keys to kingdom open on the internet without any authentication mechanism in place. Hence, we will be protecting it with the ALB auth method.
ConsoleMe supports webapp authentication via –
ALB Auth (Recommended)
Google groups
OIDC/OAuth2
SAML
Headers
As recommended by ConsoleMe, we will move ahead with ALB Auth.
Let’s get into it.
Pre-requisites
ConsoleMe is up and running
ALB is configured to listen on HTTPS with target group configured on 8081 port with ConsoleMe instance as target
Above mentioned setup is working correctly and you are able to open up ConsoleMe webapp using ALB DNS name/DNS ALIAS you configured for ALB.
Before you proceed, you need to make few configuration changes in the Amazon Cognito user pool if you have followed the above link to create it.
Edit/ make sure you have the below configurations in the Cognito user pool’s App client settings :
Callback URLs are set to http://DNS-NAME/auth, http://DNS-NAME/oauth2/idpresponse. Where DNS-NAME is DNS name of ALB or the ALIAS defined for DNS name.
Enable Authorization code grant
Allowed Oauth scope has email, openid and profile enabled.
Apart from the steps in the above link, you need to add extra rules in the HTTPS listener that forwards below path patterns directly to the target group bypassing Cognito authentication.
/api/v1/get_roles*
/api/v2/mtls/roles/*
/api/v1/get_credentials*
/api/v1/myheaders/?
/api/v2/get_resource_url*
/noauth/v1/challenge_poller/*
/noauth/v1/challenge_generator/*
ConsoleMe leverages these path patterns to perform CLI actions and authentication.
After adding them, your listener should look like –
CLI path rules
Now, the final step is to do the custom configuration on the ConsoleMe part. As you are aware from the ConsoleMe installation that it runs the default example configuration. We should override that open configuration with the one that supports ALB Auth. Copy the sample configuration file from GitHub here.
Save this file as in the installation directory. We installed ConsoleMe in /consoleme directory, so it should be saved as /consoleme/consoleme.yaml
Edit below parameters in the configuration file –
application_admin: Email that will receive the approval requests
metadata_url: Replace region and Cognito pool id.
is_example_config: false
ses: Edit if you are using SES
aws: with relevent details
Restart application
systemctl restart consolme
OR
python consoleme/__main__.py
Now, your application is reading the newly created configuration file. Next, open up ALB’s DNS/ ALIAS DNS, and you will be prompted to log in from Cognito. We discussed this part in an earlier article on Cognito authentication.
Once you are successfully authenticated, you should see the ConsoleMe console with custom config!
Notice that the example configuration notice is vanished now. Also, you can see Cognito user ID’s email as a logged-in user in ConsoleMe!
We successfully enabled ALB Auth for securing ConsoleMe webapp!
A step by step guide to install ConsoleMe on Ubuntu Linux machine
ConsoleMe Ubuntu Local Install
ConsoleMe is an open-source web service published by Netflix. It is designed to make life easy for end-users and cloud administrators. Using ConsoleMe, cloud administrators can manage IAM permissions/credentials for IAM roles, S3 buckets, SQS queues, and SNS topics across multiple AWS accounts from a single interface. It also provides CLI called weep for AWS credentials management. That’s a fair introduction if you are not aware of the tool. Next, let’s get into the installation part.
ConsoleMe offers docker and local installs. We will walk you through local install in this article.
Pre-requisite:
A machine running Ubuntu 19.04+ with root access. I used Ubuntu 20.04 LTS x86.
Active and working package manager subscription to install packages
AWS access keys for above user if you are not using roles. I used keys (steps below)
Installation
We are installing ConsoleMe in /consoleme directory. If you want to install in another location, make the necessary changes in the commands below. Let me give you a list of commands you need to run as root –
Here, the first few commands are installing all the dependencies and related software/tools. Then, we are cloning the GitHub repo of the tool in /consoleme and lastly, we are running two containers.
These are Redis and dynamodb containers that ConsoleMe leverages for caching and aggregating the AWS accounts information. You can make use of AWS Redis and dynamodb table services, but for now, we will run these containers locally so that ConsoleMe will talk to them rather than AWS services.
I am avoiding putting up console outputs for frequently used commands like package installations etc., here.
Make sure both containers are up and running before proceeding to the next step –
root@kerneltalks:/consoleme# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5333cdee2202 cnadiminti/dynamodb-local "java -jar DynamoDBL…" 10 seconds ago Up 4 seconds 8000/tcp, 0.0.0.0:8005->8005/tcp consoleme-dynamodb
19ac354c3d70 redis:alpine "docker-entrypoint.s…" 10 seconds ago Up 4 seconds 0.0.0.0:6379->6379/tcp consoleme-redis
4cf931d38652 aaronshaf/dynamodb-admin:latest "node bin/dynamodb-a…" 10 seconds ago Up 4 seconds 0.0.0.0:8001->8001/tcp consoleme-dynamodb-admin
Now, you need to prepare the machine to talk with AWS for fetching account details in the upcoming install steps. Ensure that you have set up account and permissions perfectly in IAM (mentioned in the pre-requisite above) to avoid any issues. You can do that by configuring AWS profile –
root@kerneltalks:/consoleme# aws configure
AWS Access Key ID [None]: AKIAQX3STVKIYRO36XEC
AWS Secret Access Key [None]: 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
Default region name [None]:
Default output format [None]:
Lastly, create a new python environment and run the final install step. This will take a while to complete since at the end of make install command, it also fetches and caches the AWS account details in the local Redis cache –
python3 -m venv env
. env/bin/activate
make install
After successful installation, you should be able to start the application.
Running ConsoleMe
On a current shell, you can run the ConsoleMe with the command. If you are in another shell, activate the python environment again –
But, it will exit out when you terminate the command or shell. It’s safe to run it in the background or, even better, run it as a Linux service. For running ConsoleMe as a service, create below two files –
Now that your console service is running, you should load its GUI on a web browser. The service listens on the 8081 port, so you need to navigate the server address with port 8081. Make sure the security group is allowing 8081 traffic if you are installing on EC2.
At this point, ConsoleMe is running with the default open example configuration. It’s very well highlighted on the web app as a warning. It would be best if you were editing this configuration to make your ConsoleMe more secure. ConsoleMe recommends Application Load Balancer authentication for securing your web app GUI. Refer to our next article on how to secure the ConsoleMe web app using ALB authentication.
Navigate to the Load Balancing > Load Balancers in left sidebar menu
On Load balancers page, select the Application load balancer that needs to be configured
Click on Listeners tab in below details pane.
Click on View/Edit rules against HTTPS 443 listner
You should see the editor window where you need to click on pen icons to open up the rule editor.
Edit 443 rule
Now, click on + Add action button. Select Authenticate from the drop-down. The rule will be listed in the editor where you need to select from dropdowns-
Cognito user pool
Client ID
Once you select Cognito user pool it will accordingly populate values for Client ID in that pool. Choose respectively. Keep other options untouched unless you have other requirements.
New Authenticate action will be prioritized to rule 1 and existing Forward to action will be getting down to number 2. Your rule windows should look like this –
Adding Cognito authenticate action in ALB listener rule
Click on Update to save the rule.
If you are facing any issue to populate User pool ID and seeing Too many requests error in console then you can opt to re-create ALB with this configuration using AWS CLI or CloudFormation. Because using AWS CLI, you can not edit the default rule. If you have ALB created from the Cloudformation template already then you just need to tweak your template and add the below code for the ALB listener resource.
Make sure your HTTP listener rule (port 80) is set to redirect to HTTPS.
ALB HTTP redirect rule
Now ALB is ready to authenticate users before their requests are forwarded to target group targets. Grab the ALB DNS from the load balancers page. If you are having CNAME/ALIAS entry for ALB then you should be using the custom DNS you have defined.
Navigate to the Load Balancing > Load Balancers in left sidebar menu
On Load balancers page, select the Application load balancer that needs to be configured
Click on Description tab in below details pane.
Copy DNS name
Getting the Application load balancer DNS name
Paste DNS name in the browser and you should be redirected to the authentication page from Amazon Cognito. You can customise this page along with its domain as we explained in our earlier article on the Amazon Cognito user pool.
Amazon Cognito authentication in action!
Notice that we configured a custom domain for Amazon Cognito hence our request is served with a custom domain auth.kerneltalks.com. If you observe the URL carefully, you can see Client ID, Callback URL, etc. details configured in Amazon Cognito when we created the user pool.
For any Cognito user, the default behaviour is to force change password on the first login which we observed here –
Cognito force change password on first login
During change password, you can even see password policy applied while creating the Amazon Cognito user pool is being checked –
Cognito user pool password policy in action
For testing purposes, if you want to bypass all the above password change procedures then you can set the password from AWS CLI using the below command –
Once the password is changed successfully, you will be redirected to the Callback URL configured in Amazon Cognito. We configured it to the ALB DNS name. So we redirected to it and backend EC2 target served us a sample webserver page!
Thus we configured Amazon Cognito authentication on the Application Load balancer and secured our targets!
A step by step procedure to create an Amazon Cognito user pool. All available options are explained.
Amazon Cognito user pool!
One of the best features of AWS application load Balancers (ALB) is authentication! You can offload authentication to ALB that leverages Amazon Cognito in the backend. Amazon Cognito offers identity management through user pools or federated identities. This article will walk you through creating a user pool in Amazon Cognito that is used for ALB authentication. Without further delay, let’s get into it.
On the User pools page, click on Create a user pool button on top right hand corner of the page.
That should start user pool creation wizard. Lets go through it one by one –
Name
Enter the Pool name and click on the Step through settings button.
User pool creation wizard
Attributes
Settings on this page can not be edited later so choose wisely!
The first thing you need to choose is the end user’s sign-in method. They should use a username or email address/phone number to signup/sign in. I am choosing a username and also, allowing them to use email addresses while logging in later once they sign up. I am also selecting case sensitive usernames because that makes more sense.
via CognitoChoose the way end-user sign in
The next section of attributes, let you choose through the list of attributes you want the end user to provide when they sign up in Cognito. You can also choose to add a custom attribute here if one is not listed in the standard list.
Set end-user attributes
Policies
End-user password policies and controls are defined in this section. All the fields are pretty self-explanatory.
Cognito user pool password policies and account control
MFA and verifications
An extra layer of account security can be defined here. MFA and related configurations. Please note that if you are enabling MFA for end-users then you should be enabling phone number attributes in earlier settings and text messages (verification and subsequent messaging) will incur extra charges. Amazon pretty much explained each option here.
Cognito MFA settings
If you are opting for adding and managing phone number attributes then you need to create a role that provides access to Cognito for sending text messages on your behalf.
SMS access related settings
Messages customizations
In this section, you should be customizing the email or SMS messages being sent out by Amazon Cognito on your behalf. It’s a place if you want to have company branding in the communications! Make sure you have a verified email address in Amazon SES to set it as From email address.
Cognito messages customizations
In the later part of the page, you can configure how you want verification to be done using codes or clickable links. Also, you can customize the text of the message here.
Tags
A place that is crucial but mostly ignored by everyone! Tagging for the user pool.
Amazon Cognito user pool tags
Devices
Choose if Cognito should remember the user’s device. This will enhance the user experience. But, to use this feature you should have MFA enabled for end-users. Since we did not opt for it, we will simply say No and move forward.
Remember user device
App clients
In this section, you should create an app client which will access this user pool. On the creation of the app client, you will receive the app ID and secret key that you can configure in your applications to access this user pool.
Click on Add an app client
Cognito app client settings
App client name: Add unique name
Refresh token expiration: Refresh tokens are used to retrieve new ID and access token. Control their expiration here. Read more about refresh tokens
Access token expiration: Used for autorizing the API operation. Control expiration here. Read more about access tokens
ID token expiration: It used to claim the authenticated user’s identity. Define its expiration limits here. Read more about ID tokens.
Auth Flows Configuration: Enable depends on your integration requirements. I selected ALLOW_USER_PASSWORD_AUTH and left others untouched.
Cognito app client security settings
Security Configuration: It allows to send back generic error. Select recommened unless you have any other reason not to!
Advanced token settings: Enable or disable token revocations.
Attributes read and write permissions: Select list of attributes which this app client can read or write.
Click on Create app client. It will be created along with the user pool when you completes the whole wizard.
Click on the Next step to move forward in the user pool creation wizard.
Cognito app client
Triggers
On this page, you can configure lambda functions to be triggered on specific actions or workflow. You need to create Lambda functions in advance to select here from the dropdown. List of triggers available here –
Pre sign up
Pre authentication
Custom message
Post authentication
Post confirmation
Define Auth Challenge
Create auth challlenge
Verify authc challenge response
User migration
Pre token generation
All triggers are listed with descriptions for easy understanding of when they will be activated and execute related Lambda functions. For the simplicity of this article, we are not adding any.
Cognito Lambda triggers
Review
Review all the details you supplied throughout the wizard. You can make edits if necessary and then lastly click on Create pool
User pool created!
You should be greeted with a success message and the user pool management page. You can note the user pool ID generated for this user pool.
Amazon app clients settings
Now, that you created a user pool and app client for it. Let’s look at some of the settings those needs to be checked or changed to make sure your app client is ready to be consumed.
Navigate to App integration > App client settings on the left sidebar menu on the user pool page.
Enable Cognito user pool under Enabled identity providers.
You should be having Callback URLs handy to fill in here. Those are URLs where app will be navigated once successful authentication happens. Your application developers should be able to help you with these details.
Sign out URLs are those where user will be redirected once its signed out from IDP session
OAuth 2.0 settings should be discussed with developer and configured as the app requirement
app client IDP settings
What is Amazon Cognito domain and how to configure it?
It’s a domain prefix with FQDN https://<prefix>.auth.<region>.amazoncognito.com where,
prefix : unique identifier of your choice
region: AWS region where user pool is hosted.
This domain is used to host sign-up and sign-in pages by Amazon Cognito. You can edit those pages for your company branding as well as explained in the next step.
Navigate to App integration > Domain name on the left sidebar menu on the user pool page.
Amazon Cognito Domain
Enter the prefix in the given text box and click Check availability. It will make sure you chose a unique prefix. Click on Save changes
You can opt to choose your own domain as well. You need to have an associated SSL certificate in ACM and permission to add the ALIAS record in the DNS hosted zone.
Custom domain in Aamzon Cognito
Once done, Cognito will create Amazon Cloudfront distribution for that domain in the backend and supply you with the alias target value to be configured in the hosted zone.
Amazon Cognito custom domain alias
Add ALIAS record (CNAME for non-Route53) for Domain name and Alias target mentioned above. Once done and CloudFront distribution is created, your domain status will be set to ACTIVE.
Cognito custom domain set
How to change login UI of Amazon Cognito?
Navigate to App integration > App client settings on the left sidebar menu on the user pool page.
On the last part of the page, you can find Hosted UI settings. There you will be able to play around with CSS, logo files to create a new custom login page.
Make sure you have Amazon Cognito domain name defined and at least one OAuth scope defined (above step)
How to retrieve Amazon app client secret?
Navigate to General settings > App clients on the left sidebar menu on the user pool page. And there you can retrieve app client secret.
