In late Nov 2023, amazon announced the new CloudFormation Git sync feature. Let’s explore this new feature; how it works, how it impacts CD patterns of Infrastructure as Code (IaC), etc.

What is the CloudFormation Git sync?

Recently announced CloudFormation Git sync feature lets customers deploy and sync the CloudFormation IaC code directly from remote Git repositories. This will be a game changer in the future as this feature might empower customers to omit the Continuous Deployment tools like Jenkins, GitHub Actions, etc. altogether and hence their maintenance.

In summary, the customer is required to establish a GitHub Connection with AWS and subsequently generate a CloudFormation stack using the relevant IaC template repository information. Once the stack is set up, CloudFormation continually monitors the template files in the remote Git repository. Any new commits in the remote repo, trigger the automatic deployment of the corresponding changes to AWS.

Pre-requisite for CloudFormation Git sync

• A Git repository with a valid CloudFormation IaC template
• A GitHub Connector configured for the target AWS account
• IAM role for Git Sync operations. It should have –
  • Access IAM policy
  • Trust policy

IAM policy

For tighter security control, one can scope down the IAM policy for certain CloudFormation Stacks using a resource block in the SyncToCloudFormation statement.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SyncToCloudFormation",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateChangeSet",
                "cloudformation:DeleteChangeSet",
                "cloudformation:DescribeChangeSet",
                "cloudformation:DescribeStackEvents",
                "cloudformation:DescribeStacks",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:GetTemplate",
                "cloudformation:ListChangeSets",
                "cloudformation:ListStacks",
                "cloudformation:ValidateTemplate"
            ]
        },
        {
            "Sid": "PolicyForManagedRules",
            "Effect": "Allow",
            "Action": [
                "events:PutRule",
                "events:PutTargets"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "events:ManagedBy": [
                        "cloudformation.sync.codeconnections.amazonaws.com"
                    ]
                }
            }
        },
        {
            "Sid": "PolicyForDescribingRule",
            "Effect": "Allow",
            "Action": "events:DescribeRule",
            "Resource": "*"
        }
    ]
}

Trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TrustPolicy",
            "Effect": "Allow",
            "Principal": {
                "Service": "cloudformation.sync.codeconnections.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

How to create a stack using CloudFormation Git Sync?

Let’s look at the step-by-step procedure to deploy a new CloudFormation stack using Git Sync.

Environment setup considered here –

• An AWS account configured with GitHub Connection in the Developer Tools Console.
• A Private GitHub repository cfn-git-sync-poc that hosts a valid CloudFormation template at cloudformation/s3-stack.yaml
• An IAM role was cfn-github-role created on the target AWS account with above stated policies.

Now, it’s time to get our hands dirty!

• Log in to the AWS CloudFormation console. Click on the Create stack dropdown button and choose With new resources (standard)
• On a Create stack page, select Template is ready and Sync from Git

• Start providing the stack details like the stack name of your choice and select automatic Deployment file creation. If you wish to create a deployment file, please refer to this file format details.

• Then, specify the Git repository information. Ensure that you have a configured and active GitHub connection to fill in the fields in this segment. Choose the repository containing the Infrastructure as Code (IaC), specify the deployment branch, and designate a Deployment file path where AWS will store the newly committed deployment file. Lastly, furnish the IAM role details that will be utilized for executing Git Sync operations.

• Lastly, enter the file path for the CloudFormation IaC template inside the remote Git repo and define the parameters that are declared in the template. These details will be used to build the Deployment file.

• Click the Next button and you will enter the Stack options page.
• Stack options are the same as any other CloudFormation stack hence we will not discuss them here.
• After verifying/modifying Stack options click Next button on the page.
• Now, you will enter the review page where you can see the Deployment file constructed using the details provided and it will be committed to the remote Git repo on defined location

• Review the rest of the configurations and click the Submit button.
• Now, CloudFormation will commit and push the Deployment file to the remote Git repo by raising a Pull Request (PR). The message will be flagged on the page. Meanwhile, since the Deployment file is not found in the remote repo, the Provisioning status will be marked as Failed. Once PR is approved, Deployment file will be available in the remote repo.

• Head over to GitHub and you should see a PR raised by Amazon to commit the deployment file into the remote repo.

• Review, approve, and merge the PR.
• Once PR is merged the code modification is detected by CloudFormation and it starts provisioning the stack.

• Once all defined resources are created, stack creation will complete and CloudFormation keep monitoring the Git repository for any new changes.

In multiple environments such as development, testing, staging, and production, it’s possible to utilize distinct Stack Deployment files while leveraging a single CloudFormation IaC template file. These varied Deployment files can be organized into separate folders within the same repository for better segregation. By selecting the appropriate deployment file based on the environment where the CloudFormation stack is being established, the process becomes more streamlined. Something like –

cloudformation
├── dev
│   └── deployment.yaml
├── test
│   └── deployment.yaml
├── staging
│   └── deployment.yaml
├── production
│   └── deployment.yaml
└── template.yaml

By selecting the appropriate deployment file based on the environment where the CloudFormation stack is being established, the process becomes more streamlined. I anticipate that the Git Sync feature will evolve with additional capabilities in the future, potentially prompting customers to reconsider the necessity of separate Continuous Deployment (CD) software.

What do you think about this feature?

What are your thoughts on the CloudFormation Git Sync feature? Could it potentially revolutionize the game? Will it make several Continuous Deployment tools obsolete? It feels like an ArgoCD for Kubernetes, operating in a remarkably similar manner. For sure, it may not yet offer extensive control over commit-wide deployments, but the potential for future enhancements is exciting. This feature appears to transform the Infrastructure as Code (IaC) landscape for AWS customers, possibly luring some back from alternative IaC platforms. Witnessing its development and utilization in enterprise productions is going to be exciting!

GitHub action code for automated CloudFormation template linting on PR

GitHub is a popular version control software used widely by companies. And it is the best place to manage your AWS IaC i.e. CloudFormation templates! With the ever-growing AWS infrastructure and hence the template versions, it’s always a good practice to have your CloudFormation templates linted for any syntax errors. It saves a lot of time as you know the errors beforehand and not at the time of deployment on AWS! It will be more time saving if it just gets linted when someone raises the pull request (PR) so that the code owner, as well as the developer, knows the code modifications in PR are linted and sane to be approved.

cfn-linter is the best CloudFormation linter available. It can be implemented via GitHub actions for automated lint actions on PR submission. I will walk you through the process of setting it up.

Understanding the flow

First of all, you need to get the list of modified files in the PR so that you can run a linter against it. It can be managed by using readymade available actions or using git commands. There are a couple of actions like tj-actions, Get all changed files, etc.

Once we got the list, we need to filter out files that are potentially not CloudFormation templates. You don’t want to feed non-template files to linter as that would result in failure. I did this using grep and also allowed the shell to continue even if grep exists with a non-zero exit code. This will prevent GitHub action from failing if there are no template files modified in the given PR.

Lastly, lint all templates one by one using cfn-lint. I am ignoring the warning using -i W flag to avoid failing GitHub actions due to warnings.

Code

All these points are summarised in below GitHub Action code –

name: Lint CloudFormation Templates
on: 
    pull_request:
      branches:
        - main
jobs:
  cloudformation-linter:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Cloud Formation Linter with Latest Version
        uses: scottbrenner/cfn-lint-action@v2     
      - name: Fetch changed file list
        uses: tj-actions/changed-files@v17.2
        id: changed-files
      - name: Run Linter
        shell: bash
        run: |
          > list
          for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
            echo $file >> list
          done
          set +e
          cat list | grep -e json -e yaml -e yml | grep -v .github > lint_list
          set -e
          if [ -s lint_list ]; then
          for i in `cat lint_list`
          do
          echo "Linting template: $i"
          cfn-lint -t $i -i W
          done
          else
          echo "No Cloudformation template detected in commit!"
          exit 0
          fi

You need to place this file with the name of your choice under <repo>/.github/workflows the directory. If you have some different master branch naming conventions or different strategies on when code should be listed, then make necessary changes in on: section.

GitHub Action

Once the action config is in place, PR submission will see automated checked in it.

If you click on Details, you will see the details about the action.

Your CloudFormation templates are being linted when PR is raised!