Skip to main content

CI/CD Pipelines

CI/CD pipelines, or Continuous Integration/Continuous Deployment pipelines, function as an automated process that triggers on a determined action and automatically builds and hosts a project. This process is implemented to achieve better testing on a 'live'/'hosted' environment while also improving efficiency by removing the repeated effort associated with deployment.

Overview & Architecture​

There are three major portions of the CI/CD pipeline: GitHub workflow files, an internal Azure Linux virtual machine, and two or more (likely external) managed identities.

The process starts when some action on GitHub, such as a push to a branch or a release being made, triggers one of the GitHub actions workflows. The workflow leverages the Azure VM to build the project, and then hosts it to the Azure Web App using the managed identities for authentication.

Anatomy of Process​

This graphic shows the process in whole (click to open full in new tab). Below each piece is explained in detail.

GitHub​

Workflow Files

Generated initially by Azure (if following the guide below), workflow files are written in YAML and are stored in GitHub repositories under .github/workflows. They function as the procedures that GitHub actions follow when triggered. They specify what trigger begins the flow and what all occurs in the GitHub action. They're highly customizable and can be made to handle testing, add comments to PRs, create documentation, send notifications, manage tickets, and more.

Environments

GitHub natively allows the creation of 'Environments' under Settings -> Environments which can be used as permission-controlled layers for deploying since only org admins and repo owners can edit environments. When referenced by Azure managed identities and GitHub workflows, Environments can be used to ensure any trigger of the workflow or use of the identity follows the branch/tag name protection rules set by the environment. Additionally, environment-level encrypted secrets may be defined in environments and used in workflows.

FortisureIT Azure Tenant​

'GitHub-Actions' Virtual Machine

Currently hosted on Azure, under the [Fit-Internal] resource group, is a Linux x64 virtual machine named 'GitHub-Actions'. It is configured to have a dedicated ip that is allowed through our GitHub org ip restriction rules, allowing it to run build operations on any FortisureIT repo. This virtual machine is prepped with an org-wide GitHub runner that can be used to handle one GitHub action at a time, but is organized so repository-specific runners can be added for concurrent actions.

IMPORTANT NOTE: The VM only handles 'building' the image. Once it is built, no further step in the workflow requires access to the codebase, so GitHub spins up it's own ubuntu instance to communicate with Azure and handle hosting. The hosting step CAN be handled by the VM as well, but, following the guide below, has been left as a free GitHub job to save resources on our VM.

Client Web App Azure Tenant​

Managed Identities

Azure handles the authentication of non-user entities through managed identities. Typically these are connected to a resource, but they can be made as their own resource as well. Each managed identity may be given federations which function as an authentication method for the identity (A GitHub org-repo-environment combo as example). These identities may also be given any set of permissions ('website contributor', which is needed to deploy to a web app, as example).

Deployment Slots

Azure Web Apps, at premium scale (P0V3) or higher, allow for the creation of deployments slots. These slots are usually configured as a pair of two: 'dev' for internal testing and 'staging' for client testing. The guide below follows this setup, but can be tweaked to fit any setup. Each deployment slot, and the prod web app, can be configured to use 'GitHub actions' as a source under their respective deployment center.

FAQ​

The VM is on our tenant, but the web app is on the client's - will CI/CD still work?

  • Yes! The VM only handles building the project image. GitHub then spins up its own ubuntu instance and the deployment authentication is done between GitHub and the managed identities on the client's tenant. Review the 'GitHub-Actions' Virtual Machine section above for more specifics on this.

Are workflow files branch specific?
  • Yes! If changes are made to the workflow files in Main and not pulled to your current branch, the current branch's workflow files will be used when a trigger occurs on that branch (such as a PR or commit push).
How does role-based security actually work?

  • Two major portions protect the dev and staging branches (using the setup in this guide as example).
    • GitHub environment rules are protected by org/repo owner permissions.
    • Azure managed identities are protected by explicitly assigned Azure permissions.
  • Ex. a staging environment can be set up to only allow GitHub releases -> the managed identity can be set up to only federate on the staging GitHub environment -> the staging deployment slot can be set up to use the staging managed identity as its authentication for image pushes and hosting changes.
    • So, changing a workflow to push a branch to staging will always fail since it is not a tag and does not match the environment rules, meaning you must have permissions to make a release to push to the staging deployment slot.
      • Configuring GitHub rules and preventing developers with 'write' permissions from making releases can be found under other (currently TBA) documentation.

Why does it matter if secrets are repo-level or environment-level in GitHub?

  • Neither repo-level nor environment-level secrets can be read by any user. However, Repo-level secrets can be changed or deleted by any developer with 'write' permission or higher on the GitHub repo, while environment-level secrets can only be changed or deleted by org/repo owners. A small guide on how to migrate repo-level secrets to environment-level secrets can be found at the end of the guide below.

Setup & Implementation​

The following is a guide to adding a basic CI/CD pipeline to any given repo. This basic setup handles a dev and staging deployment slot structure. It assumes an intention of having release branches be hosted to the dev deployment slot for testing after each issue branch is merged in. Additionally it assumes an intention of having versioned releases on GitHub be hosted to the staging deployment slot (with a version-based tag) for easy swapping with production.

No further functionality is added by this basic setup, but workflows can be further developed to handle an array of tasks like testing, adding comments to PRs, creating documentation, sending notifications, managing tickets, and more.

1.) Setting Up Azure Tenant Resources​

Navigate to the Azure tenant that the target web app is hosted on - this is likely the client's external tenant.

Managed Identities​

First, create the managed identities that will allow GitHub to authenticate with your web app's deployments slots. Each should federate with GitHub specific information and have the role necessary to actually push an image to a web app deployment slot.

  1. Navigate to Managed Identities through the Azure resource search bar.
  2. Create two new managed identities in the same region and resource group as the target web app.
    • Best practice naming scheme: [GitHub repo name]-dev and [GitHub repo name]-staging.
    • Best practice tagging: add 'CreatedBy', 'CreatedOn', and 'Purpose' tags.
  3. Add a federation to each via Settings -> Federated credentials -> Add Credential
    • For Organization use 'Fortisure-IT'.
    • For the Repository use [your repo name].
    • For Entity use 'Environment'.
    • For the Staging identity's federation Environment use 'staging'
    • For the Dev identity's federation Environment use 'dev'
  4. Add the 'Website Contributor' role to each via Azure role assignments -> Add role assignment
    • Use resource group level scope and select the group associated with the target web app.
    • Ensure the selected role is 'Website Contributor'

Deployment Slot Setup​

Next Azure web app deployment slots can be configured to use GitHub actions as their source for images to host, using the above created managed identities as an authentication method.

  1. Have a GitHub admin temporarily disable IP protection.
    • Enterprise Settings -> Authentication Security -> IP allow list -> Enable 'All IPs'
  2. Navigate to the target web app deployment slots via App Services -> [your web app] -> Deployment slots
  3. For both the dev and staging slot, navigate to the slots' Deployment Center
  4. Select the 'GitHub Actions' Source option.
  5. Select GitHub repository information, using 'Main' as the selected branch.
    • You may have to refresh this page multiple times if repositories or branches are not found.
    • Beginning to type the repository or branch name can sometimes get the list to update.
    • Ensure that the GitHub IP protection is disabled during this step - it may take some time to apply.
  6. Select the 'Add a workflow' workflow option.
  7. Select 'User-assigned identity' for the authentication type, then select the identity that matches the deployment slot.
    • For the dev slot use the '[GitHub repo name]-dev' identity.
    • For the staging slot use the '[GitHub repo name]-staging' identity.
  8. Select the 'Azure Container Registry' image source, and select your container repository.
  9. After successfully saving the source, have a GitHub admin re-enable IP protection.
    • Enterprise Settings -> Authentication Security -> IP allow list -> Disable 'All IPs'

⚠ī¸ NOTE: Remember to re-enable GitHub IP protection.

2.) Setting Up GitHub Resources​

Navigate to the source GitHub repository. Note that there are now two new GitHub workflow files under .github/workflows.

Environments​

Once at the correct GitHub repository, an environment for each identity will need created. Rules are then applied to these environments by repo owners and GitHub admins to ensure that any deployments to the respective deployment slots follow said rules in GitHub (Ex. all dev slot deployments must be from a branch that follows a release branch naming pattern).

NOTE: You must be a GitHub admin or repo owner to complete this step.

  1. Navigate to Environments via Settings -> Environments
  2. Create two new environments: 'staging' and 'dev'
  3. Configure environment protection rules via Deployment branches and tags -> Add deployment branch or tag rule
    • For staging, add the tag pattern '*'
    • (optional) For dev, add the branch pattern '[0-9][0-9]?.[0-9][0-9]?-[A-Za-z]-*'
      • Note that protection rules are not necessary for dev - what branches automatically push to the dev slot are configured in the workflow file, but these rules will allow only certain branches to be applicable (in this case, only release branches).

Workflows​

Configuring the deployment slots like we've done in the 'Deployment Slot Setup' will automatically create some default workflow files in the specified GitHub repo. Now, we need to set them up to run with more specific triggers, to use the new environments, and to use our internal Azure virtual machine for building images.

  1. Updating the dev workflow file under .github/worflows

    • Update the trigger from 'any push to the main branch' to be 'any push to any release branch'.
      • At the top of the file replace Main with '[0-9][0-9]?.[0-9][0-9]?-[A-Za-z]*'
    • Update the build process from running on GitHub to running on our self-hosted virtual machine.
      • Under 'build:' replace ubuntu-latest with [self-hosted] or [self-hosted, repo-runner]
      • [self-hosted] will use the runner shared across our org, your CI/CD pipeline may wait in queue at times.
      • [self-hosted, repo-runner] will use a repo-level runner which will need set up using recommend steps below.
    • Specify the environment to use for authentication
      • Under both 'runs-on:' lines, add the line 'environment: dev' without indenting the line.

  2. Updating the staging workflow file under .github/workflows

    • Update the trigger from 'any push to the main branch' to be new releases.
      • At the top of the file replace the 'on:' section of the code with the following block
          on:
              release:
                  types: [published, prereleased]
              workflow_dispatch:
    • Update the build process from running on GitHub to running on our self-hosted virtual machine.
      • Under 'build:' replace ubuntu-latest with [self-hosted] or [self-hosted, repo-runner]
      • [self-hosted] will use the runner shared across our org, your CI/CD pipeline may wait in queue at times.
      • [self-hosted, repo-runner] will use a repo-level runner which will need set up using recommend steps below.
    • Specify the environment to use for authentication
      • Under both 'runs-on:' lines, add the line 'environment: staging' without indenting the line.
    • Change image tag to using release tag instead of GitHub sha (for better identifying release images in Azure)
      • Replace all instances of 'github.sha' with 'github.event.release.tag_name'

ℹī¸ NOTE: At the bottom of this page is an example version of each of these edited workflows.

3.) Setting Up Azure VM Resources​

The default runner already installed on the virtual machine can handle all repositories in the GitHub organization, but will slow if it is being used by multiple projects/teams at the same time. This make the default ideal for testing new pipelines, but shouldn't be used long-term.

While not expressly needed, it is recommended that with each new pipeline a new repo-specific runner be added to the virtual machine to handle concurrent build actions for that repository.

ℹī¸ NOTE: This section may need re-visited in the future to balance VM resource load vs storage load as additional runners take storage space.

NOTE: You must have, or rely on someone who has, admin access to the 'GitHub-Actions' Virtual Machine for this step.

  1. Log into the 'GitHub-Actions' virtual machine using the credentials for "GitHub Actions VM (Azure)" stored in Keeper.
    • Instructions on how to log in are stored in the 'Notes' section of the credential in Keeper.
  2. Navigate to the repo-runners folder using cd ./repo-runners.
    • Alternatively, the default runner is stored in 'root/actions-runner' if you need to access it.
  3. Create a new directory for your repository's runner using mkdir ./[repo name]-runner.
  4. Navigate into your new folder with cd ./[repo name]-runner.
  5. In a new browser instance, navigate to your GitHub repo and create a new runner via repo -> settings -> actions -> runners -> new-self-hosted-runner.
  6. Run the commands GitHub gives you in the terminal.
    • Skip the first step $ mkdir actions-runner; cd actions-runner since you've already created and navigated to a custom directory.
    • Skip the last step runs-on: self-hosted since we've already edited the workflow files.
  7. Once the runner is added, install GitHub Actions runner as a system service using sudo ./svc.sh install.
  8. Run the service using sudo ./svc.sh start
    • The runner service will now start on boot and allow for background execution or concurrent actions.
    • You can check the status of the runner with sudo ./svc.sh status

Use & Notes​

Your CI/CD pipeline will run based on the workflow files, identities, and slots you set up, and should automatically run when the trigger you specified takes place.

Using the Pipeline​

As mentioned, the pipeline is automatic and should run on the trigger you set in your workflow files during set up. If the set up process was followed exactly, the pipeline will: host release branches to the dev slot after every new push, and host each new release/pre-release to the staging slot on release creation. Do note that for the flows to function as intended, the finalized version of the workflows should be pulled to all branches in use.

To monitor and debug your workflows, the 'actions' tab at the top of your GitHub repository can be viewed. Additionally, during and after a workflow has ran on a specific branch, the latest state of the workflow is shown after the name of the latest commit on the 'code' page of the GitHub repository for that branch.

Additional Functions on Workflows/Actions​

Workflow files are highly customizable and can be made to handle testing, add comments to PRs, create documentation, send notifications, manage tickets, and more. Further, what triggers them can be changed to many options. The guide above is designed to get a basic CI/CD pipeline in place, but adjustment and improvement is encouraged.

We currently do not have any documentation on improvements/additions to these flows.

ℹī¸ NOTE: Documentation on any useful additions is more than welcome and should most likely go in a 'standards' or 'processes' directory on the docs site rather than in the tools section. However, adding a link to that directory/documentation here once it is created would be ideal.

GitHub Secret Key Environment-Level Security (Optional)​

As mentioned above, moving GitHub secrets from repository-level (Settings -> Secrets and variables -> Actions) to environment-level (Settings -> Environments -> Environment secrets) prevents developers with 'write' access to the repository from being able to change and delete the keys used by the workflows. Allowing changing of these keys could be dangerous as a developer could delete a key that is difficult to retrieve or, more impactfully, replace a key with their own value and cause the CI/CD pipeline to push to a different identity/web app entirely. These scenarios, however, would likely have to be intentional and malicious as it is not easy to accidentally access the secrets. So, moving secrets to the environment-level is best practice, but not functionally necessary.

Currently, we are unaware of an easy way to migrate keys to the environment-level since they cannot be read in GitHub. Likely the easiest/only way to achieve this is by recreating them one-by-one on the environment-level and regathering the values from Azure, then replacing their references in the workflow files.

ℹī¸ NOTE: If a better method is discovered, please document it here!

Example Workflow Files​

main_[repo-name](dev).yml 

    # Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
    # More GitHub Actions for Azure: https://github.com/Azure/actions

    name: Build and deploy container app to Azure Web App - #####

    on:
    push:
        branches:
        - '[0-9][0-9]?.[0-9][0-9]?-[A-Za-z]*' 
    workflow_dispatch:

    jobs:
    build:
        runs-on: [self-hosted, repo-runner]
        environment: dev
        permissions:
        contents: read #This is required for actions/checkout

        steps:
        - uses: actions/checkout@v4

        - name: Set up Docker Buildx
            uses: docker/setup-buildx-action@v2

        - name: Log in to container registry
            uses: docker/login-action@v2
            with:
            registry: #####.azurecr.io/
            username: ${{ secrets.##### }}
            password: ${{ secrets.##### }}

        - name: Build and push container image to registry
            uses: docker/build-push-action@v3
            with:
            context: .
            push: true
            tags: #####.azurecr.io/${{ secrets.##### }}/#####/#####/#####:${{ github.sha }}
            file: ./Dockerfile

    deploy:
        runs-on: ubuntu-latest
        environment: dev
        permissions:
        id-token: write #This is required for requesting the JWT
        contents: read #This is required for actions/checkout

        needs: build
        
        steps:
        
        - name: Login to Azure
            uses: azure/login@v2
            with:
            client-id: ${{ secrets.##### }}
            tenant-id: ${{ secrets.##### }}
            subscription-id: ${{ secrets.##### }}

        - name: Deploy to Azure Web App
            id: deploy-to-webapp
            uses: azure/webapps-deploy@v2
            with:
            app-name: '#####'
            slot-name: 'dev'
            images: '#####.azurecr.io/${{ secrets.##### }}/#####/#####/#####:${{ github.sha }}'

main_[repo-name](staging).yml 

    # Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
    # More GitHub Actions for Azure: https://github.com/Azure/actions

    name: Build and deploy container app to Azure Web App - #####

    on:
    release:
        types: [published, prereleased]
    workflow_dispatch:

    jobs:
    build:
        runs-on: [self-hosted, repo-runner]
        environment: staging
        permissions:
        contents: read #This is required for actions/checkout

        steps:
        - uses: actions/checkout@v4

        - name: Set up Docker Buildx
            uses: docker/setup-buildx-action@v2

        - name: Log in to container registry
            uses: docker/login-action@v2
            with:
            registry: #####.azurecr.io/
            username: ${{ secrets.##### }}
            password: ${{ secrets.##### }}

        - name: Build and push container image to registry
            uses: docker/build-push-action@v3
            with:
            context: .
            push: true
            tags: #####.azurecr.io/${{ secrets.##### }}/#####/#####/#####/#####:${{ github.event.release.tag_name }}
            file: ./Dockerfile

    deploy:
        runs-on: ubuntu-latest
        environment: staging
        permissions:
        id-token: write #This is required for requesting the JWT
        contents: read #This is required for actions/checkout

        needs: build
        
        steps:
        
        - name: Login to Azure
            uses: azure/login@v2
            with:
            client-id: ${{ secrets.##### }}
            tenant-id: ${{ secrets.##### }}
            subscription-id: ${{ secrets.##### }}

        - name: Deploy to Azure Web App
            id: deploy-to-webapp
            uses: azure/webapps-deploy@v2
            with:
            app-name: '#####'
            slot-name: 'staging'
            images: '#####.azurecr.io/${{ secrets.##### }}/#####/#####/#####/#####:${{ github.event.release.tag_name }}'

FAQ​

For the staging slot, why trigger on releases instead of pushes to Main?

  • When triggering on pushes to main, the only unique identifier is the GitHub commit sha, which is relatively unidentifieable to humans. This becomes particularly impactful when it comes time to swap images from staging to production in Azure. Instead, we use the tag given to GitHub releases so versioning between GitHub and Azure images is paired.

How do I know what image and tag are hosted in a CI/CD deployment slot?

  • Navigate to the deployment center of the deployment slot you want to identify, then select the 'Logs' tab at the top. Once the logs have loaded use ctrl+F to search for the name of your image (likely the repo name) with a colon afterwards. Or if you are unsure, use the name of your image repository on Azure, which will help you locate the subsequent image name. The currently hosted version will display after the colon.
    • As example cicdtest:v1.0.1 would imply the version v1.0.1.

ℹī¸ NOTE: This is typically most helpful in verifying a slot's current image before a swap is initiated.

How do I check workflow statuses and debug workflows?

  • The 'actions' tab at the top of your GitHub repository can be used to see all current and past runs of workflows in your repository. This does include all branches in the repository.
    • Failed runs will be displayed and can be clicked to view more details on what exactly failed.
    • Current runs can be clicked to view a real-time log of the process.
    • Some or multiple steps of the process can be re-ran from here.

How do I restart or check the status of runners?

  • To check the status of a runner on the 'GitHub Actions' virtual machine, first log into the VM using credentials in Keeper. Then navigate to the directory of the runner you are trying to check or restart (likely either cd actions-runner for the default or cd repo-runners/[repo name]-runner for a repository-specific runner).
    • To check the runner status use sudo ./svc.sh status
    • To restart the runner use sudo ./svc.sh stop; sudo ./svc.sh start

Last updated on by Zach Sedlack