As companies advance their IT operating infrastructure in the Cloud, inevitably the conversation of automating Infrastructure as Code (IaC) comes up. At Foghorn we have utilized several different automation tools in conjunction with Infrastructure as Code. One such tool is Azure DevOps Pipelines, a mature SaaS CI/CD automation tool managed by Microsoft that supports Azure, AWS, and GCP. Additionally, with hundreds of official and 3rd party tasks in the Azure DevOps marketplace, it is easy to see how this tool can bring great power and flexibility to any Cloud infrastructure.
Classic Pipelines
For years, Azure DevOps has supported what are called Classic GUI release pipelines. These GUI based automation pipelines were powerful but had some drawbacks.
- Build and Release pipelines are separate entities necessitating more time for engineers to configure
- Build pipeline takes a long time to run even though all that is being done in some instances is just checking out the repository code
- Lack of source control options to determine what has changed in the pipeline, and to subsequently roll back changes if needed
- No option for templated tasks
- Pipelines cannot be built from a source controlled file
- Classic GUI pipelines are no longer the default option whenever creating a new pipeline in Azure DevOps
YAML Multi-Stage Pipelines
In this post, the focus will be on deploying both infrastructure and code onto the Microsoft Azure Cloud using YAML Multi-Stage Pipelines in Azure. Azure DevOps YAML pipelines offer a robust, powerful, and repeatable deployment structure for a wide range of architectures and are now the default option when deploying new pipelines in Azure DevOps. A relatively new offering that only became generally available a few months ago, YAML Pipelines have several advantages to the Classic GUI experience:
- All new pipeline features in Azure DevOps will be released for YAML pipelines before the Classic GUI pipelines
- Environment level approval gates
- Source controlled pipelines
- Run multiple stages within one pipeline
- The ability to utilize YAML templates to increase code reuse
With these great new benefits for YAML pipelines in Azure DevOps, engineering teams can minimize the time it takes to ship their infrastructure and provide greater control and stability for deployments in the pipeline.
Prerequisites
Azure DevOps Pipelines must be configured to deploy to your Microsoft environment. When initially deploying an Azure DevOps pipeline, the following prerequisites must be completed:
- Create and enable new Azure DevOps Organization and Project.
- Link your Azure DevOps Organization to your Azure AD environment.
- Create and add all applicable users from your Azure AD and grant them the appropriate permissions to your project.
- Through the marketplace install any tasks that you would like available in your Azure DevOps organization or use any of the built-in tasks.
- Create a new service connection to any subscriptions that you would like to deploy infrastructure to in Azure.
- Decide on a code repository to use with Azure DevOps. We have chosen Github in this instance, it is tightly integrated with Microsoft and has an Azure DevOps pipeline integration app.
Once all the prerequisites have been built out and met, we are ready to begin with building a pipeline for deploying infrastructure.
Pipeline Schema and Architecture
Within each stage of the pipeline, a new virtual machine or container is created to run the pipeline code that is specified in your tasks. Microsoft provides a wide variety of hosted agents to run your code on, such as Ubuntu, Windows, and MacOS. This is useful when running a diverse set of tasks that require different OS environments across different stages but within the same pipeline. Organizations may have many use cases for Multi-Stage pipelines such as building the infrastructure in one stage and then deploying the application to the infrastructure in the next stage. In order to pass artifacts in between stages, Azure DevOps Pipelines has several built-in tasks to pass pipeline artifacts quickly and securely.
Before creating a pipeline in Azure DevOps, we must first create the YAML pipeline file in our IDE. Azure’s YAML Pipeline Schema can be found here. The basic outline is below:
Within each stage, there can be jobs, steps, tasks, scripts, and repo checkouts. Pipeline variables can also be clearly defined in the pipeline to pass into the various tasks that are a part of the pipeline, allowing for greater reusability of code. Additionally, to create repeatable steps to use in other pipelines, Azure YAML templates can be utilized. For this exercise, we will create a multistage pipeline that checks out the code from our github repository and proceeds to build out a storage account in Azure via PowerShell. To make this a repeatable process we will also use a yaml template.
Pipeline Code
First, we will create two new yaml files in our github repo. The first yaml will be for our pipeline, we will set the pipeline to only be triggered manually so we can fully test without triggering any pipeline runs. Alternatively, you could set the trigger to your master branch to automatically build the pipeline when new code is merged into master.
trigger: none
pr: noneNext we will define any variables that we will need during this exercise. We will see these referenced later with the $(variable) signification.
variables:
- name: cluster_prefix
value: 'prod'
- name: service_connection_name
value: 'prod'
- name: subscriptionid
value: 'zzzzzzz-xxxx-xxxx-xxxx-yyyyyyyyy'Lastly, we will reference the template that will read in the environment variables:
stages:
- template: Infra-template.ymlThen we need to take a look at the template itself. In this second yaml file, the first stage of the yaml is defined and built on the latest Windows image. Within the deployment we can see that the template will do a checkout of the current github repo, which is referred to as self. Then a script and a task are run, 1) to echo out one of the variables, 2) to actually run the PowerShell script.
stages:
- stage: Build
jobs:
- deployment: Build
pool:
vmImage: ‘windows-latest’
environment: 'Build'
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: |
echo Cluster Prefix =$(cluster_prefix)
displayName: 'Echo Cluster Prefix'
enabled: true
- task: AzurePowerShell@4
displayName: 'Azure PowerShell: Create Storage Account'
inputs:
azureSubscription: '$(service_connection_name)'
ScriptType: InlineScript
Inline: |
$SubscriptionID = "$(subscriptionid)"
$Cluster_Prefix = "$(cluster_prefix)"
$resourceGroup = $Cluster_Prefix + "_RG"
$StorageAccount = $Cluster_Prefix + "Storage"
$location = "centralus"
New-AzStorageAccount -ResourceGroupName $resourceGroup -Name $StorageAccount -Location $location -SkuName Standard_ZRS -Kind StorageV2 -EnableHttpsTrafficOnly $true
azurePowerShellVersion: LatestVersion
enabled: trueWith the yaml pipeline and template files created within the github repo, we can now create a new Azure DevOps pipeline to deploy using the yaml files.
- Use an Existing YAML Pipeline from your repo and branch and then identify the path to the existing yaml pipeline and you are then presented with the YAML pipeline that has been created based upon your file.
- You can now run the Yaml Pipeline with Azure DevOps and utilize the full power of the Azure YAML Pipelines.
Conclusion
In this post, we looked at the power of deploying code with Azure DevOps yaml multi-stage pipeline technology and how it can benefit DevOps shops. This powerful tool can be used to deploy to on-premise or in the Cloud and offers a wide array of flexibility. If you are interested in exploring how this technology can be deployed in your environment, please reach out to the Azure team at Foghorn and we would be more than happy to assist in building out this automation. In my next post we will dive in a little deeper on how to deploy infrastructure with Terraform in an Azure DevOps yaml pipeline.
