With the rise of DevOps, organizations are now required to have a continuous delivery pipeline. Azure Pipelines is an easy way to get started with building and maintaining your own pipelines for any application or service running on Microsoft’s cloud platform.
The “azure devops pipeline medium” is a blog post that walks through the steps of building an Azure DevOps Pipeline from soup to nuts. The article’s author, Chris, even includes a video tutorial at the end of the article.
You may access many blog articles, documentation, and tutorials on Azure DevOps by searching online. All of these materials are useful, but only a few of them lead you through a real-world situation. Many people overlook the security side, putting passwords in plain text for convenience or a finished product that accomplishes nothing. Let’s make a difference.
This article/tutorial will teach you how to design a true Azure DevOps release pipeline that automates infrastructure from start to finish. You’ll learn how to leverage Azure DevOps to create a continuous deployment pipeline to provide Azure virtual machines in more detail.
You will have a completely functional Azure pipeline at the conclusion of this project. It will: from a single GitHub repo commit
- Create an Azure resource group that is just temporary.
- Using an ARM template, create an Azure VM.
- Incorporate the ARM template into a CI/CD process.
- Start a template validation test whenever the template is changed.
- Install the Azure ARM template.
- Examine the installed infrastructure.
- Dismantle all Azure resources
Let’s get started!
Overview of the Project
This project will be divided into six distinct components. They are:
Getting Azure Resources Ready
You’ll learn how to set up all of the required resources in Azure in this section. You’ll find:
- For each job in the pipeline, create an Azure service principal.
- Set up an Azure Key Vault with the pipeline’s secrets.
- Establish proper access controls for ARM installations and pipeline use.
Preparing for Azure DevOps
After you’ve set up all of your Azure resources, it’s time to get Azure DevOps ready for your pipeline. This section contains the following information:
Overview of script/template
The ARM template for building the server, as well as the Pester tests, are among the project’s artifacts. We’ll go through what the template is supplying and what Pester is testing in the pipeline in this part.
Creating a pipeline
This is when the real excitement starts. You’ll start constructing the pipeline itself. You’ll learn how to build up this full orchestration with only one YAML file in this section.
The Multi-Stage Pipeline UI will be used to construct the pipeline. This feature is currently under Preview as of this writing.
Demonstration of pipeline
You must see the pipeline in action after it has been constructed. This part will teach you how to activate the pipeline and sit back and watch the magic unfold.
Cleanup
Finally, since this is simply a demo, you’ll have access to a script that will break down everything you developed throughout the course.
Does this seem excessive? It is! But don’t worry; you’ll learn by doing as you go through each chore one by one.
You can locate demo.ps1 in the ServerAutomationDemo GitHub repo, which contains all of the Azure CLI instructions needed to construct this pipeline.
Prerequisites
You’ll learn a lot, but you’ll also be expected to bring a few things to the table. If you want to follow along, make sure you have these items:
The key vault is accessible to ARM installations.
- If operating locally, use Cloud Shell or PowerShell 6+. Although the examples may work with Windows PowerShell, they have not been thoroughly tested. Although all of the examples will be run locally using a PowerShell terminal, the Cloud Shell will also function. Building the pipeline will be automated.
- Install Azure CLI (if operating locally) – In this post, you’ll learn how to use the Azure CLI to conduct tasks. The Azure Portal, PowerShell, and the Azure SDK may all be used to execute the same tasks.
Warning: Unless you have Azure credit, the acts you’re about to do will cost real money. The most expensive resource you’ll create in Azure is a virtual machine, but only for a limited time.
Before You Begin
In this session, you’ll be performing a lot of setup. Please make sure you have the following things before you begin.
- The name of the Azure subscription resource to which the resources will be deployed — in this case, Adam the Automator.
- The subscription’s identifier
- The Azure Active Directory tenant ID
- The name of the DevOps organization – adbertram will be used in the instances.
- The area in which you’ll be allocating resources – in the examples, eastus will be used.
- The name of the Azure resource group where the temporary key vault will be stored – in the examples, ServerAutomationDemo will be used.
- A password to provide to a deployed VM’s local administrator account – the examples will use “I enjoy azure.”
- The GitHub URL will be https://github.com/adbertram/ServerAutomationDemo in the examples.
Using the Azure CLI to log in
Prepare to work with the Azure CLI a lot in this article. Although I prefer the Azure PowerShell cmdlets, the Azure CLI can presently do more DevOps activities.
The first step is to open a PowerShell 6+ console. Using the command az login, log in to Azure from the terminal. This command will open a browser window and ask for your username and password.
Once you’ve been verified, make sure your subscription is set to default. Setting it as the default will save you the trouble of having to define it every time.
—subscription ‘Adam the Automator’ az login az account set
Getting Azure Resources Ready
It’s time to get to work using the Azure CLI after you’ve signed in. There are several dependencies and knobs to adjust in an Azure Pipeline. You’ll learn how to set up and prepare your environment for your pipeline in this first segment.
The Azure CLI DevOps Extension is installed.
You’ll need a means to use the Azure CLI to create the different Azure DevOps components. That capability is not included by default. You’ll need to install the DevOps extension to administer Azure DevOps via the Azure CLI.
Fortunately, the extension may be installed with only one line, as seen below.
—name azure-devops az extension add
Set your organization as the default after installing the extension to avoid having to specify it again.
—defaults organization=https://dev.azure.com/adbertram az devops configure
Putting Together a Resource Group
Despite the fact that the pipeline will establish a temporary resource group, you should build one for any resources mentioned in this sample. You’ll build an Azure Key Vault in this resource group, precisely.
—location “eastus” —name “ServerAutomationDemo” az group create
The Azure Service Principal is created.
The next step is to set up an Azure service principle. To authenticate to the Azure Key Vault, you’ll need an Azure service principal. This service principal will also be used to authenticate a service connection. As illustrated below, create the service principle for both the key vault and the final ARM deployment.
“http://ServerAutomationDemo” as $spIdUri —name $sp = az ad sp create-for-rbac ConvertFrom-Json | $spIdUri
It’s a good idea to preserve the value of $sp.appId someplace at this stage. You’ll need this later when you’re Putting Together the Pipeline!
Some PowerShell commands, such as ConvertFrom-Json, are used in the examples. Because the Azure CLI only gives JSON strings, converting to a PowerShell object makes it simpler to access properties.
Constructing the Key Vault
A handful of credentials are required in this tutorial’s pipeline. Let’s do it the proper way instead of keeping passwords in plain text. An Azure Key Vault will be used to store all sensitive data.
Use the az keyvault create command to construct the key vault, as illustrated below. This command builds the key vault in the resource group that was previously formed. The enabled-for-template-deployment option is also worth noting. This modifies the key vault access policy to enable access to the key vault by future ARM deployments.
—name “ServerAutomationDemo-KV” —resource-group “ServerAutomationDemo” —enabled-for-template-deployment true az keyvault create —location $region
Giving ARM access to the keyvault
Secrets of the Key Vault
It’s time to make the secrets now that the key vault has been built. Create two secrets named ServerAutomationDemo-AppPw and StandardVmAdminPassword for this demonstration. The AppPw password is the service principal’s password. The local administrator account on the deployed VM will be given the VM password.
—name “ServerAutomationDemo-AppPw” —value az keyvault secret set $sp.password “ServerAutomationDemo-KV” —vault-name —name StandardVmAdminPassword —value “I enjoy Azure” az keyvault secret set “ServerAutomationDemo-KV” —vault-name
In this example, you’re utilizing already created PowerShell variables. The service principal password ($sp.password) you got previously is entered here.
Providing access to the Key Vault to the Pipeline
The pipeline then requires authorization to access the key vault. Relax the rules on key vault access a little. To handle key vault secrets, give the service principal generated get and list rights.
—name “ServerAutomationDemo-KV” —spn $spIdUri —secret-permissions get list az keyvault set-policy
Getting Azure DevOps Ready
You now have all of the Getting Azure Resources Ready done. It’s time to do some preparation work in Azure DevOps.
Pester Extension Installation
Installing the PesterRunner Azure DevOps extension is the first step. To confirm that the VM ARM deployment was successful, the pipeline will execute two sets of Pester tests. The PesterRunner addon is one of the simplest ways to run Pester tests.
Use the command below to install the extension.
—extension-id PesterRunner —publisher-id Pester az devops extension install
Establishing an Azure DevOps Project
It’s finally time to start working on the pipeline project. The Azure CLI makes creating an Azure DevOps pipeline a breeze. To create the project and make it your default, just perform the instructions below.
—name “ServerAutomationDemo” az devops project create project=ServerAutomationDemo az devops setup —defaults
Developing Service Connections
Your pipeline must log in to two services: ARM and your GitHub repository. This requires the creation of two service connections.
Create the ARM service endpoint first. The password for the service principal will be prompted by the command below. Make sure it’s visible on the terminal before copying it to your clipboard.
Make sure to change the subscription name with your subscription ID and tenant ID.
$sp.password should be run and copied to the clipboard. ## construct the service endpoint az devops service-endpoint $sp.Password azurerm —azure-rm-service-principal-id $sp.appId —azure-rm-subscription-id “YOURSUBSCRIPTIONIDHERE” —azure-rm-subscription-name ‘Adam the Automator’ create —azure-rm-subscription-name ‘Adam the Automator’ —name ‘ARM’ —azure-rm-tenant-id $tenantId
Create a GitHub service connection next. The pipeline must be able to read the repo since it will be activated by a Git commit.
That GitHub personal access token comes in helpful at this time. You’ll need to put the service principal password in again here. Both service connections are utilizing the same service principle.
$gitHubServiceEndpoint = az devops github create —github-url ‘https://github.com/adbertram/ServerAutomationDemo’ —name ‘GitHub’ | ConvertFrom-Json ## when asked, put in the GitHub token
Putting Together the Variable Group
For two passwords, the pipeline will refer to key vault secrets. To do so safely, you’ll need to create a variable group and connect it to the key vault.
Make the variable group first, as seen below.
—name “ServerAutomationDemo” —authorize true —variables foo=bar az pipelines variable-group create
Have you seen the variable foo=bar? This isn’t utilized, however the variable group must be created with just one variable.
Adding a Key Vault to the Variable Group
Unfortunately, you’ll have to visit the Azure DevOps interface at this stage. As of this writing, there is no method to attach a key vault to a variable group using the Azure CLI.
Select Library from the Azure DevOps project menu. The ServerAutomationDemo variable group should appear as shown below. Select the variable group ServerAutomationDemo.
Group of variables available
Click on Link secrets from an Azure key vault as variables once you’re in the variable group. After that, you’ll be alerted that you’ll be removing all variables, and you’ll need to click Confirm. You’ll find instructions for doing so below. This operation is acceptable since the foo variable was always temporary.
Connecting the variable group to the pipeline
As seen here, validate the ARM service connection and the ServerAutomationDemo-KV key vault you built before. Select Add.
establishing a service connection
Now, as shown below, check both of the secrets you made previously and click OK and Save to save your modifications.
Choosing keyvault secrets to utilize in the pipeline
Overview of Project Files
Congratulations for making it this far! You may now start constructing the pipeline. But hold on…more! there’s
This tutorial creates a pipeline using “unit” and “acceptance” tests to make developing an Azure Pipeline more realistic. This adds to the tutorial’s appeal, but it also necessitates some further explanation of what’s going on.
A few files are included in the GitHub repository for this lesson, as mentioned below. It’s a good idea to either clone this repo or construct your own from the files right now.
List of GitHub files
- The completed YAML pipeline is azure-pipelines.yml.
- connect-azure.ps1 is a PowerShell script that allows you to connect to an Azure subscription.
- server.infrastructure.tests.ps1 – A quick Pester test to ensure the VM setup is correct.
- server.json — An Azure ARM template for creating a virtual machine.
- server.parameters.json — An Azure ARM parameters template contains parameter values for the ARM template.
Make that the server.parameters.json file has your subscription ID and key vault name for the key vault IT.
- Pester “unit” tests to ensure the ARM template is valid in server.templates.tests.ps1.
In a moment, you’ll see how each of these files fits into the process.
Putting Together the Pipeline
It’s time to set up the pipeline, assuming you’ve cloned my GitHub repo or created one on your own. Run the az pipelines create command to do so. The command below generates a pipeline named ServerAutomationDemo, which uses the specified GitHub repository as a trigger. It will examine the master branch and make use of the service connection established before.
az pipelines create —repository “https://github.com/adbertram/ServerAutomationDemo” —name “ServerAutomationDemo” —service-connection $gitHubServiceEndpoint.id —branch master —skip-run
You may or may not get feedback like the one below, depending on whether you have the azure-pipelines.yml file in your GitHub project. Your console will appear same in any case. Remember to bring your GitHub personal access token!
YAML Pipeline Analysis
Your pipeline is now ready to operate, but it’s critical to first comprehend the YAML pipeline. Examine the azure-pipelines.yml configuration file. When utilizing the multi-stage YAML pipeline functionality, this file is the pipeline.
Let’s have a look at the different parts that make up this YAML pipeline.
The Initiator
Since you’re building a CI pipeline that automatically runs, you need a trigger. The Initiator below below instructs the pipeline to run when a commit is detected in the Git master branch.
Also, take note of the pathways section. If you don’t specify which files or folders to include or exclude in a CI build, the pipeline will execute when any file is committed. Because this project is based on an ARM template, you don’t want to execute the pipeline if you make a change to a Pester test, for example.
branches: include: trigger: – master routes consist of: – json-server – parameters.server.json
The Pool
An agent is required for every construction. Every build agent must run on a virtual machine. The VM is running on the ubuntu-latest VM image in this situation. This is the build’s default image, which was set when it was first built. Because of the “simplicity” of this process, nothing has changed.
vmImage: “ubuntu-latest” pool
Variables
Following that are all of the variables as well as the variable group. Reading data like as the Azure subscription ID, tenancy ID, and application ID for the service principal are required by the many activities in this pipeline. They are defined as variables rather than copying static values in each activity.
Also, pay attention to the group element. The variable group you generated previously is referenced by this element. At this point, be sure to update the subscription id and tenant id.
Remember in the The Azure Service Principal is created. section you were reminded to save the value of $sp.appId somewhere? This is where you’ll need it. Assign the value of that service principal application ID to application_id as shown below.
variables: – ServerAutomationDemo group – value: “ServerProvisionTesting-$(Build.BuildId)” – name: azure resource group name – value: “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX – value: “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX – name: tenant id value: “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The azure resource group name variable’s value should be noted. You’ll see $ within that value (Build.BuildId). This is a system variable that holds the current job’s build ID. It is being utilized in this situation to verify that the temporary resource group formed is unique.
Preparation Tasks for PowerShell
The following tasks run PowerShell code. For testing purposes, this pipeline example utilizes PowerShell to establish and delete a temporary resource group. Two instances of PowerShell code invocation may be found in these deployment tasks.
The first job runs the connect-azure.ps1 script from the GitHub repository. This job connects to the Azure subscription in order to perform the Azure PowerShell instructions that follow.
This Azure PowerShell connect task runs the script and passes the pipeline variables subscription id, application id, and tenant id along with a key vault secret value (ServerAutomationDemo-AppPw).
The second duty is to execute PowerShell code inline, which means that no script is already in place. Instead, the value of the azure resource group name pipeline variable is used to specify PowerShell code in the pipeline YAML.
– responsibility: [email protected] inputs: “connect-azure.ps1” filePath ‘-ServicePrincipalPassword “$(ServerAutomationDemo-AppPw)” as parameters -SubscriptionId $(subscription id) -ApplicationId $(application id) – responsibility: [email protected] targetType: “inline” inputs $(azure resource group name) -Location eastus -Force New-AzResourceGroup
Pester Template Evaluation
The first Pester test is coming next. It’s critical to have a few distinct levels of tests in a CI/CD pipeline like this. You could write numerous unit tests if you were building a pipeline for a software project.
Because this sample pipeline is based on a single ARM VM deployment, the JSON template’s validity will be the first “unit” test. You may include as many different tests on the ARM template file itself as you’d like within the server.templates.tests.ps1 file.
The pipeline below uses a variety of system variables. These variables correspond to the location of the files on the build agent after they are loaded.
The PesterRunner job writes the test results to an XML file, which is then read in the pipeline.
– responsibility: [email protected] script as input Folder: “[email protected]; @Path=’$(System.DefaultWorkingDirectory)/server.template.tests.ps1′; {ResourceGroupName=’$(azure resource group name)’}} “$(System.DefaultWorkingDirectory)/server.template.tests.XML” resultsFile” use True run32Bit: PSCore False
The ARM Virtual Machine Deployment
We’ve arrived to the ARM deployment. This is critical since the whole pipeline is built on the ability to deploy a VM. This job deploys the ARM template, which includes all of the necessary properties to make it work.
The deploymentOutputs: arm output property should be noted. A task has to connect to the deployed VM in the following stage. Returning the DNS name or IP address of this VM through the ARM deployment is a fantastic approach to retrieve it. The pipeline variable created by the deploymentOutputs option may be accessed in other tasks.
– responsibility: [email protected] deployment inputs “Resource Group” is the scope of the project. “ARM” azureResourceManagerConnection “1427e7fb-a488-4ec5-be44-30ac10ca2e95” subscriptionId resourceGroupName: $(azure resource group name) action: “Create Or Update Resource Group” template “East US” location “Linked artifact” is the location. csm deploymentFile: “server.parameters.json” csmParametersFile: “server.parameters.json” “Incremental” deployment mode “arm output” is an output.
“Acceptance” Test by Pester
You should run a “integration” or “acceptance” test once the VM has been deployed to check it was correctly deployed. This PesterRunner job runs Pester and another set of infrastructure-related checks to confirm the VM was successfully installed.
We’re using the ArmDeploymentJsonOutput argument to feed in the value of the ARM deployment’s output. A parameter is defined in the Pester test script file that accepts the value and reads the VM’s DNS hostname.
– responsibility: [email protected] script as input Folder: “[email protected]; @Path=’$(System.DefaultWorkingDirectory)/server.infrastructure.tests.ps1′; {ArmDeploymentJsonOutput=’$(arm output)’}} “$(System.DefaultWorkingDirectory)/server.infrastructure.tests.XML” resultsFile: “$(System.DefaultWorkingDirectory)/server.infra” use True run32Bit: PSCore False
Below are the results of his server.infrastructure.tests. Looks like the ps1 PowerShell script. It’s worth noting that it uses the VM’s DNS hostname to do a basic open port check.
$ArmDeploymentOutput = $ArmDeploymentJsonOutput | convertfrom-json ## $ArmDeploymentOutput = $ArmDeploymentJsonOut Test-Connection -TCPPort 3389 -TargetName $ArmDeploymentOutput.hostname.value -Quiet | should -Be $true -Description ‘Network Connnectivity’ it ‘the VM has RDP/3389 open’
Test Cleanup for “Acceptance”
The sole reason the pipeline built infrastructure was to verify the ARM template’s legitimacy. This infrastructure must be cleaned up since it is just transitory. The pipeline removes the resource group established before, as well as everything in it, in the final PowerShell operation.
– responsibility: [email protected] inputs: inline target type Remove-AzResourceGroup -Force | Get-AzResourceGroup -Name $(azure resource group name)
Publishing by Pester Test
Finally, we’ve arrived at the last set of chores. Publish Test Results is an Azure Pipelines job. This job uses the build agent to read an XML file and provides test results in Azure DevOps. This is a convenient method to examine the results of all tests performed.
– responsibility: [email protected] test inputs “NUnit” test results format $(System.DefaultWorkingDirectory)/server.infrastructure.tests.XML fails true TaskOnFailedTests – responsibility: [email protected] test inputs “NUnit” test results format “$(System.DefaultWorkingDirectory)/server.template.tests.XML” is failing. true TaskOnFailedTests
A pipeline’s Tests segment runs.
The Azure DevOps Pipeline in Action
Finally, we’re ready to test the pipeline’s functionality. Make sure you’re in the ServerAutomationDemo project in the Azure DevOps web UI. After that, go to Pipelines and look for the ServerAutomationDemo pipeline.
Clicking on the three dots on the far right, as seen below, is one method to start the pipeline. Then press the Run pipeline button. This is where the automated fun begins.
Setting up a pipeline
The pipeline will continue to operate and complete each job as directed. As illustrated below, all green check marks for each task accomplished by the job should be visible by the end.
Execution of work tasks with success
Organizing
You should tidy up once you’ve fiddled about with the pipeline and everything else you’ve done here. After all, this was supposed to be an instructional, not a production project!
Some commands to clear up everything produced in this post are listed below. The service principal, Azure AD application, resource group and everything in it, and the Azure DevOps project are all removed with this code.
$spId = ((az ad sp list –all | ConvertFrom-Json) | ? { ‘<https://ServerAutomationDemo>’ -in $_.serviceprincipalnames }).objectId az ad sp delete –id $spId ## Remove the resource group az group delete –name “ServerAutomationDemo” –yes –no-wait ## remove project $projectId = ((az devops project list | convertfrom-json).value | where { $_.name -eq ‘ServerAutomationDemo’ }).id az devops project delete –id $projectId –yes
Summary
This lesson was designed to provide you a glimpse into the process of creating a genuine Azure DevOps infrastructure automation pipeline. Even though there are several more methods to construct pipelines, the abilities you’ve gained in this course should help you in a variety of situations.
Now go out there and start automating more!
The “build and release pipeline in azure devops” is a blog post that explains how to build an Azure DevOps pipeline from soup to nuts. The article starts with the basics of what Azure DevOps is, and then walks through the configuration steps for setting up a new build and release pipeline.
Related Tags
- how to create pipeline in azure devops
- how to clone a pipeline in azure devops
- how to add task in azure devops pipeline
- azure devops build and deploy app using ci and cd pipeline
- azure devops pipeline pdf
