Azure DevOps is a set of tools for managing software development and deployment. The following article explains what variables are in Azure DevOps, how to create them and using them.
The “azure devops predefined variables” is a list of predefined variables that can be used in Azure DevOps. The list includes the name, type, and description.
If you’ve used Azure DevOps Pipeline to create a CI/CD pipeline, you’ve almost certainly encountered circumstances where dynamically manipulating configuration settings in builds and releases is required. You require variables to provide a build version to a PowerShell script, send dynamic parameters to build jobs, and use strings across builds and releases.
If you’ve ever asked yourself the following questions:
- In a PowerShell script, how can I leverage Azure DevOps build Variables in Pipelines?
- What’s the best way to communicate variables across builds and releases?
- What’s the difference between preset, user-defined, and Secret Parameters?
- What is the purpose of Groups That Change?
…then you’re in for a treat! We’ll address all of these questions and more in this post.
You’ll know how Azure DevOps build variables function in Azure Pipelines by the conclusion of this post!
What are the variables in the Azure DevOps Pipeline?
What are variables and how can they help you develop and automate efficient build and release pipelines before we get into the details?
Variables let you pass pieces of data between different areas of your pipeline. Variables are useful for storing text and numbers that change as the pipeline progresses. Rather of hard-coding values in scripts and YAML specifications, you can set and read variables practically anywhere in a pipeline.
This article will solely discuss YAML pipelines. There will be no discussion of legacy traditional pipelines. With a few small exceptions, the online UI will not teach you how to operate with variables. We’ll be using YAML exclusively.
At runtime, variables are referenced and some are declared (see Variables that are defined by the user). Various processes handle these variables and send their values to other parts of the system when a pipeline starts a task. This approach allows you to perform pipeline tasks dynamically without having to worry about constantly modifying build definitions and scripts.
Don’t worry if you don’t understand the notion of variables yet. The remainder of this post will instruct you on anything you need.
Environments that Change
Before jumping into the variables themselves, it’s first important to cover Azure pipeline Environments that Change. You’ll see various references to this term throughout the article.
Within a pipeline, there are two areas where you may interact with variables that are referred to as environments. You may deal with variables in The Pipeline Situation of a YAML build specification or in Scripting Environment of a script performed through a task.
The Pipeline Situation
When you’re defining or reading build variables from within a YAML build definition, this is called The Pipeline Situation. For example, below you can see the variables section defined in a YAML build definition setting a variable called foo to bar. In this context, the variable is being defined within The Pipeline Situation
Scripting Environment
Variables may also be manipulated using code specified in the YAML declaration or scripts. If you don’t already have a script, you may use the YAML definition to define and read variables, as seen below. Later on, you’ll learn the syntax for working with these variables in this context.
steps: – bash: ## In this section, you’ll set and obtain variables.
You could alternatively stay within Scripting Environment by adding this same syntax into a Bash script and executing it. This is the same general concept.
Environmental Factors
Within Scripting Environment, when a pipeline variable is made available, it’s done so by creating an environment variable. These Environmental Factors can then be accessed via the language of choice’s typical methods.
Variables in Pipelines exposed as Environmental Factors will always be upper-cased and any dots replaced with underscores. For example, you’ll see below how each scripting language can access the foo pipeline variable as shown below.
- FOO%% batch
- $env:FOO in PowerShell
- $FOO bash script
“Execution Phases” of Pipeline
When a pipeline “runs,” it accomplishes more than that. When a pipeline operates, it goes through many phases, much like the stages it includes. I’m calling this “execution stages” since there is no official name in the Microsoft documentation.
When a pipeline is started, it passes through three stages: queue, compilation, and execution. It’s crucial to grasp these contexts since you’ll notice references to these phrases throughout the Microsoft documents.
Waiting Time
The first phase a pipeline goes through when triggered in queued. In this phase, the pipeline hasn’t started yet but is queued up and ready to go when the agent is available. When defining variables, you can set them to be made available at Waiting Time by not defining them in the YAML file.
You’ll be able to define variables at Waiting Time when the pipeline is initially queued as shown below. When this variable is added, it will then be made available a global variable in the pipeline and can be overridden by the same variable name in the YAML file.
Option to run Azure DevOps Pipeline
Compile
Finally, a pipeline is in the compilation “phase” when it analyzes a YAML file and moves on to the stages that need script execution. The agent is running the code described in the script steps in this context.
Runtime
The next stage is runtime. This is the stage in which the YAML file is parsed. Each stage, task, and step are processed during this phase, but no scripts are performed.
Expansion Variable
Another important topic to understand is Expansion Variable. Expansion Variable, in simplest terms, is when the variable returns a static value. The variable expands to reveal the value it’s holding. This is done for you with no effort when you read a variable but that expansion can be done at different times during a pipeline run that might trip you up.
This concept of Expansion Variable and compile vs. runtime will come up a lot when you get into understanding Syntax Variable.
As you learned above, the pipeline covers different “phases” when it runs. You’ll need to be aware of these phases most likely when troubleshooting Expansion Variable.
You can see an example below. The concept of these “phases” is closely related to Environments that Change.
steps: # FOO is set to be “some value” in the script and subsequent ones – bash: | FOO=”runtime value” echo “##vso[task.setvariable variable=FOO]” echo “##vso[task.setvariable variable=FOO]” $FOO” # The pipeline extends the variable using the $() syntax at “compile time” – bash: | echo “$(FOO)”” # When a script uses an environment variable in the “runtime” context, bash extends the variable – bash: | echo “$FOO”
Variables are enlarged once at the start of the pipeline run and again at the start of each phase. A basic illustration of this behavior is shown below.
jobs: – task: construct variables – foo: bar steps: # This will be bar echo ‘##vso[task.setvariable variable=foo]’ $(foo) expands before the step, so this will also be bar. baz’ echo $(foo) # This will also be bar. echo $ – bash (foo) # This is going to be baz. Just before the step, the variable is enlarged.
Syntax Variable
As you’ve seen, there are two “environments” in which you may set or read variables: the pipeline and script environments. The way you refer to variables varies depending on the context. There are some subtleties to be aware of.
Variables in Pipelines
Variables in Pipelines are referenced in the YAML build definitions and can be referenced via three different syntax methods – macro, template expression and runtime expression.
Syntax in Macro
The most common syntax you’ll find is Syntax in Macro. Syntax in Macro references a value for a variable in the form of $(foo). The parentheses represent an expression that is evaluated at runtime.
When Azure Pipelines processes a variable defined as a macro expression, it will replace the expression with the contents of the variable. When defining variables with Syntax in Macro, they follow the pattern <variable name>: $(<variable value>) eg. foo: $(bar).
If you attempt to reference a variable with Syntax in Macro and a value does not exist, the variable will simply not exist. This behavior differs a bit between syntax types.
Syntax for Template Expressions
Another kind of Syntax Variable is called template expression. Defining Variables in Pipelines this way takes the form of ${{ variables.foo }} : ${{ variables.bar }}. As you can see, it’s a bit more long form than Syntax in Macro.
Syntax for Template Expressions has an added feature too. Using this syntax, you can also expand template parameters. If a variable defined with Syntax for Template Expressions is referenced, the pipeline will return an empty string vs. a null value with Syntax in Macro.
Variables in template expressions are processed at build time and then overridden at runtime (if specified).
Syntax of Runtime Expressions
As the syntax type, suggested runtime expression variables are expanded only at runtime. These types of variables are represented via the format $[variables.foo]. Like Syntax for Template Expressions variables, these types of variables will return an empty string if not replaced.
Like Syntax in Macro, Syntax of Runtime Expressions requires the variable name on the left side of the definition such as foo: $[variables.bar].
Variables in Scripts
Working with variables inside of scripts is a bit different than Variables in Pipelines. Defining and reference Variables in Pipelines exposed in task scripts can be done one of two ways; with logging command syntax or Environmental Factors.
Commands for Logging
One way to define and reference Variables in Pipelines in scripts is to use logging command syntax. This syntax is a bit convoluted but you’ll learn is necessary in certain situations. Variables are defined in this manner must be defined as a string in the script.
Using logging command syntax, creating a variable named foo with the value bar would look like this.
“##vso[task.setvariable variable=foo;]bar” ##vso[task.setvariable variable=foo
I could not find a way to get the value of variables using Commands for Logging. If this exists, let me know!
Environmental Factors
When Variables in Pipelines are turned into Environmental Factors in scripts, the variable names are changed slightly. You’ll find that variable names become uppercase and periods turn into underscores. You’ll find many predefined or system variables have dots in them.
For example, if a pipeline variable called [foo.bar](<http://foo.bar>) was defined, you’d reference that variable via the script’s native environment variable reference method such as $env:FOO_BAR in PowerShell or $FOO_BAR in Bash.
We covered more of Environmental Factors in Scripting Environment section above.
Various Scopes
A pipeline has various stages, tasks and jobs running. Many areas have predefined Various Scopess. A scope is namespace where when a variable is defined, its value can be referenced.
There are essentially three different Various Scopess in a hierarchy. They are variables defined at:
- variables are made accessible to all tasks in the pipeline at the root level
- variables accessible to a given stage at the stage level
- factors accessible to a given job at the job level
Variables declared at “lower” levels, such as a job, override variables defined at the stage and root level, for example. Variables declared at the stage level take precedence over variables defined at the “root” level, whereas variables specified at the job level take precedence.
Below is an example YAML build specification that shows how each scope is utilized.
# This is a global variable accessible to all stages and jobs stages: global variable: value – stage: Build variables: stage variable1: value3 # This is only accessible in BuildJob steps: job variable1: value1 – bash: echo $(stage variable1) ## works – bash: echo $(global variable) ## works – bash: echo $(job variable1) ## works – bash: echo $(global variable) ## works
Precendence that varies
You can come into a scenario where the same variable is set in many scopes. When this occurs, the value of that variable will be rewritten in a certain order, with the nearest “action” taking priority.
The sequence in which the variables will be rewritten is shown below, beginning with a variable set inside a task. This will be given the highest priority.
- At the job level, a variable is set (set in the YAML file)
- At the stage level, a variable is set (set in the YAML file)
- Set at the pipeline level (global) variable (set in the YAML file)
- Variable set at Waiting Time
- Pipeline variable specified in the UI of the Pipeline settings
Take a look at the YAML definition below, for example. The same variable is set in several different sections in this example, but it eventually ends up with the value stated in the task. The value of the variable is rewritten with each step, and the pipeline begins to work.
variables: ‘global variable’ is foo. stages: – stage: construct variables – value: ‘defined at stage level’ – name: foo jobs: – assignment: compile variables – value: ‘defined at job level’ – name: foo steps: – bash: echo $(foo) # ‘Defined at job level’
Types of Variables
In this article, you’ve learned about variables, how they work, the settings in which they might be used, and more. But we haven’t mentioned that not all variables are the same. Some variables are present when a pipeline begins and cannot be modified, whilst others may be added, updated, or removed at any time.
Predefined or system variables, Variables that are defined by the user, Variables for Output, and Secret Parameters are the four categories of variables. Let’s look at each of them in detail and learn about each kind of variable.
Variables that are predefined
Within all builds and releases, you’ll find many different variables that exist by default. These variables are called predefined or system variables. Variables that are predefined are all read-only and, like other types of variables, represent simple strings and numbers.
The software agent that does the build, jobs that are fired up when a deployment starts, and other information all make up an Azure pipeline. Predefined or system variables are loosely divided into five groups to reflect all of these areas:
- Agent
- Build
- Pipeline
- Job of deployment
- System
There are dozens of variables spread across each of these five categories. You’re not going to learn about all of them in this article. If you’d like a list of all Variables that are predefined, take a look at the Microsoft documentation.
Variables that are defined by the user
When you create a variable in a YAML definition or via a script, you’re creating a user-defined variable. Variables that are defined by the user are simply all of the variables you, the user, define and use in a pipeline. You can use just about any name you’d like for these variables with a few exceptions.
Endpoint, input, secret, and securefile are all variables that cannot be defined. These labels are off-limits because they are case-insensitive and designated for system use.
In addition, any variables you create must only include letters, integers, dots, or underscores. Your YAML build specification will fail if you try to declare a variable that does not match this syntax.
Variables for Output
A build definition contains one or more tasks. Sometimes a task sends a variable out to be made available to downstream steps and jobs within the same stage. These types of variables are called Variables for Output.
Variables for Output are used to share information between components of the pipeline. For example, if one task queries a value from a database and subsequent tasks need the result returned, an output variable can be used. You then don’t have to query the database every time. Instead, you can simply reference the variable.
Note: Variables for Output are scoped to a specific stage. Do not expect an output variable to be made available in your “build” stage and also in your “testing” stage, for example.
Secret Parameters
The final type of variable is the secret variable. Technically, this isn’t it’s own independent type because it can be a system or user-defined variable. But Secret Parameters need to be in their own category because they are treated differently than other variables.
A secret variable is a standard variable that’s encrypted. Secret Parameters typically contain sensitive information like API keys, passwords, etc. These variables are encrypted at rest with a 2048-bit RSA key and are available on the agent for all tasks and scripts to use.
– Do NOT define Secret Parameters inside of your YAML files – Do NOT return secrets as Variables for Output or logging information
Secret Parameters should be defined in the pipeline editor. This scopes Secret Parameters at the global level thus making them available to tasks in the pipeline.
In the logs, secret values are partially concealed. This is why they should never be included in a YAML file. You should also avoid keeping any “organized” information a secret. If “foo”: “bar” is specified as a secret, for example, bar will not be masked from the logs.
Secrets are not automatically decrypted and mapped to Environmental Factors. If you define a secret variable, don’t expect it to be available via $env:FOO in a PowerShell script, for example.
Groups That Change
Finally, we come to Groups That Change. Groups That Change, as you might expect, are “groups” of variables that can be referenced as one. The primary purpose of a variable group is to store values that you want to make available across multiple pipelines.
Unlike variables, Groups That Change are not defined in the YAML file. Instead, they are defined in the Library page under Pipelines in the UI.
Use a variable group to store values that you want to control and make available across multiple pipelines. You can also use Groups That Change to store secrets and other values that might need to be passed into a YAML pipeline. Groups That Change are defined and managed in the Library page under Pipelines as shown below.
Viewing Groups That Change
Once the variable group is created in the pipeline library, you may access it in the YAML file using the syntax below.
variables: – first group
Groups That Change are not, by default, available to all pipelines. This setting is made available when creating the group. Pipelines must be authorized to use a variable group.
Once a variable group has been made accessible in the YAML file, you may access the variables inside it just like any other variable. When referring variables in the group, the name of the variable group is not utilized.
For example, if you created a variable group1 that included a variable named foo, you would refer to the foo variable like any other eg. $ (foo).
Secret Parameters defined in a variable group cannot be accessed directly via scripts. Instead, they must be passed as arguments to the task.
If a variable inside a variable group is changed, that change will be made accessible to any pipelines that are permitted to utilize that group.
Summary
Azure Pipelines variables should now be second nature to you. In this article, you learnt just about every notion there is regarding variables! Now go ahead and apply everything you’ve learned to your Azure DevOps Pipelines and automate everything!
Azure DevOps variables are a great way to capture the output of commands and analyze it. This is especially useful for debugging. The “azure devops output variables” command will print out all of your variables in a neat table format.
Related Tags
- azure devops yaml variables
- azure devops release variables
- azure pipelines variables
- azure devops concatenate variables
- pipeline variables
