Table of Contents

Provisioning Solutions - Azure DevOps

Provisioning Solutions - Azure DevOps

Provisioning Introduction

Context & Problem

This document describes patterns to automate the deployment of an application to a target environment. The major players in this documentation are (1) the process as a whole (aka provisioning), (2) automation code and (3) application/ infra code to be deployed. Additionally a third aspect is cross functionalities (3) that affect the automation and the application/infra code such as compliance.

Automation code:
The automation code is about automating parts of deployment cycle. In an ideal world this covers the entire code development cycle from opening the first branch to the final deployment. Possible activities in this cycle are:
- quality gates
- build
- deployment
Quality gates refer to testing or approval. Approval might include a manual approval to deploy to sensitive environments like production or enforcing a review before the commit through a pull request. Failing a quality gate should stop the workflow to proceed. Ideally they are maximizing application code coverage and kick in as early as possible. From a timeline perspective major events that are kicking off actions from the automation code are:
- Pull request/ commit (Only triggers quality gates that don’t take too long)
- Build
  
  They can be manually, scheduled (e.g. as part of a nightly build) or automatically started. Additional quality gates ensure code quality.
- Deplyoment to target environment
  
  A typical quality gate for sensible environments such as production are manual approvals.
The major construct is a pipeline that implements a certain activity or a combination. The trigger defines the condition that kick offs a pipeline. Kicking off a pipeline usually includes parameters such as the name of the target environment. The support of different triggers is essental to cover the entire lifecycle. Pipelines can be implemented using a UI driven or programatic approach.

Pipelines are built in a modular way which also adds intermediate steps such as placing the built output in a build artefact repository for later deployment. It also introduces the need to orchestrate them into larger workflows such as creating an entire environment from scratch.

Pipelines must ensure traceability of the performed actions across the entire chain including source code repo, the targeted environment or intermediate stores. This includes a versioning schema for built artefacts. Branches in source repos must be tagged to be able to reconstruct the code behind a versioned artefact.
Application/ infra code:
The base for automation code activities is the code creating or updating the infrastructure and the application on top. Focus in this documentation is the interaction between automation code and application/ infra code. This includes:
- Programming approach (Language or UI) for deploying infrastructure and application code
- Interaction standard problems between automation code and application/ infra code
  
  Problems also depend on the chosen programing approach.
Not relevant are:
- Application code: Structuring in repo, programming language specifics and details of build mechanism
- Specific deployment options of the platform services forming the infrastructure
Provisioning

The process must work in a compliant way end-to-end. Compliance includes the general concern security and to adhere to constraints from the organization’s perspective that need to be enforced.

Provisoning should be subject to monitoring and must be able to adapt to the organizational structure.

Configuration is also a cross concern that affects both sides. From the automation side this includes properties of the environment and settings of the involved automation code. The application specific settings are out of scope in this pattern description. However, provisioning must enable the application code to access settings.

As shown automation does not only include pipelines but also additional configuration settings. Automation code must be able to access the infra/ app code for deployment. This pattern includes guidelines regarding options to store the code and how to structure it in a repository. Application code structure is treated as black box.

In most cases enterprises have already existing technologies in place that might not be based on cloud of a certain provider. Integration with other third party offerings to cover functionally also from elsewhere is also important.

The picture below summarizes the major aspects:

Standard Problems

The following standard problems will be addressed in subsequent paragraphs:

Automation code
Regarding pipelines the folloqing aspects will be detailed:
- Modeling environments
- Pipeline Implementation (Programing approaches, Triggers, Paramterization, Store Code, Quality Gates)
- Orchestration
- Traceability (Versioning, Tagging)
Infra/ Application code (Programming approach, Interaction standard problems)
Provisioning
- Organizational Mapping
- Integration
- Code Repository (Automation/ Infra & App Code)
- Configuration
- Compliance

For the following aspectcs check out the other defined patterns:

Monitoring infrastructure and application code
General guidelines for structuring repositories (e.g. mono vs. multi-repo)
General guidelines for defining landing zones

Provisioning Platforms

Azure

Overview

This chapter lists major features/ concrete services for provisioning of the Azure platform. This architecture pattern builds on the general problem description for monitoring. The picture below summarizes major services and concepts that are discussed in detail in the next chpater.

Automation code

Modelling Environments

Azure provides the following structural elements to model an environment:

Resource groups: Smallest possible container
Subscriptions: One subscription can contain many resource groups. With a subscription discounts for dev/ test environments are possible.
Management groups: One management group can contain many subscriptions. They can be used enforce policies across subscriptions if environments share common characteristics.

An environment can be linked to another environment. Linking an environment to multiple environment is beneficial for addressing cross concerns such as monitoring (Similar to Hub/ Spoke topology in networks).

Pipeline Implementation

The programming approach can be either UI driven or programmatic. Pipeline programming languages such as YAML structure the actions to be performed by the pipeline and provide basic mechansims such as downloading code from the repo, parameter handling, stating triggers and triggering other programming languages. These other languages are then used to setup infrastructure such as terraform or deploying application code.

Azure allows to trigger pipelines upon:

a push to repo
a pull request to repo
a schedule
a pipeline completion

The platform allows to pass parameters by various mechanisms to pipelines(Explicit per user input, programmatically). Parameters can be passed by group identifier or explicitly as key value pairs. Complex structured objects as known from object programming languages are not directly possible (Require parsing of files with object structure). Parametrization might be constrained by the used service in certain areas.

The platform provides support for quality gates as follows:

Static code analysis

Microsoft does not provide own tools for static code analysis but allows integration of others.
Automated tests (Unit, Integration, End-To-End)

Microsoft provides services that include test management e.g. creating test suites with test cases and getting an overview about the results.
Approval

Azure services support approval for a certain environments and enforcing pull requests as quality gates.

The Azure platform provides the following basic options to store automation code:

Services that provide repositories
Integration of various external code repositories

Orchestration

To orchestrate pipelines the two following basic mechanisms can be used:

Implicit Chaining

In that case the complete workflow is not explicitly coded in a dedicated pipeline. Pipelines are chained implicitly by triggering events. The biggest problem with that approach is the missing single pane of control. The current state in the overall workflow is for instance only implicitly given by the currently running pipeline.
Creating a dedicated orchestration pipeline

An additional pipeline triggers in this scenario other pipelines acting as building blocks. Pipelines can run separately (just run the deployment) or as part of a bigger workflow (e.g. create environment from scratch).

Orchestration must take dependencies into account. They might result from the deployed code or the scope of the pipeline (scope is e.g. a single microservice and code includes the libraries neede). Orchestrated pipelines must pass data between them. The recommended method is to use key vault.

Recreation of resources in short intervals can cause pipelines to fail since the previously deleted resource still exists in the background.(Even although soft delete is not applicable). Whether Azure really deleted everything depends on the service. For instance Azure API management seemed to be affected by that problem.

Traceability

Traceability requires an identifier for referencing artefacts. A standard schema is a semantic version. The platform only supports partial support for number generation such as incrementing numbers. Linking the code in the repo to a certain version depends on used repository.

Infrastructure/ Application code

A programming language is either "declarative" or "imperative". Declarative programming languages state the target state and it is the job of the declarative programming language how to get there. The following rules are applied to achieve that:

create a resource if not there
update an existing resource if different properties
delete resource if not there

Imperative programming languages state the how. The internal delta calculation needs to be explicitly programmed here. If possible declarative programming languages are recommended due to automatic delta calculation. Typical case is infrastructure.

solution_provisioning_platforms_azure_dec_opt

Provisioning

Organizational Mapping

The provisioning must match the organizational requirements of your organization. Azure provides services to model sub units within your organization such as departments, projects and teams.

Integration

Platform allows a modular approach to outsource certain functionality to third party software such as code repository. Which parts is service specific.

External tools providing pipelines can be integrated in two conceptual ways:

Trigger automation pipelines from external: This involves the configuration of a CI pipeline in the external tool such as Jenkins and mechanism in the automation service that invokes the CI process when source code is pushed to a repository or a branch.
Run external pipelines from within the platform: In this approach automation reaches out to an external tool to work with the results.

Configuration

Configuration for provisioning is required in various areas:

Environment: E.g. name of resource group per potential target environment
Repository: E.g. relevant repos/ branching
Pipelines: Parameters pipelines run with such as the technical user name or settings required by the built/ deployed code.

Concrete features used for the above three points depend on the used services. A general storage for sensitive data (keys, secrets, certificates) in Azure is always Azure Key Vault.

Compliance

The standard concept for role-based access controls is called RBAC in Azure. It assigns principals (humans or technical accounts) permissions for a certain resource. Regarding provisioning the following users are relevant:

Technical user (service principal) the pipelines are running with
Users for administrating the provisioning service

Azure Active Directory is the central service in Azure that defines and controls all principals (human/ service) per tenant.

Granularity of roles that can be granted depend on the service. The boundaries in which users exist or permissions can be assigned is also service specific.

Provisioning Solutions - Azure DevOps

Overview

The cloud native provisioning service in Azure is Azure DevOps formerly known as Team Foundation Server (TFS). Azure DevOps covers the full CI/ CD requiremenzs including project management. Due to its extensive extension features you can also replace features with other alternatives (see also under variations).

Note: Azure DevOps will be superseeded by GitHub in the long run after Microsoft acquired GitHub. New features will be initially implemented there.

The services that (can) complement Azure DevOps:

Azure Key Vault for storing secrets/ exchange of settings
Azure App Configuration

This service provides settings (key-value pairs) and feature toggles. Native integrations exist for typical application programming languages like .NET/ Java. However native integrations with terraform do not exist and it is also not hardened for sensitive information as key vault. Therefore, it is recommended to use that service as special case for application layer if feature toggles are needed.
Azure AD

Azure Active Directory provides the service principal the pipelines run with.
Monitoring
Azure DevOps generates metrics to check the health pipelines and displays te state in the Azure DevOps portal. However no built-in forwarding to App Insights independent from the deployed application exists. The following options are available:
- Continous monitoring: Monitoring which assumes Web Applications.
- Rest API to read service health information manually as explained here. Check out the pattern monitoring cloudnative how polling and forwarding to azure monitor can be achieved.
- audit streaming is similar to diagnostic settings (In preview as of 21.09.2021). It allows to configure a constant forwarding of telemetry to selected targets as described here.
Structural elements to model environments

The picture illustrates the setup with the major dependencies:

Pattern Details

Geting Started

Many aspects influence the setup of the service. Following a top down approach the following decisions have to be made:

Define landing zone of the service itself in Azure (out of scope)
Organizational mapping

This yields the structural components to host provisioning which will be detailed in the next chapter. It introduces the possible components and guidelines for its structuring.
Modelling other outlined aspects across automation, infra/ app code and provisioning

The structural components are organizations, teams and projects. A team is a unit that supports many team-configurable tools. These tools help you plan and manage work, and make collaboration easier. Every team owns their own backlog, to create a new backlog you create a new team. By configuring teams and backlogs into a hierarchical structure, program owners can more easily track progress across teams, manage portfolios, and generate rollup data.

A project in Azure DevOps contains the following set of features:

Boards and backlogs for agile planning
Pipelines for continuous integration and deployment
Repos

The service comes with hosted git repositories inside that service. You can also use the following external source repositories: Bitbuckt Cloud, GitHub, Any generic git repo, Subversion
Testing
Azure DevOps supports the following testing by defining test suites with test cases:
- Planned manual testing. Manual testing by organizing tests into test plans and test suites by designated testers and test leads.
- User acceptance testing. Testing carried out by designated user acceptance testers to verify the value delivered meets customer requirements, while reusing the test artifacts created by engineering teams.
- Exploratory testing. Testing carried out by development teams, including developers, testers, UX teams, product owners and more, by exploring the software systems without using test plans or test suites.
- Stakeholder feedback. Testing carried out by stakeholders outside the development team, such as users from marketing and sales divisions.
Tests can also be integrated in pipelines. Pipelines support a wide range of frameworks/ libraries.
Each organization contains one or more projects

Your business structure should act as a guide for the number of organizations, projects, and teams that you create in Azure DevOps. Each organization gets its own free tier of services (up to five users for each service type) as follows. You can use all the services, or choose just what you need to complement your existing workflows.

Azure Pipelines: One hosted job with 1,800 minutes per month for CI/CD and one self-hosted job
Azure Boards: Work item tracking and Kanban boards
Azure Repos: for version control and management of source code and artifacts
Azure Artifacts: Package management
Testing: Continuous test integration throughout the project life cycle

Adding multiple projects makes sense in the following cases (see azure projects):

To prohibit or manage access to the information contained within a project to select groups
To support custom work tracking processes for specific business units within your organization
To support entirely separate business units that have their own administrative policies and administrators
To support testing customization activities or adding extensions before rolling out changes to the working project
To support an Open Source Software (OSS) project

Adding teams instead of projects is recommended over projects due to agile culture:

Visibility: It’s much easier to view progress across all teams
Tracking and auditing: It’s easier to link work items and other objects for tracking and auditing purposes
Maintainability: You minimize the maintenance of security groups and process updates.

solution_provisioning_azure_devops_struct

Remaining goals (Automation Code)

This chapter details how the above conceptual features can be achieved with Azure DevOps pipelines.

The pipeline programming approach can be either UI driven or programmatic by using YAML. YAML organizes pipelines into a hierarchy of stages, jobs and tasks. Tasks are the workhorse where activities are implemented. Tasks support scripting languages as stated below. They in turn allow to install additional libraries frameworks from third party providers such as terraform (or you use extensions that give you additional task types). The list below highlights a few YAML points you have to be aware of:

Passing files/ artefacts between jobs/ pipelines

Passing between jobs within the same pipeline requires publishing the files as pipeline artefacts and downloading it afterwards. Passing between syntax requires a different syntax and also requires a version.
Variables

Variables can have different scopes. A special syntax is required to publish them at runtime and to consume them in a different job (requires declaration). (Link). Various predefined exist.
Obtaining client secret

Scripting languages such as terraform might require the client secret for embedded scripting blocks. However, terraform does not provide a way to get it. The only way was to include an AzureCLI scripting task. Setting the argument "addSpnToEnvironment" to true makes the value for scripting languages as environment variable. A script can then publish the variable so that the value is available in the YAML pipeline.

Pipelines that shall be triggered by pushing to the repo state in the trigger element the details like branch when they shall run. The example below shows a scheduled trigger:

# Disable all other triggers
pr: none
trigger: none

# Define schedule
schedules:
# Note: Azure DevOps only understands the limited part of the cron
#       expression below. See this link for further details:
#       https://docs.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml
# Note: With DevOps organization setting of UTC+1 Berlin,...
#       for a given hour x you have to specify x-2 e.g. 16:00 will be
#       started 18:00 o'clock
- cron: "30 5 * * MON,TUE,WED,THU,FRI"
  displayName: Business daily morning creation
  always: true # also run if no code changes
  branches:
    include:
    - 'refs/heads/master'

Pull request (PR) triggers cause a pipeline to run whenever a pull request is opened with one of the specified target branches, or when changes are pushed to such a pull request. In Azure Repos Git, this functionality is implemented using branch policies. To enable pull request validation in Azure Git Repos, navigate to the branch policies for the desired branch, and configure the Build validation policy for that branch. For more information, see Configure branch policies. Draft pull requests do not trigger a pipeline even if you configure a branch policy. Building pull requests from Azure Repos forks is no different from building pull requests within the same repository or project. You can create forks only within the same organization that your project is part of (see PR triggers). To trigger a pipeline upon the completion of another pipeline, specify the triggering pipeline as a pipeline resource. The following example has two pipelines - app-ci (the pipeline defined by the YAML snippet), and security-lib-ci (the triggering pipeline referenced by the pipeline resource). We want the app-ci pipeline to run automatically every time a new version of security-lib-ci is built.

# this is being defined in app-ci pipeline
resources:
  pipelines:
  - pipeline: securitylib   # Name of the pipeline resource
    source: security-lib-ci # Name of the pipeline referenced by the pipeline resource
    project: FabrikamProject # Required only if the source pipeline is in another project
    trigger: true # Run app-ci pipeline when any run of security-lib-ci completes

Implicit Chaining for orchestration is possible by using trigger condition. Calling pipelines explicitly is so far only possible with scripting.

solution_provisioning_azure_devops_orch

As part of the configuration Azure DevOps provides the possibility to provide various settings that are used for development such as enforcing pull requests instead of direct pushes to the repo. The major configuration mechanisms in YAML are variables, parameters and variable groups. Variable groups bundle multiple settings as key value pairs. Parameters are not possible in a variable section (Dynamic inclusion of variable groups is possible via file switching). If they are declared on top level they have to be passed when the pipeline is called programmatically or manually by the user.

Quality gates can be enforced as follows:

Static code analysis:

Various tool support exists depending on the programming language.

Automated tests (Unit, Integration, End-To-End)

Tests can be included in pipelines via additional libraries and additional previous installment through scripting. The task below uses an Azure CLI task to run tests for terraform:

  - task: AzureCLI@2
    displayName: Run terratest
    inputs:
      azureSubscription: ${{parameters.svcConn}}
      scriptType: bash
      scriptLocation: 'inlineScript'
      addSpnToEnvironment: true
      inlineScript: |
        # Expose required settings as environment variables
        # ARM_XXX initialized by task due to addSpnToEnvironment = true
        subsid=`az account show --query id -o tsv`
        echo "client_id:"$servicePrincipalId
        echo "client_secret:"$servicePrincipalKey
        echo "subscription_id:"$subsid
        echo "tenant_id:"$tenantId
        export ARM_SUBSCRIPTION_ID=$subsid
        export ARM_CLIENT_ID=$servicePrincipalId
        export ARM_CLIENT_SECRET=$servicePrincipalKey
        export ARM_TENANT_ID=$tenantId
        # Backend settings
        export storage_account_name=${{parameters.bkStname}}
        export container_name=${{parameters.bkCntName}}
        export key=${{parameters.bkRmKeyName}}
        # Other settings
        export resource_group_name=${{parameters.rgName}}
        # Switch to directory with tests
        pwd
        cd test
        # Testfile must end with "<your name>_test.go"
        go test -v my_test.go

Manual approval e.g. for production

YAML allows deployments to named environments. Approvers can then be defined for the named environments in the portal what causes the deployment pipeline to wait. However Approval must be done multiple times if you have multiple deplyoment blocks. The example below shows a deployment to the environment "env-demo":

jobs:
- deployment:
  displayName: run deploy template
  pool:
    vmImage: 'ubuntu-latest'
environment: env-demo
  strategy:
    runOnce:
      deploy:
        steps:
        # - 1. Download artefact
        - task: DownloadPipelineArtifact@2
          displayName: Get artefact
          inputs:
            downloadPath: '$(build.artifactstagingdirectory)'
            artifact: ${{parameters.pipelineArtifactName}}

Remaining goals (Provisioning)

Configuration settings can be broken down into key value pairs. As already stated key vault is the recommended place for storage. Azure App Configuration and variable groups can reference values in Key Vault. Key Value pairs must be selected in YAML based on the target environment. Switching based on the parameter value is possible by constructing filenames based on the parameter value. The resolved filenam contains then the variable group or the key value pairs. as shown below:

(1) Main pipeline that requires switching

...
# Switch in the pipeline which is implemented in a shared repository
variables:
- template: ./pipelines/configurations/vars-env-single-template.yaml@repo-shared
  parameters: ${{parameters.envName}}
...

(2) Shared: Switch to correct configuration file

...
parameters:
- name: envName
  displayName: name of environment
  type: string

# Load filename with resolved parameter value
variables:
- template: vars-env-def-${{parameters.envName}}-template.yaml

(3) Shared: Configuration file vars-env-def-dev1.yaml

variables:
  envNamePPE1MainScriptLocation: app/dev
  envNamePPE1SvcLevel: Full
  envNamePPE1BranchName: dev
  envNamePPE1KvEnvName: $(envNameCRST)1

Azure DevOps can integrate with various external tools. Pipelines can be called from external and allow calling external tools. Various third party tools can be manually installed or used via extensions.

For compliance Azure DevOps provides various settings inside Azure DevOps itself and via Azure Active Directory. Portal access to Boards, Repos, Pipelines, Artifacts and Test Plans can be controlled through Azure DevOps project settings (Link). Azure DevOps supports the following autthentication mechanisms to connect to services and resources in your organization (Link):

OAuth to generate tokens for accessing REST APIs for Azure DevOps. The Organizations and Profiles APIs support only OAuth.
SSH authentication to generate encryption keys for using Linux, macOS, and Windows running Git for Windows, but you can’t use Git credential managers or personal access tokens (PATs) for HTTPS authentication.
Personal access token (PAT) to generate tokens for:
- Accessing specific resources or activities, like builds or work items
- Clients like Xcode and NuGet that require usernames and passwords as basic credentials and don’t support Microsoft account and Azure Active Directory features like multi-factor authentication
- Accessing REST APIs for Azure DevOps

User permissions for team members are split in access levels and project permissions inside Azure DevOps. The Basic and higher access levels support full access to all Azure Boards features. Stakeholder access provides partial support to select features, allowing users to view and modify work items without having access to all other features. Additional restrictions are possible by Azure Active Directory settings using conditional access policies and MFA. Azure DevOps honors all conditional access policies 100% for our Web flows. For third-party client flow, like using a PAT with git.exe, IP fencing policies are supported only (no support for MFA policies). Permissions to work with repositories can be set under project’s repositories settings which also allows to disable forks. Many forks makes it hard to keep the overview and forking allows to download code into someones private account. Azure DevOps supports creating branch protection policies, which protect the code committed to the main branches (project settings ⇒ repo ⇒ branch policies). Compliance affects dealing with sensitive settings. As already stated key vault is the standard service for storing them at runtime. Exports from key vault can only be decrypted in a key vault instance. Hence, secrets can be stored in a repository in a safe way without having to store the values in plain. Using them later should be done in a safe way. This includes publishing them in a safe way and passing them from YAML to terraform by avoiding log output in plain text. Avoiding log output passing them as environment variables/ files.

The following repository structure shows a conceptual breakdown that covers most aspects:

1. Infra
1.1. Infrastructure
1.1.1. Other landing zones

Represents other areas with shared functionality that are required. Examples are environments for monitoring, the environment containing Azure DevOps, Key Vault settings etc.
1.1.2. App Environments

Represents the environments where application is deployed to.
1.1.2.1. Envs

This level contains all infrastructure code for seting up en environment. The split between dev and non-dev leverages cost savings for less performant dev environments e.g. by picking cheaper service configurations or totally different Azure services.
1.1.2.1.1. Dev
1.1.2.1.2. Non-Dev
1.1.2.1.3. Modules

Factored out modules for shared reuse. One example is a central module to generate the name for a given module.
1.1.2.2. Envs-Mgmt

Captures aspects assumed by the chosen programming language such as terraform for managing an environment. This includes for instance the backend creation code.
1.2. Pipelines

Pipelines for automating infrastrcuture deployment.
2. App
2.1. Application (Black Box)
2.2. Pipelines

Pipelines for automating app code deployment.
3. Shared

Captures shared aspects between infrastructure and application code such as publishing key vault secrets for a pipeline or triggering another pipeline.

Variations

The following features of Azure DevOps can be replaced with alternatives:

Repos
Boards for project management
Azure Artefacts

The following alternatives on service level exist:

Azure Lab Services for Dev/ Test scenarios
Kubernetes
- Azure DevSpaces (Deprecated) in favor of “Bridge-to-kubernetes” for Dev/ Test scenarios
- Bridge-to-Kubernetes

When to use

Azure DevOps makes sense if you want to provision to Azure due to its tight integration into the Azure platform. The following circumstances also speak for Azure DevOps:

You need a mature service on enterprise level
You don’t need cloud agnostic pipelines e.g. due to a multi-cloud scenario
Your code repository is not GitHub and Azure DevOps provides integrations for it (GitHub Actions are a better alternative if your code repos are already on GitHub.)
You need a full blown provisioning service and plan to use the integrated repos that come with Azure DevOps

Credits

_{Last updated 2023-11-20 10:40:32 UTC}