Ubuntu | Tom's Notes

Using a self-hosted runner with GitHub Actions

By Tomica Kaniski in Other 17/10/2022 0 Comment

As I was going through the excellent short course called Azure Infrastructure as Code with GitHub (by fellow MVP, Barbara Forbes), a thought appeared – what do I need to do to use my custom runner machine inside a pipeline for… I don’t know… security/privacy concerns, isolation, special requirements, different OS, control, price… or just to complicate things a bit?

Of course, GitHub supports this and it’s called a self-hosted runner.

So, what do I need to do to use this self-hosted runner with my GitHub Actions?

It’s relatively simple – there is an application package, which will be installed on your runner machine, and which will listen for and eventually do all the work defined in your workflow!

But first, let’s introduce my environment.

I have a simple GitHub Action (workflow), which creates a simple storage account on my Azure environment (there is actually no need to convert Bicep to ARM before deployment, but it seemed cool 😀). It’s currently using the „ubuntu-latest“ runner, provided by GitHub… which has also all the needed components inside (like Azure CLI, Azure PowerShell, …).

And it works fine. When there is a push to my GitHub repository, GitHub Actions starts and does what is needed on my Azure environment via this workflow:

name: deploy-to-azure
on:
  push:
    branches: [ "main" ]
  workflow_dispatch:

env:
  resourceGroupName: SimpleAzureRG

jobs:
  deploy-to-azure:
    runs-on: ubuntu-latest

    steps:
      - name: "Checkout the code from GitHub"
        uses: actions/checkout@v3

      - name: "Build Bicep code into ARM (JSON)"
        uses: Azure/cli@v1
        with:
          inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

      - name: "Login to Azure"
        uses: Azure/login@v1.4.6
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true

      - name: "Create the resource group, if needed"
        uses: Azure/powershell@v1
        with:
          inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force
          azPSVersion: latest

      - name: "Deploy the ARM template"
        uses: Azure/arm-deploy@v1.0.8
        with:
          scope: resourcegroup
          resourceGroupName: ${{ env.resourceGroupName }}
          template: main.json
          parameters: storageAccountPrefix=simplegh
          deploymentName: "gh${{ github.run_id }}"

on:

push:

branches: [ "main" ]

workflow_dispatch:

env:

resourceGroupName: SimpleAzureRG

jobs:

deploy-to-azure:

runs-on: ubuntu-latest

steps:

- name: "Checkout the code from GitHub"

uses: actions/checkout@v3

- name: "Build Bicep code into ARM (JSON)"

uses: Azure/cli@v1

with:

inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

- name: "Login to Azure"

uses: Azure/[email protected]

with:

creds: ${{ secrets.AZURE_CREDENTIALS }}

enable-AzPSSession: true

- name: "Create the resource group, if needed"

uses: Azure/powershell@v1

with:

inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force

azPSVersion: latest

- name: "Deploy the ARM template"

uses: Azure/[email protected]

with:

scope: resourcegroup

resourceGroupName: ${{ env.resourceGroupName }}

parameters: storageAccountPrefix=simplegh

deploymentName: "gh${{ github.run_id }}"

And the mighty Bicep file (😀) it’s using for the deployment is:

@maxLength(11)
param storageAccountPrefix string = 'bicep'
param location string = resourceGroup().location

var sta = '${storageAccountPrefix}${uniqueString(subscription().id)}'

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-02-01' = {
  name: sta
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
}

@maxLength(11)

param storageAccountPrefix string = 'bicep'

param location string = resourceGroup().location

var sta = '${storageAccountPrefix}${uniqueString(subscription().id)}'

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-02-01' = {

location: location

kind: 'StorageV2'

sku: {

}

Of course, this runs just fine on a standard (hosted) runner:

To run this workflow (successfully) not that much is needed.

First, I’ve created a new virtual machine (I’ll use a simple Ubuntu Hyper-V VM, no autoscaling, no… nothing) called hermes (god of speed 😀), with freshly installed Ubuntu 22.04.1-LTS (minimized).

After that, I went to the Settings of my GitHub repository and got the download and install scripts for the x64 Linux runner:

### get and install the runner application (script provided by GitHub)
## download the runner application
# create a runner folder
mkdir actions-runner && cd actions-runner

# download the latest runner package
curl -o actions-runner-linux-x64-2.298.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.298.2/actions-runner-linux-x64-2.298.2.tar.gz

# (optional) validate the hash
echo "0bfd792196ce0ec6f1c65d2a9ad00215b2926ef2c416b8d97615265194477117  actions-runner-linux-x64-2.298.2.tar.gz" | shasum -a 256 -c

# extract the installer
tar xzf ./actions-runner-linux-x64-2.298.2.tar.gz


## configure the runner application
# create the runner and start the configuration experience
./config.sh --url https://github.com/TomicaKaniski/AzureIaCGH --token "<<--TOKEN_GOES_HERE!-->>"

# last step, run it! (we will actually run it via crontab at machine restart!)
# ./run.sh

### get and install the runner application (script provided by GitHub)

## download the runner application

# create a runner folder

mkdir actions-runner && cd actions-runner

# download the latest runner package

curl -o actions-runner-linux-x64-2.298.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.298.2/actions-runner-linux-x64-2.298.2.tar.gz

# (optional) validate the hash

echo "0bfd792196ce0ec6f1c65d2a9ad00215b2926ef2c416b8d97615265194477117 actions-runner-linux-x64-2.298.2.tar.gz" | shasum -a 256 -c

# extract the installer

tar xzf ./actions-runner-linux-x64-2.298.2.tar.gz

## configure the runner application

# create the runner and start the configuration experience

./config.sh --url https://github.com/TomicaKaniski/AzureIaCGH --token "<<--TOKEN_GOES_HERE!-->>"

# last step, run it! (we will actually run it via crontab at machine restart!)

# ./run.sh

As you can see, I’ll be using crontab later to automatically (re)start my self-hosted runner.

If everything went well, you should see your runner “up and running” (😀) in the GitHub portal:

Next, I’ll use the following script to install all prerequisites for my workflow (like Azure CLI, Azure PowerShell, etc. – it really depends on your workflow and things you use):

### install (my) workflow prerequisites on self-runner machine
sudo su -

## install Azure CLI (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)
curl -sL https://aka.ms/InstallAzureCLIDeb | bash

## install PowerShell  (https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.2#installation-via-direct-download)
wget -q "https://github.com/PowerShell/PowerShell/releases/download/v7.2.6/powershell-lts_7.2.6-1.deb_amd64.deb" && dpkg -i powershell-lts_7.2.6-1.deb_amd64.deb

## install Bicep
az bicep install

## install other useful things
apt install apt-utils cron vim -y

## install Docker (https://cloudcone.com/docs/article/how-to-install-docker-on-ubuntu-22-04-20-04/)
# install prerequisites
apt install apt-transport-https curl gnupg-agent ca-certificates software-properties-common -y

# add GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add repository
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" -y

# install docker
apt install docker-ce docker-ce-cli containerd.io -y

# create docker group
newgrp docker
exit

# add current user (tomica) to docker group
sudo usermod -aG docker $USER

# configure Docker autostart
sudo systemctl enable docker
sudo systemctl restart docker
sudo systemctl status docker

## install Az module for PowerShell (https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.0.0)
pwsh
Install-Module Az -Repository PSGallery -Confirm:$false -Force
exit


### configure ./run.sh (GitHub runner application) to start automatically
crontab -e

# add the following line (uncommented, of course; and check if the path is right!):
# @reboot /home/tomica/actions-runner/run.sh

sudo reboot

### install (my) workflow prerequisites on self-runner machine

sudo su -

## install Azure CLI (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)

curl -sL https://aka.ms/InstallAzureCLIDeb | bash

## install PowerShell (https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.2#installation-via-direct-download)

wget -q "https://github.com/PowerShell/PowerShell/releases/download/v7.2.6/powershell-lts_7.2.6-1.deb_amd64.deb" && dpkg -i powershell-lts_7.2.6-1.deb_amd64.deb

## install Bicep

az bicep install

## install other useful things

apt install apt-utils cron vim -y

## install Docker (https://cloudcone.com/docs/article/how-to-install-docker-on-ubuntu-22-04-20-04/)

# install prerequisites

apt install apt-transport-https curl gnupg-agent ca-certificates software-properties-common -y

# add GPG key

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add repository

add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" -y

# install docker

apt install docker-ce docker-ce-cli containerd.io -y

# create docker group

newgrp docker

exit

# add current user (tomica) to docker group

sudo usermod -aG docker $USER

# configure Docker autostart

sudo systemctl enable docker

sudo systemctl restart docker

sudo systemctl status docker

## install Az module for PowerShell (https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.0.0)

pwsh

Install-Module Az -Repository PSGallery -Confirm:$false -Force

exit

### configure ./run.sh (GitHub runner application) to start automatically

crontab -e

# add the following line (uncommented, of course; and check if the path is right!):

# @reboot /home/tomica/actions-runner/run.sh

sudo reboot

Once this is done, my self-hosted runner hermes should be ready to run the workflow.

To try this, I need to make a slight update to my workflow file – line 12 inside the job configuration should be updated from “runs-on: ubuntu-latest” to “runs-on: self-hosted“.

So, my workflow YAML file now looks like this:

name: deploy-to-azure
on:
  push:
    branches: [ "main" ]
  workflow_dispatch:

env:
  resourceGroupName: SimpleAzureRG

jobs:
  deploy-to-azure:
    runs-on: self-hosted

    steps:
      - name: "Checkout the code from GitHub"
        uses: actions/checkout@v3

      - name: "Build Bicep code into ARM (JSON)"
        uses: Azure/cli@v1
        with:
          inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

      - name: "Login to Azure"
        uses: Azure/login@v1.4.6
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true

      - name: "Create the resource group, if needed"
        uses: Azure/powershell@v1
        with:
          inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force
          azPSVersion: latest

      - name: "Deploy the ARM template"
        uses: Azure/arm-deploy@v1.0.8
        with:
          scope: resourcegroup
          resourceGroupName: ${{ env.resourceGroupName }}
          template: main.json
          parameters: storageAccountPrefix=simplegh
          deploymentName: "gh${{ github.run_id }}"

on:

push:

branches: [ "main" ]

workflow_dispatch:

env:

resourceGroupName: SimpleAzureRG

jobs:

deploy-to-azure:

runs-on: self-hosted

steps:

- name: "Checkout the code from GitHub"

uses: actions/checkout@v3

- name: "Build Bicep code into ARM (JSON)"

uses: Azure/cli@v1

with:

inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

- name: "Login to Azure"

uses: Azure/[email protected]

with:

creds: ${{ secrets.AZURE_CREDENTIALS }}

enable-AzPSSession: true

- name: "Create the resource group, if needed"

uses: Azure/powershell@v1

with:

inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force

azPSVersion: latest

- name: "Deploy the ARM template"

uses: Azure/[email protected]

with:

scope: resourcegroup

resourceGroupName: ${{ env.resourceGroupName }}

parameters: storageAccountPrefix=simplegh

deploymentName: "gh${{ github.run_id }}"

And once I push the configuration to my GitHub, my workflow automatically starts and runs on hermes, my self-hosted runner:

If we prepared our runner right, all is good! 😊

Of course, our resources are deployed successfully:

So, this is how you can use your own, self-hosted runner, to execute your GitHub Actions (workflows).

Cheers!

UniFi Network Application on Ubuntu Server on Raspberry Pi 3 (arm64)

By Tomica Kaniski in Other 29/12/2021 2 Comments

Another catchy title, right? 😀

Decided that I want to move my UniFi network controller to Ubuntu based installation, which will be running on top of Raspberry Pi (3) device which was collecting dust.

UPDATE: If you’re having issues with Java 11 or path not being set, take a look at comment by G. Faria below.

This can be easily achieved by following these steps:

take a backup of your current configuration (my controller was offline, so I’ve just copied the last automatic backup from /var/lib/unifi/backup/autobackup)
prepare the SD card with OS installation (detailed info here) – I’ve selected Ubuntu Server 21.10:

with prepared SD card, boot Ubuntu Server on your Raspberry Pi device
first login is ubuntu/ubuntu, and you’ll need to change password immediately after
next, you’ll probably want to set your Raspberry Pi to use static IP configuration – I’m using netplan to set it up:
- just in case, I removed all *.yaml files inside /etc/netplan/
- create new netplan template (YAML file) called 00-eth0.yaml in /etc/netplan/ (watch those white spaces!)

# 00-eth0.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: false
      set-name: eth0
      match:
        driver: "smsc95xx"
      addresses: [192.168.13.4/24]
      routes:
      - to: default
        via: 192.168.13.1
      nameservers:
        addresses: [192.168.13.1]

# 00-eth0.yaml

network:

version: 2

renderer: networkd

ethernets:

eth0:

dhcp4: false

set-name: eth0

match:

driver: "smsc95xx"

addresses: [192.168.13.4/24]

routes:

- to: default

via: 192.168.13.1

nameservers:

addresses: [192.168.13.1]

- apply the configuration (your IP address will be reconfigured, so you’ll also lose the current SSH connection, if connected remotely(!))

netplan generate
netplan apply

1 2	netplan generate netplan apply

prepare for UniFi Network Application installation:
- two important prerequisites are MongoDB (available releases) and LibSSL (found it here):

sudo su -
cd ~

wget http://launchpadlibrarian.net/555133932/libssl1.0.0_1.0.2n-1ubuntu5.7_arm64.deb
dpkg -i libssl1.0.0_1.0.2n-1ubuntu5.7_arm64.deb

wget https://repo.mongodb.org/apt/ubuntu/dists/xenial/mongodb-org/3.6/multiverse/binary-arm64/mongodb-org-server_3.6.23_arm64.deb
dpkg -i mongodb-org-server_3.6.23_arm64.deb

sudo su -

cd ~

wget http://launchpadlibrarian.net/555133932/libssl1.0.0_1.0.2n-1ubuntu5.7_arm64.deb

dpkg -i libssl1.0.0_1.0.2n-1ubuntu5.7_arm64.deb

wget https://repo.mongodb.org/apt/ubuntu/dists/xenial/mongodb-org/3.6/multiverse/binary-arm64/mongodb-org-server_3.6.23_arm64.deb

dpkg -i mongodb-org-server_3.6.23_arm64.deb

install the UniFi Network Application:
- there is a nice official guide here, once you take care of prerequisites, but basically:

# install prerequisite packages
apt-get update && apt-get install ca-certificates apt-transport-https

# add Ubiquity repo (for ARM)
echo 'deb [arch=armhf] https://www.ui.com/downloads/unifi/debian stable ubiquiti' | tee /etc/apt/sources.list.d/00-ubnt-unifi.list

# download and install GPG keys
wget -O /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ui.com/unifi/unifi-repo.gpg

# prevent updating OpenJDK
apt-mark hold openjdk-11-*

# update repos and install UniFi Network Application
apt-get update && apt-get install unifi -y

# install prerequisite packages

apt-get update && apt-get install ca-certificates apt-transport-https

# add Ubiquity repo (for ARM)

echo 'deb [arch=armhf] https://www.ui.com/downloads/unifi/debian stable ubiquiti' | tee /etc/apt/sources.list.d/00-ubnt-unifi.list

# download and install GPG keys

wget -O /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ui.com/unifi/unifi-repo.gpg

# prevent updating OpenJDK

apt-mark hold openjdk-11-*

# update repos and install UniFi Network Application

apt-get update && apt-get install unifi -y

restore from backup and start using it:

Note: If you’ll check your unifi service status, you may see “WARN Unable to load properties from ‘/usr/lib/unifi/data/system.properties’“:

This can easily be resolved by enabling the built-in “uncomplicated firewall” (ufw) – don’t forget to open ports you’ll need when it’s active! (such as SSH (22/tcp), inform endpoint for your devices (8080/tcp), UniFi Network Application web (8443/tcp), etc.):

# check if UFW is enabled already (probably not)
ufw status

# set exceptions
ufw allow 22/tcp
ufw allow 8080/tcp
ufw allow 8443/tcp

# enable UFW and recheck
ufw enable
ufw status

# restart unifi service
service unifi restart
service unifi status

# check if UFW is enabled already (probably not)

ufw status

# set exceptions

ufw allow 22/tcp

ufw allow 8080/tcp

ufw allow 8443/tcp

# enable UFW and recheck

ufw enable

ufw status

# restart unifi service

service unifi restart

service unifi status

* can be done more restricted, if needed
** full list of ports is available at https://help.ui.com/hc/en-us/articles/218506997-UniFi-Ports-Used – add them if needed (for instance, 6789/tcp is used when testing upload/download with mobile app)
*** be careful – it’s a firewall!

Note: If your access point is shown offline/timeout, maybe you forgot to open up the “inform port” 8080/tcp in UFW (been there, of course 😀):

And that’s it – you now have the fully functioning UniFi controller/UniFi Network Application running on top of your Ubuntu Server powered Raspberry Pi device!

Cheers!

P.S. Enable the autobackup feature… it’s useful (sometimes)! 😀

Bad Request for url (error 400) in AKS

By Tomica Kaniski in Other 14/12/2020 2 Comments

I’ve decided to go through the **awesome** AKS Workshop on Microsoft Learn and had some issues (with my setup), which I wanted to share, in case someone else hits them.

It was all good until I got to the part of creating the AKS cluster with Azure CLI – I was using Windows Terminal with WSL (Ubuntu 20.04) instead of using Azure Cloud Shell as suggested. I’ve gone through the steps of preparing variables needed for creating the cluster as it says, and when I tried to finally create the cluster by using “az aks create” command, I’ve got an error:

Error states that something is wrong with our request and neither –verbose or –debug options were giving me any useful details (actually, it was in front of me all the time, but I didn’t see it 😊). I’ve rechecked/reset the variables, tried once more and once more… it was all the same. As Google was conveniently down at the time (who would say, right?!), I’ve had to try and figure it out by myself. So, I’ve looked at the error once again:

Operation failed with status: ‘Bad Request’. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/aks-workshop/providers/Microsoft.Network/virtualNetworks/aks-vnet/subnets/aks-subnet%0D/providers/Microsoft.Authorization/roleAssignments?$filter=atScope%28%29&api-version=2018-09-01-preview

… and then it struck me!

There’s some trash in the URL (more precisely – my AKS subnet ID was having “%0D” added to the end)!

And if we check what “%0D” exactly stands for, it says “carriage return” (which I’ve obviously didn’t want to be a part of my subnet ID) – so, even it all seemed fine when looking at the variable content, now I know it wasn’t.

Easy-peasy, we can fix the part where we’re extracting this subnet ID or we can just replace the variable’s value with the right one (without the %0D at its end, that is).

That got me going… towards the next error. This one was actually more descriptive (yes, and the first one is descriptive enough, if you read it carefully 😊) – it said that I’ve got additional content inside my Kubernetes version variable:

Operation failed with status: ‘Bad Request’. Details: Error to parse agent pool version “1.19.3\r”: Invalid character(s) found in patch number “3\r”

You can see the extra “\r“, which again, is here because of bad value assigned to the variable $VERSION.

Which can also be easily fixed.

One other funny thing I’ve observed was, when getting my Kubernetes cluster credentials, as you can see below, they were actually merged to C:\Users\tomica\.kube\config:

This was funny because I’m inside WSL… which doesn’t actually have C:\Users\tomica\.kube\config, right? (and no, credentials weren’t merged to /home/tomica/.kube/config, which kubectl there uses by default, so… they are actually at /mnt/c/Users/tomica/.kube/config – funny, will check with the MS folks) 😊

Fair enough – we can merge them manually or just select the right file and we’re good to go:

There you go – if you get stuck on similar things, maybe this can help you. 😊

Cheers!

Azure VM with CloudFlare DNS… by Terraform

By Tomica Kaniski in Other 25/08/2020 0 Comment

I like Terraform! 🙂

Had an idea the other day – I wanted to make myself an Ubuntu virtual machine on Microsoft Azure and also create a corresponding DNS record inside my kaniski.eu domain (currently hosted at CloudFlare… like, I guess, half of the Internet). Of course, I can do all that by using both of the portals, but this time I wanted to do it with Terraform, so it can be easily brought up and down, reusable, etc.

For this, you’ll need:

Microsoft Azure subscription
CloudFlare account (with a domain attached to its DNS servers, of course)
Terraform (v0.13)
(optional) Visual Studio Code
(or you can all do it inside Azure Cloud Shell)

To configure Terraform with your Azure subscription, you’ll need:

Azure Subscription ID
Tenant ID
Client ID
Client secret

There’s a nice document about setting it all up – https://docs.microsoft.com/en-us/azure/developer/terraform/overview.

For accessing CloudFlare API via Terraform, you’ll need the following info from your CloudFlare account page:

e-mail address
CloudFlare zone ID
CloudFlare account ID

CloudFlare API key

Be really careful with using your Azure/CloudFlare keys (can’t be stressed enough!)!

Also note that the following code is only one way of doing it, one that suits me for this purpose, not best practice or the prettiest code ever! 🙂

The whole example (main.tf) looks like this (it can/should be improved, but that’s not really the point here and now):

# providers
provider "azurerm" {
  version = "~> 2.24"

  subscription_id = var.subscription_id
  tenant_id       = var.tenant_id
  client_id       = var.client_id
  client_secret   = var.client_secret

  features {}
}

provider "cloudflare" {
  version = "~> 2.10.0"

  email      = var.cf_email
  api_key    = var.cf_apikey
  account_id = var.cf_accountid
}

# resource group
resource "azurerm_resource_group" "example-rg" {
  name     = "example-rg"
  location = "West Europe"
}

# network
resource "azurerm_virtual_network" "example-net" {
  name                = "example-net"
  location            = azurerm_resource_group.example-rg.location
  resource_group_name = azurerm_resource_group.example-rg.name
  address_space       = ["10.11.0.0/16"]
}

# subnet
resource "azurerm_subnet" "example-sub" {
  name                 = "example-sub"
  resource_group_name  = azurerm_resource_group.example-rg.name
  virtual_network_name = azurerm_virtual_network.example-net.name
  address_prefixes     = ["10.11.12.0/24"]
}

# nics
resource "azurerm_network_interface" "myvm-nic" {
  name                 = "myvm-nic"
  location             = azurerm_resource_group.example-rg.location
  resource_group_name  = azurerm_resource_group.example-rg.name
  enable_ip_forwarding = true

  ip_configuration {
    name                          = "myvm-nic-ip-config"
    subnet_id                     = azurerm_subnet.example-sub.id
    private_ip_address_allocation = "Static"
    private_ip_address            = "10.11.12.4"
    public_ip_address_id          = azurerm_public_ip.myvm-ip.id
  }
}

# public ips
resource "azurerm_public_ip" "myvm-ip" {
  name                = "myvm-ip"
  location            = azurerm_resource_group.example-rg.location
  resource_group_name = azurerm_resource_group.example-rg.name
  allocation_method   = "Static"
}

# network security group
resource "azurerm_network_security_group" "example-nsg" {
  name                = "example-nsg"
  location            = azurerm_resource_group.example-rg.location
  resource_group_name = azurerm_resource_group.example-rg.name

  security_rule {
    name                       = "SSH"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "10.11.12.4"
  }
}

resource "azurerm_subnet_network_security_group_association" "example-sub-nsg-assoc" {
  subnet_id                 = azurerm_subnet.example-sub.id
  network_security_group_id = azurerm_network_security_group.example-nsg.id
}

# vms
resource "azurerm_virtual_machine" "myvm" {
  name                          = "myvm"
  location                      = azurerm_resource_group.example-rg.location
  resource_group_name           = azurerm_resource_group.example-rg.name
  network_interface_ids         = [azurerm_network_interface.myvm-nic.id]
  vm_size                       = "Standard_DS1_v2"
  delete_os_disk_on_termination = true

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "19_10-daily-gen2"
    version   = "latest"
  }

  storage_os_disk {
    name              = "myvm-osdisk"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
  }

  os_profile {
    computer_name  = "myvm"
    admin_username = var.admin_username
  }

  os_profile_linux_config {
    disable_password_authentication = true
    ssh_keys {
      key_data = file("~/.ssh/id_rsa.pub")
      path     = "/home/${var.admin_username}/.ssh/authorized_keys"
    }
  }
}

# cloudflare dns
resource "cloudflare_record" "myvm-dns" {
  zone_id = var.cf_zoneid
  name    = "myvm"
  value   = azurerm_public_ip.myvm-ip.ip_address
  type    = "A"
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

# providers

provider "azurerm" {

version = "~> 2.24"

subscription_id = var.subscription_id

tenant_id = var.tenant_id

client_id = var.client_id

client_secret = var.client_secret

features {}

}

provider "cloudflare" {

version = "~> 2.10.0"

email = var.cf_email

api_key = var.cf_apikey

account_id = var.cf_accountid

}

# resource group

resource "azurerm_resource_group" "example-rg" {

name = "example-rg"

location = "West Europe"

}

# network

resource "azurerm_virtual_network" "example-net" {

name = "example-net"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

address_space = ["10.11.0.0/16"]

}

# subnet

resource "azurerm_subnet" "example-sub" {

name = "example-sub"

resource_group_name = azurerm_resource_group.example-rg.name

virtual_network_name = azurerm_virtual_network.example-net.name

address_prefixes = ["10.11.12.0/24"]

}

# nics

resource "azurerm_network_interface" "myvm-nic" {

name = "myvm-nic"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

enable_ip_forwarding = true

ip_configuration {

name = "myvm-nic-ip-config"

subnet_id = azurerm_subnet.example-sub.id

private_ip_address_allocation = "Static"

private_ip_address = "10.11.12.4"

public_ip_address_id = azurerm_public_ip.myvm-ip.id

}

# public ips

resource "azurerm_public_ip" "myvm-ip" {

name = "myvm-ip"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

allocation_method = "Static"

}

# network security group

resource "azurerm_network_security_group" "example-nsg" {

name = "example-nsg"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

security_rule {

name = "SSH"

priority = 100

direction = "Inbound"

access = "Allow"

protocol = "Tcp"

source_port_range = "*"

destination_port_range = "22"

source_address_prefix = "*"

destination_address_prefix = "10.11.12.4"

}

resource "azurerm_subnet_network_security_group_association" "example-sub-nsg-assoc" {

subnet_id = azurerm_subnet.example-sub.id

network_security_group_id = azurerm_network_security_group.example-nsg.id

}

# vms

resource "azurerm_virtual_machine" "myvm" {

name = "myvm"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

network_interface_ids = [azurerm_network_interface.myvm-nic.id]

vm_size = "Standard_DS1_v2"

delete_os_disk_on_termination = true

storage_image_reference {

publisher = "Canonical"

offer = "UbuntuServer"

sku = "19_10-daily-gen2"

version = "latest"

}

storage_os_disk {

name = "myvm-osdisk"

caching = "ReadWrite"

create_option = "FromImage"

managed_disk_type = "Standard_LRS"

}

os_profile {

computer_name = "myvm"

admin_username = var.admin_username

}

os_profile_linux_config {

disable_password_authentication = true

ssh_keys {

key_data = file("~/.ssh/id_rsa.pub")

path = "/home/${var.admin_username}/.ssh/authorized_keys"

}

# cloudflare dns

resource "cloudflare_record" "myvm-dns" {

zone_id = var.cf_zoneid

name = "myvm"

value = azurerm_public_ip.myvm-ip.ip_address

type = "A"

}

Variables used in this simple example are (terraform.tfvars):

subscription_id = ""
tenant_id       = ""
client_id       = ""
client_secret   = ""
admin_username  = ""
cf_zoneid       = ""
cf_email        = ""
cf_apikey       = ""
cf_accountid    = ""

subscription_id = ""

tenant_id = ""

client_id = ""

client_secret = ""

admin_username = ""

cf_zoneid = ""

cf_email = ""

cf_apikey = ""

cf_accountid = ""

Just note one more thing – I’m using the current version of Terraform (v0.13) and version 2 of the CloudFlare provider (notable changes between the v1 and v2 are listed at https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/guides/version-2-upgrade).

As always, running terraform init will take care of the missing providers and install them as needed. Also, running terraform plan can give us an estimation of what exactly will be done and terraform fmt will make our code look nice! 🙂

When I run this code (terraform apply), it completes successfully (of course) and provisions 9 items and, more important – gives me info about the DNS name and IP address of the provisioned VM (which I already know, but still):

Next is a series of checks:

DNS record really exists in CloudFlare DNS?

resources are really provisioned in Azure?

SSH into the VM works?

And that’s it – when you’re done, you can use terraform destroy and it’s all gone! 🙂

Cheers!

Creating some virtual machines in Azure with PowerShell

By Tomica Kaniski in Other 15/03/2019 0 Comment

The other day I was creating some Linux virtual machines (I know, I know…) and, with Azure being my preferred hosting platform, I’ve decided to create this machines by using a simple PowerShell script. Not because I’m so good at PowerShell, but because I like it… and sometimes I really don’t like clicking through the wizard to create multiple machines.

I wanted to create multiple machines with ease, each with “static” IP address from the provided subnet, accessible via the Internet (SSH, HTTP) and running the latest Ubuntu Linux, of course.

So, I was browsing through the official documentation (a.k.a. docs.com, more specifically https://docs.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-powershell), and I’ve come up with this (my version of the official docs):

# prerequisites
Install-Module -Name Az -AllowClobber -Scope CurrentUser

# connect to Azure
Connect-AzAccount
Select-AzSubscription "<YOUR_SUBSCRIPTION_ID>"

# create VMs
$vmname = "myVM1"
$ip = "10.11.11.11"
$rg = "myResourceGroup"
$loc = "WestEurope"
$vnet = Get-AzVirtualNetwork -Name $rg -ResourceGroupName $rg
$subnet = Get-AzVirtualNetworkSubnetConfig -Name $subnet.Name -VirtualNetwork $vnet
$pip = New-AzPublicIpAddress -Name "$vmname-pip" -ResourceGroupName $rg -Location $loc -AllocationMethod Dynamic -IdleTimeoutInMinutes 4
$nsg_ssh = New-AzNetworkSecurityRuleConfig -Name "ssh" -Protocol "Tcp" -Direction "Inbound" -Priority 300 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 22 -Access "Allow"
$nsg_web = New-AzNetworkSecurityRuleConfig -Name "web" -Protocol "Tcp" -Direction "Inbound" -Priority 301 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 80 -Access "Allow"
$nsg = New-AzNetworkSecurityGroup -Name "$vmname-nsg" -ResourceGroupName $rg -Location $loc -SecurityRules $nsg_ssh,$nsg_web
if(Test-AzPrivateIPAddressAvailability -IPAddress $ip -ResourceGroupName $rg -VirtualNetworkName $vnet.Name){
  $ipconfig = New-AzNetworkInterfaceIpConfig -Name "$vmname-ip" -Subnet $subnet -PrivateIpAddress $ip -PublicIpAddress $pip
  $nic = New-AzNetworkInterface -Name "$vmname-nic" -ResourceGroupName $rg -Location $loc -IpConfiguration $ipconfig -NetworkSecurityGroupId $nsg.Id
}
$securePassword = ConvertTo-SecureString ' ' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential ("tomica", $securePassword)
$vmConfig = New-AzVMConfig -VMName $vmname -VMSize "Standard_A1_v2" | Set-AzVMOperatingSystem -Linux -ComputerName $vmname -Credential $cred -DisablePasswordAuthentication | Set-AzVMSourceImage -PublisherName "Canonical" -Offer "UbuntuServer" -Skus "18.10" -Version "latest" | Add-AzVMNetworkInterface -Id $nic.Id
$sshPublicKey = cat ~/.ssh/id_rsa.pub
Add-AzVMSshPublicKey -VM $vmconfig -KeyData $sshPublicKey -Path "/home/tomica/.ssh/authorized_keys"
New-AzVM -ResourceGroupName $rg -Location $loc -VM $vmConfig

# prerequisites

Install-Module -Name Az -AllowClobber -Scope CurrentUser

# connect to Azure

Connect-AzAccount

Select-AzSubscription "<YOUR_SUBSCRIPTION_ID>"

# create VMs

$vmname = "myVM1"

$ip = "10.11.11.11"

$rg = "myResourceGroup"

$loc = "WestEurope"

$vnet = Get-AzVirtualNetwork -Name $rg -ResourceGroupName $rg

$subnet = Get-AzVirtualNetworkSubnetConfig -Name $subnet.Name -VirtualNetwork $vnet

$pip = New-AzPublicIpAddress -Name "$vmname-pip" -ResourceGroupName $rg -Location $loc -AllocationMethod Dynamic -IdleTimeoutInMinutes 4

$nsg_ssh = New-AzNetworkSecurityRuleConfig -Name "ssh" -Protocol "Tcp" -Direction "Inbound" -Priority 300 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 22 -Access "Allow"

$nsg_web = New-AzNetworkSecurityRuleConfig -Name "web" -Protocol "Tcp" -Direction "Inbound" -Priority 301 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 80 -Access "Allow"

$nsg = New-AzNetworkSecurityGroup -Name "$vmname-nsg" -ResourceGroupName $rg -Location $loc -SecurityRules $nsg_ssh,$nsg_web

if(Test-AzPrivateIPAddressAvailability -IPAddress $ip -ResourceGroupName $rg -VirtualNetworkName $vnet.Name){

$ipconfig = New-AzNetworkInterfaceIpConfig -Name "$vmname-ip" -Subnet $subnet -PrivateIpAddress $ip -PublicIpAddress $pip

$nic = New-AzNetworkInterface -Name "$vmname-nic" -ResourceGroupName $rg -Location $loc -IpConfiguration $ipconfig -NetworkSecurityGroupId $nsg.Id

}

$securePassword = ConvertTo-SecureString ' ' -AsPlainText -Force

$cred = New-Object System.Management.Automation.PSCredential ("tomica", $securePassword)

$vmConfig = New-AzVMConfig -VMName $vmname -VMSize "Standard_A1_v2" | Set-AzVMOperatingSystem -Linux -ComputerName $vmname -Credential $cred -DisablePasswordAuthentication | Set-AzVMSourceImage -PublisherName "Canonical" -Offer "UbuntuServer" -Skus "18.10" -Version "latest" | Add-AzVMNetworkInterface -Id $nic.Id

$sshPublicKey = cat ~/.ssh/id_rsa.pub

Add-AzVMSshPublicKey -VM $vmconfig -KeyData $sshPublicKey -Path "/home/tomica/.ssh/authorized_keys"

New-AzVM -ResourceGroupName $rg -Location $loc -VM $vmConfig

If this helps you with similar task – you’re welcome.

Cheers!

Deploying Linux machines by using System Center 2016: Virtual Machine Manager templates

By Tomica Kaniski in Other 04/09/2017 0 Comment

In light of “Microsoft loves Linux” initiative, you can now deploy your Linux virtual machines by using templates in the System Center 2016: Virtual Machine Manager. As I was searching on how to do this (successfully), there were couple of articles that helped, so I’ve decided to do a short list of all the necessary steps (in one place).

Steps to make your Linux VM template deployments work:

create a new (Generation 2) virtual machine (as you would normally do)
install the Linux operating system in that virtual machine (as you would normally do)
- HINT: A list of supported Linux distributions and versions on Hyper-V is available here.
install the Linux Integration Services (LIS) (as per this post):
- open the “modules” file
  
  Shell
  
  sudo nano /etc/initramfs-tools/modules
  
  1
  
  sudo nano /etc/initramfs-tools/modules
- add the following to the end of this file:
  
  hv_vmbus hv_storvsc hv_blkvsc hv_netvsc
  
  1
  2
  3
  4
  
  hv_vmbus
  hv_storvsc
  hv_blkvsc
  hv_netvsc

- save it (Ctrl+X and Y)
- install LIS and reboot the machine by using the following commands:
  
  Shell
  
  sudo apt-get install --install-recommends linux-tools-virtual-lts-xenial linux-cloud-tools-virtual-lts-xenial sudo update-initramfs -u sudo reboot
  
  1
  2
  3
  
  sudo apt-get install --install-recommends linux-tools-virtual-lts-xenial linux-cloud-tools-virtual-lts-xenial
  sudo update-initramfs -u
  sudo reboot
- check if the services are running by using the command:
  
  Shell
  
  sudo lsmod
  
  1
  
  sudo lsmod
install the Virtual Machine Manager agent (as per this post):
- share the folder C:\Program Files\Microsoft System Center 2016\Agents\Linux on your VMM machine
- copy the VMM agent files to Linux virtual machine
  - as a real Windows admin, I did it through the GUI
- install the agent:
  
  Shell
  
  sudo ./install scvmmguestagent.1.0.2.1075.x64.tar
  
  1
  
  sudo ./install scvmmguestagent.1.0.2.1075.x64.tar
fix the boot for Generation 2 virtual machine (boot information is by default stored in the VM configuration file, not on disk – Ben wrote a great article on this “issue”)
- Ben’s way (didn’t work for me):
  - change directory to the boot EFI directory
    
    Shell
    
    sudo cd /boot/efi/EFI
    
    1
    
    sudo cd /boot/efi/EFI
  - copy the ubuntu directory in to a new directory named boot
    
    Shell
    
    sudo cp -r ubuntu/ boot
    
    1
    
    sudo cp -r ubuntu/ boot
  - change directory to the newly created boot directory
    
    Shell
    
    sudo cd boot
    
    1
    
    sudo cd boot
  - rename the shimx64.efi file
    
    Shell
    
    sudo mv shimx64.efi bootx64.efi
    
    1
    
    sudo mv shimx64.efi bootx64.efi
- TriJetScud’s way in the comments (worked for me with Ubuntu 16.04 Generation 2 VM):
  
  Shell
  
  sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --no-nvram --removable
  
  1
  
  sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --no-nvram --removable
shutdown the virtual machine and copy its VHDX to the VMM Library
- HINT: Don’t forget to refresh the VMM Library.
go to the VMM Library, right-click the copied VHDX and select the Create VM template option
proceed with creating the template as you normally would, to the part Configure Operating System
- HINT: If you are using Secure boot, don’t forget to select the MicrosoftUEFICertificateAuthority template in hardware settings.
there, under Guest OS profile, you select the option to create a new Linux operating system customization settings
next you specify your guest OS settings and finish creating the template
now you can create a new Linux virtual machine from the template you’ve configured!

Hope it helps!

Cheers!