cloud | Tom's Notes

Using a self-hosted runner with GitHub Actions

By Tomica Kaniski in Other 17/10/2022 0 Comment

As I was going through the excellent short course called Azure Infrastructure as Code with GitHub (by fellow MVP, Barbara Forbes), a thought appeared – what do I need to do to use my custom runner machine inside a pipeline for… I don’t know… security/privacy concerns, isolation, special requirements, different OS, control, price… or just to complicate things a bit?

Of course, GitHub supports this and it’s called a self-hosted runner.

So, what do I need to do to use this self-hosted runner with my GitHub Actions?

It’s relatively simple – there is an application package, which will be installed on your runner machine, and which will listen for and eventually do all the work defined in your workflow!

But first, let’s introduce my environment.

I have a simple GitHub Action (workflow), which creates a simple storage account on my Azure environment (there is actually no need to convert Bicep to ARM before deployment, but it seemed cool 😀). It’s currently using the „ubuntu-latest“ runner, provided by GitHub… which has also all the needed components inside (like Azure CLI, Azure PowerShell, …).

And it works fine. When there is a push to my GitHub repository, GitHub Actions starts and does what is needed on my Azure environment via this workflow:

name: deploy-to-azure
on:
  push:
    branches: [ "main" ]
  workflow_dispatch:

env:
  resourceGroupName: SimpleAzureRG

jobs:
  deploy-to-azure:
    runs-on: ubuntu-latest

    steps:
      - name: "Checkout the code from GitHub"
        uses: actions/checkout@v3

      - name: "Build Bicep code into ARM (JSON)"
        uses: Azure/cli@v1
        with:
          inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

      - name: "Login to Azure"
        uses: Azure/login@v1.4.6
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true

      - name: "Create the resource group, if needed"
        uses: Azure/powershell@v1
        with:
          inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force
          azPSVersion: latest

      - name: "Deploy the ARM template"
        uses: Azure/arm-deploy@v1.0.8
        with:
          scope: resourcegroup
          resourceGroupName: ${{ env.resourceGroupName }}
          template: main.json
          parameters: storageAccountPrefix=simplegh
          deploymentName: "gh${{ github.run_id }}"

on:

push:

branches: [ "main" ]

workflow_dispatch:

env:

resourceGroupName: SimpleAzureRG

jobs:

deploy-to-azure:

runs-on: ubuntu-latest

steps:

- name: "Checkout the code from GitHub"

uses: actions/checkout@v3

- name: "Build Bicep code into ARM (JSON)"

uses: Azure/cli@v1

with:

inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

- name: "Login to Azure"

uses: Azure/[email protected]

with:

creds: ${{ secrets.AZURE_CREDENTIALS }}

enable-AzPSSession: true

- name: "Create the resource group, if needed"

uses: Azure/powershell@v1

with:

inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force

azPSVersion: latest

- name: "Deploy the ARM template"

uses: Azure/[email protected]

with:

scope: resourcegroup

resourceGroupName: ${{ env.resourceGroupName }}

parameters: storageAccountPrefix=simplegh

deploymentName: "gh${{ github.run_id }}"

And the mighty Bicep file (😀) it’s using for the deployment is:

@maxLength(11)
param storageAccountPrefix string = 'bicep'
param location string = resourceGroup().location

var sta = '${storageAccountPrefix}${uniqueString(subscription().id)}'

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-02-01' = {
  name: sta
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
}

@maxLength(11)

param storageAccountPrefix string = 'bicep'

param location string = resourceGroup().location

var sta = '${storageAccountPrefix}${uniqueString(subscription().id)}'

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-02-01' = {

location: location

kind: 'StorageV2'

sku: {

}

Of course, this runs just fine on a standard (hosted) runner:

To run this workflow (successfully) not that much is needed.

First, I’ve created a new virtual machine (I’ll use a simple Ubuntu Hyper-V VM, no autoscaling, no… nothing) called hermes (god of speed 😀), with freshly installed Ubuntu 22.04.1-LTS (minimized).

After that, I went to the Settings of my GitHub repository and got the download and install scripts for the x64 Linux runner:

### get and install the runner application (script provided by GitHub)
## download the runner application
# create a runner folder
mkdir actions-runner && cd actions-runner

# download the latest runner package
curl -o actions-runner-linux-x64-2.298.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.298.2/actions-runner-linux-x64-2.298.2.tar.gz

# (optional) validate the hash
echo "0bfd792196ce0ec6f1c65d2a9ad00215b2926ef2c416b8d97615265194477117  actions-runner-linux-x64-2.298.2.tar.gz" | shasum -a 256 -c

# extract the installer
tar xzf ./actions-runner-linux-x64-2.298.2.tar.gz


## configure the runner application
# create the runner and start the configuration experience
./config.sh --url https://github.com/TomicaKaniski/AzureIaCGH --token "<<--TOKEN_GOES_HERE!-->>"

# last step, run it! (we will actually run it via crontab at machine restart!)
# ./run.sh

### get and install the runner application (script provided by GitHub)

## download the runner application

# create a runner folder

mkdir actions-runner && cd actions-runner

# download the latest runner package

curl -o actions-runner-linux-x64-2.298.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.298.2/actions-runner-linux-x64-2.298.2.tar.gz

# (optional) validate the hash

echo "0bfd792196ce0ec6f1c65d2a9ad00215b2926ef2c416b8d97615265194477117 actions-runner-linux-x64-2.298.2.tar.gz" | shasum -a 256 -c

# extract the installer

tar xzf ./actions-runner-linux-x64-2.298.2.tar.gz

## configure the runner application

# create the runner and start the configuration experience

./config.sh --url https://github.com/TomicaKaniski/AzureIaCGH --token "<<--TOKEN_GOES_HERE!-->>"

# last step, run it! (we will actually run it via crontab at machine restart!)

# ./run.sh

As you can see, I’ll be using crontab later to automatically (re)start my self-hosted runner.

If everything went well, you should see your runner “up and running” (😀) in the GitHub portal:

Next, I’ll use the following script to install all prerequisites for my workflow (like Azure CLI, Azure PowerShell, etc. – it really depends on your workflow and things you use):

### install (my) workflow prerequisites on self-runner machine
sudo su -

## install Azure CLI (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)
curl -sL https://aka.ms/InstallAzureCLIDeb | bash

## install PowerShell  (https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.2#installation-via-direct-download)
wget -q "https://github.com/PowerShell/PowerShell/releases/download/v7.2.6/powershell-lts_7.2.6-1.deb_amd64.deb" && dpkg -i powershell-lts_7.2.6-1.deb_amd64.deb

## install Bicep
az bicep install

## install other useful things
apt install apt-utils cron vim -y

## install Docker (https://cloudcone.com/docs/article/how-to-install-docker-on-ubuntu-22-04-20-04/)
# install prerequisites
apt install apt-transport-https curl gnupg-agent ca-certificates software-properties-common -y

# add GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add repository
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" -y

# install docker
apt install docker-ce docker-ce-cli containerd.io -y

# create docker group
newgrp docker
exit

# add current user (tomica) to docker group
sudo usermod -aG docker $USER

# configure Docker autostart
sudo systemctl enable docker
sudo systemctl restart docker
sudo systemctl status docker

## install Az module for PowerShell (https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.0.0)
pwsh
Install-Module Az -Repository PSGallery -Confirm:$false -Force
exit


### configure ./run.sh (GitHub runner application) to start automatically
crontab -e

# add the following line (uncommented, of course; and check if the path is right!):
# @reboot /home/tomica/actions-runner/run.sh

sudo reboot

### install (my) workflow prerequisites on self-runner machine

sudo su -

## install Azure CLI (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)

curl -sL https://aka.ms/InstallAzureCLIDeb | bash

## install PowerShell (https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.2#installation-via-direct-download)

wget -q "https://github.com/PowerShell/PowerShell/releases/download/v7.2.6/powershell-lts_7.2.6-1.deb_amd64.deb" && dpkg -i powershell-lts_7.2.6-1.deb_amd64.deb

## install Bicep

az bicep install

## install other useful things

apt install apt-utils cron vim -y

## install Docker (https://cloudcone.com/docs/article/how-to-install-docker-on-ubuntu-22-04-20-04/)

# install prerequisites

apt install apt-transport-https curl gnupg-agent ca-certificates software-properties-common -y

# add GPG key

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add repository

add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" -y

# install docker

apt install docker-ce docker-ce-cli containerd.io -y

# create docker group

newgrp docker

exit

# add current user (tomica) to docker group

sudo usermod -aG docker $USER

# configure Docker autostart

sudo systemctl enable docker

sudo systemctl restart docker

sudo systemctl status docker

## install Az module for PowerShell (https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.0.0)

pwsh

Install-Module Az -Repository PSGallery -Confirm:$false -Force

exit

### configure ./run.sh (GitHub runner application) to start automatically

crontab -e

# add the following line (uncommented, of course; and check if the path is right!):

# @reboot /home/tomica/actions-runner/run.sh

sudo reboot

Once this is done, my self-hosted runner hermes should be ready to run the workflow.

To try this, I need to make a slight update to my workflow file – line 12 inside the job configuration should be updated from “runs-on: ubuntu-latest” to “runs-on: self-hosted“.

So, my workflow YAML file now looks like this:

name: deploy-to-azure
on:
  push:
    branches: [ "main" ]
  workflow_dispatch:

env:
  resourceGroupName: SimpleAzureRG

jobs:
  deploy-to-azure:
    runs-on: self-hosted

    steps:
      - name: "Checkout the code from GitHub"
        uses: actions/checkout@v3

      - name: "Build Bicep code into ARM (JSON)"
        uses: Azure/cli@v1
        with:
          inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

      - name: "Login to Azure"
        uses: Azure/login@v1.4.6
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true

      - name: "Create the resource group, if needed"
        uses: Azure/powershell@v1
        with:
          inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force
          azPSVersion: latest

      - name: "Deploy the ARM template"
        uses: Azure/arm-deploy@v1.0.8
        with:
          scope: resourcegroup
          resourceGroupName: ${{ env.resourceGroupName }}
          template: main.json
          parameters: storageAccountPrefix=simplegh
          deploymentName: "gh${{ github.run_id }}"

on:

push:

branches: [ "main" ]

workflow_dispatch:

env:

resourceGroupName: SimpleAzureRG

jobs:

deploy-to-azure:

runs-on: self-hosted

steps:

- name: "Checkout the code from GitHub"

uses: actions/checkout@v3

- name: "Build Bicep code into ARM (JSON)"

uses: Azure/cli@v1

with:

inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

- name: "Login to Azure"

uses: Azure/[email protected]

with:

creds: ${{ secrets.AZURE_CREDENTIALS }}

enable-AzPSSession: true

- name: "Create the resource group, if needed"

uses: Azure/powershell@v1

with:

inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force

azPSVersion: latest

- name: "Deploy the ARM template"

uses: Azure/[email protected]

with:

scope: resourcegroup

resourceGroupName: ${{ env.resourceGroupName }}

parameters: storageAccountPrefix=simplegh

deploymentName: "gh${{ github.run_id }}"

And once I push the configuration to my GitHub, my workflow automatically starts and runs on hermes, my self-hosted runner:

If we prepared our runner right, all is good! 😊

Of course, our resources are deployed successfully:

So, this is how you can use your own, self-hosted runner, to execute your GitHub Actions (workflows).

Cheers!

Fixing permissions for EC2 private key file

By Tomica Kaniski in Other 10/06/2022 0 Comment

This time, I was playing around with AWS and created some EC2 instances.

When you are creating and working with your instances, you will need to take care of the authentication – you would usually import or create new key pair and use private key on your machine to connect via SSH to the EC2 instance in AWS. The whole process of creating a key pair and downloading the private key is pretty simple – on the page below, you select name, type and format of your key pair and, when created, private key begins automatic download to your PC:

Now you can create your instance and select the created key pair for authentication:

If you have your private key ready and the instance is up and accessible to you, you can use (for example) SSH to connect to it:

So… we have a challenge! Looks like our private key is not secured enough and others may have access to it!

If we look at the permissions, we can see that all of them are actually inherited… so, we’ll need to remove the inheritance/inherited permissions and give them only to the account that needs it:

And after some “tweaking”:

If we retry the connection, this happens:

Excellent!

And if you’re not a fan of clicking through the permissions dialog, here are scripts that can help you with this – they basically remove the inheritance and add full access permissions to the owner of the file (needs path to your private key file as a parameter!):

the “PowerShell” flavour:

# script fixes permissions for a private key (PEM) file used to connect to EC2 instance
param (
    [Parameter(Mandatory)]
    [String] $file
)

$acl = Get-Acl -Path $file
$acl.SetAccessRuleProtection($true, $false) # disable inheritance, don't preserve existing rules
$rule = New-Object System.Security.AccessControl.FileSystemAccessRule($acl.Owner, "FullControl", "None", "None", "Allow") # allow only Full access for the file owner
$acl.SetAccessRule($rule)
Set-Acl -Path $file -AclObject $acl

# script fixes permissions for a private key (PEM) file used to connect to EC2 instance

param (

[Parameter(Mandatory)]

[String] $file

)

$acl = Get-Acl -Path $file

$acl.SetAccessRuleProtection($true, $false) # disable inheritance, don't preserve existing rules

$rule = New-Object System.Security.AccessControl.FileSystemAccessRule($acl.Owner, "FullControl", "None", "None", "Allow") # allow only Full access for the file owner

$acl.SetAccessRule($rule)

Set-Acl -Path $file -AclObject $acl

the “CMD” flavour:

@ECHO OFF
REM Script fixes permissions for a private key (PEM) file used to connect to EC2 instance
icacls.exe %1 /grant:r *S-1-3-0:(F) /inheritance:r

@ECHO OFF

REM Script fixes permissions for a private key (PEM) file used to connect to EC2 instance

icacls.exe %1 /grant:r *S-1-3-0:(F) /inheritance:r

Hope it helps!

Cheers!

P.S. Scripts are also available at my GitHub (https://github.com/TomicaKaniski/toms-notes-code/).

P.P.S. There’s also a script that restores inheritance and inherited permissions… in case you… mess something up. 😀

Open Cloud BBQ – Nano Server

By Tomica Kaniski in Other 18/06/2015 0 Comment

There was a barbecue today… on the roof… of the Microsoft office! How cool is that??? Smiješak

I’ve had an honor to join my colleagues and friends, and contribute to this great event with a presentation called “Nano Server – budućnost je tu!”, which was actually inspired by Jeffrey Snover’s Ignite speech about Nano Server – the new installation option in Windows Server 2016.

The whole Nano Server idea is very simple – let’s remove all the “unnecessary” roles and features (especially the famous GUI), and leave only the parts that we really need for whatever purpose (i.e. we’re getting the “purpose built servers”). All the purposes/roles that are supported in this preview version are Hyper-V, File Server and Failover Clustering, and they are working great on this “nano-sized” server operating system (even in this early preview).

I’m really excited about Nano Server, and I’m so glad that I’ve had a chance to talk about it at such great event. Because there were so many questions about it during the day/evening, I’ve noticed that Nano Server is a very cool subject for attendees also. Nice!

The presentation is available on SlideShare – check it, install the Nano Server and start rocking your datacenters! Smiješak

Open Cloud BBQ – Nano Server from Tomica Kaniski

Cheers!