arm | Tom's Notes

Using a self-hosted runner with GitHub Actions

By Tomica Kaniski in Other 17/10/2022 0 Comment

As I was going through the excellent short course called Azure Infrastructure as Code with GitHub (by fellow MVP, Barbara Forbes), a thought appeared – what do I need to do to use my custom runner machine inside a pipeline for… I don’t know… security/privacy concerns, isolation, special requirements, different OS, control, price… or just to complicate things a bit?

Of course, GitHub supports this and it’s called a self-hosted runner.

So, what do I need to do to use this self-hosted runner with my GitHub Actions?

It’s relatively simple – there is an application package, which will be installed on your runner machine, and which will listen for and eventually do all the work defined in your workflow!

But first, let’s introduce my environment.

I have a simple GitHub Action (workflow), which creates a simple storage account on my Azure environment (there is actually no need to convert Bicep to ARM before deployment, but it seemed cool 😀). It’s currently using the „ubuntu-latest“ runner, provided by GitHub… which has also all the needed components inside (like Azure CLI, Azure PowerShell, …).

And it works fine. When there is a push to my GitHub repository, GitHub Actions starts and does what is needed on my Azure environment via this workflow:

name: deploy-to-azure
on:
  push:
    branches: [ "main" ]
  workflow_dispatch:

env:
  resourceGroupName: SimpleAzureRG

jobs:
  deploy-to-azure:
    runs-on: ubuntu-latest

    steps:
      - name: "Checkout the code from GitHub"
        uses: actions/checkout@v3

      - name: "Build Bicep code into ARM (JSON)"
        uses: Azure/cli@v1
        with:
          inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

      - name: "Login to Azure"
        uses: Azure/login@v1.4.6
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true

      - name: "Create the resource group, if needed"
        uses: Azure/powershell@v1
        with:
          inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force
          azPSVersion: latest

      - name: "Deploy the ARM template"
        uses: Azure/arm-deploy@v1.0.8
        with:
          scope: resourcegroup
          resourceGroupName: ${{ env.resourceGroupName }}
          template: main.json
          parameters: storageAccountPrefix=simplegh
          deploymentName: "gh${{ github.run_id }}"

on:

push:

branches: [ "main" ]

workflow_dispatch:

env:

resourceGroupName: SimpleAzureRG

jobs:

deploy-to-azure:

runs-on: ubuntu-latest

steps:

- name: "Checkout the code from GitHub"

uses: actions/checkout@v3

- name: "Build Bicep code into ARM (JSON)"

uses: Azure/cli@v1

with:

inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

- name: "Login to Azure"

uses: Azure/[email protected]

with:

creds: ${{ secrets.AZURE_CREDENTIALS }}

enable-AzPSSession: true

- name: "Create the resource group, if needed"

uses: Azure/powershell@v1

with:

inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force

azPSVersion: latest

- name: "Deploy the ARM template"

uses: Azure/[email protected]

with:

scope: resourcegroup

resourceGroupName: ${{ env.resourceGroupName }}

parameters: storageAccountPrefix=simplegh

deploymentName: "gh${{ github.run_id }}"

And the mighty Bicep file (😀) it’s using for the deployment is:

@maxLength(11)
param storageAccountPrefix string = 'bicep'
param location string = resourceGroup().location

var sta = '${storageAccountPrefix}${uniqueString(subscription().id)}'

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-02-01' = {
  name: sta
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
}

@maxLength(11)

param storageAccountPrefix string = 'bicep'

param location string = resourceGroup().location

var sta = '${storageAccountPrefix}${uniqueString(subscription().id)}'

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-02-01' = {

location: location

kind: 'StorageV2'

sku: {

}

Of course, this runs just fine on a standard (hosted) runner:

To run this workflow (successfully) not that much is needed.

First, I’ve created a new virtual machine (I’ll use a simple Ubuntu Hyper-V VM, no autoscaling, no… nothing) called hermes (god of speed 😀), with freshly installed Ubuntu 22.04.1-LTS (minimized).

After that, I went to the Settings of my GitHub repository and got the download and install scripts for the x64 Linux runner:

### get and install the runner application (script provided by GitHub)
## download the runner application
# create a runner folder
mkdir actions-runner && cd actions-runner

# download the latest runner package
curl -o actions-runner-linux-x64-2.298.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.298.2/actions-runner-linux-x64-2.298.2.tar.gz

# (optional) validate the hash
echo "0bfd792196ce0ec6f1c65d2a9ad00215b2926ef2c416b8d97615265194477117  actions-runner-linux-x64-2.298.2.tar.gz" | shasum -a 256 -c

# extract the installer
tar xzf ./actions-runner-linux-x64-2.298.2.tar.gz


## configure the runner application
# create the runner and start the configuration experience
./config.sh --url https://github.com/TomicaKaniski/AzureIaCGH --token "<<--TOKEN_GOES_HERE!-->>"

# last step, run it! (we will actually run it via crontab at machine restart!)
# ./run.sh

### get and install the runner application (script provided by GitHub)

## download the runner application

# create a runner folder

mkdir actions-runner && cd actions-runner

# download the latest runner package

curl -o actions-runner-linux-x64-2.298.2.tar.gz -L https://github.com/actions/runner/releases/download/v2.298.2/actions-runner-linux-x64-2.298.2.tar.gz

# (optional) validate the hash

echo "0bfd792196ce0ec6f1c65d2a9ad00215b2926ef2c416b8d97615265194477117 actions-runner-linux-x64-2.298.2.tar.gz" | shasum -a 256 -c

# extract the installer

tar xzf ./actions-runner-linux-x64-2.298.2.tar.gz

## configure the runner application

# create the runner and start the configuration experience

./config.sh --url https://github.com/TomicaKaniski/AzureIaCGH --token "<<--TOKEN_GOES_HERE!-->>"

# last step, run it! (we will actually run it via crontab at machine restart!)

# ./run.sh

As you can see, I’ll be using crontab later to automatically (re)start my self-hosted runner.

If everything went well, you should see your runner “up and running” (😀) in the GitHub portal:

Next, I’ll use the following script to install all prerequisites for my workflow (like Azure CLI, Azure PowerShell, etc. – it really depends on your workflow and things you use):

### install (my) workflow prerequisites on self-runner machine
sudo su -

## install Azure CLI (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)
curl -sL https://aka.ms/InstallAzureCLIDeb | bash

## install PowerShell  (https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.2#installation-via-direct-download)
wget -q "https://github.com/PowerShell/PowerShell/releases/download/v7.2.6/powershell-lts_7.2.6-1.deb_amd64.deb" && dpkg -i powershell-lts_7.2.6-1.deb_amd64.deb

## install Bicep
az bicep install

## install other useful things
apt install apt-utils cron vim -y

## install Docker (https://cloudcone.com/docs/article/how-to-install-docker-on-ubuntu-22-04-20-04/)
# install prerequisites
apt install apt-transport-https curl gnupg-agent ca-certificates software-properties-common -y

# add GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add repository
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" -y

# install docker
apt install docker-ce docker-ce-cli containerd.io -y

# create docker group
newgrp docker
exit

# add current user (tomica) to docker group
sudo usermod -aG docker $USER

# configure Docker autostart
sudo systemctl enable docker
sudo systemctl restart docker
sudo systemctl status docker

## install Az module for PowerShell (https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.0.0)
pwsh
Install-Module Az -Repository PSGallery -Confirm:$false -Force
exit


### configure ./run.sh (GitHub runner application) to start automatically
crontab -e

# add the following line (uncommented, of course; and check if the path is right!):
# @reboot /home/tomica/actions-runner/run.sh

sudo reboot

### install (my) workflow prerequisites on self-runner machine

sudo su -

## install Azure CLI (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)

curl -sL https://aka.ms/InstallAzureCLIDeb | bash

## install PowerShell (https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.2#installation-via-direct-download)

wget -q "https://github.com/PowerShell/PowerShell/releases/download/v7.2.6/powershell-lts_7.2.6-1.deb_amd64.deb" && dpkg -i powershell-lts_7.2.6-1.deb_amd64.deb

## install Bicep

az bicep install

## install other useful things

apt install apt-utils cron vim -y

## install Docker (https://cloudcone.com/docs/article/how-to-install-docker-on-ubuntu-22-04-20-04/)

# install prerequisites

apt install apt-transport-https curl gnupg-agent ca-certificates software-properties-common -y

# add GPG key

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add repository

add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" -y

# install docker

apt install docker-ce docker-ce-cli containerd.io -y

# create docker group

newgrp docker

exit

# add current user (tomica) to docker group

sudo usermod -aG docker $USER

# configure Docker autostart

sudo systemctl enable docker

sudo systemctl restart docker

sudo systemctl status docker

## install Az module for PowerShell (https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.0.0)

pwsh

Install-Module Az -Repository PSGallery -Confirm:$false -Force

exit

### configure ./run.sh (GitHub runner application) to start automatically

crontab -e

# add the following line (uncommented, of course; and check if the path is right!):

# @reboot /home/tomica/actions-runner/run.sh

sudo reboot

Once this is done, my self-hosted runner hermes should be ready to run the workflow.

To try this, I need to make a slight update to my workflow file – line 12 inside the job configuration should be updated from “runs-on: ubuntu-latest” to “runs-on: self-hosted“.

So, my workflow YAML file now looks like this:

name: deploy-to-azure
on:
  push:
    branches: [ "main" ]
  workflow_dispatch:

env:
  resourceGroupName: SimpleAzureRG

jobs:
  deploy-to-azure:
    runs-on: self-hosted

    steps:
      - name: "Checkout the code from GitHub"
        uses: actions/checkout@v3

      - name: "Build Bicep code into ARM (JSON)"
        uses: Azure/cli@v1
        with:
          inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

      - name: "Login to Azure"
        uses: Azure/login@v1.4.6
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true

      - name: "Create the resource group, if needed"
        uses: Azure/powershell@v1
        with:
          inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force
          azPSVersion: latest

      - name: "Deploy the ARM template"
        uses: Azure/arm-deploy@v1.0.8
        with:
          scope: resourcegroup
          resourceGroupName: ${{ env.resourceGroupName }}
          template: main.json
          parameters: storageAccountPrefix=simplegh
          deploymentName: "gh${{ github.run_id }}"

on:

push:

branches: [ "main" ]

workflow_dispatch:

env:

resourceGroupName: SimpleAzureRG

jobs:

deploy-to-azure:

runs-on: self-hosted

steps:

- name: "Checkout the code from GitHub"

uses: actions/checkout@v3

- name: "Build Bicep code into ARM (JSON)"

uses: Azure/cli@v1

with:

inlineScript: az bicep build --file ./main.bicep --outfile ./main.json

- name: "Login to Azure"

uses: Azure/[email protected]

with:

creds: ${{ secrets.AZURE_CREDENTIALS }}

enable-AzPSSession: true

- name: "Create the resource group, if needed"

uses: Azure/powershell@v1

with:

inlineScript: New-AzResourceGroup -Name ${{ env.resourceGroupName }} -location WestEurope -Force

azPSVersion: latest

- name: "Deploy the ARM template"

uses: Azure/[email protected]

with:

scope: resourcegroup

resourceGroupName: ${{ env.resourceGroupName }}

parameters: storageAccountPrefix=simplegh

deploymentName: "gh${{ github.run_id }}"

And once I push the configuration to my GitHub, my workflow automatically starts and runs on hermes, my self-hosted runner:

If we prepared our runner right, all is good! 😊

Of course, our resources are deployed successfully:

So, this is how you can use your own, self-hosted runner, to execute your GitHub Actions (workflows).

Cheers!

What about this Bicep?

By Tomica Kaniski in Other 04/08/2021 0 Comment

You’ve probably heard about Azure Resource Manager (ARM) – the deployment and management service/layer of Azure, which enables you to manage (create, configure, delete) your Azure resources. Also, you are probably aware that ARM uses so called ARM templates – basically, JSON files that actually define the infrastructure and configuration you want to deploy to Azure (think Infrastructure as Code, IaC).

So, if you have dealt with ARM/JSON in the past, you may have been finding it difficult to start with, and somewhat complex.

Bicep is here to help.

Here is a short overview of Bicep – basically, it’s a language which enables you easier deployment of Azure resources, without messing around (too much) with JSON. To be frank, it somehow reminds of Terraform, but it’s also different. It has many cool features, immediately supports all new Azure features and APIs, can be built (converted) into .json and deployed as such or it can be deployed straight away as .bicep, doesn’t require state file, it’s open and free, has great support in Visual Studio Code and much more. And it’s still in active development!

If you’re dealing with IaC and Azure, try it.

To show you the power (and simplicity) of Bicep, here is a short example of deploying Linux virtual machine in Azure (together with a resource group, virtual network, virtual network subnet, virtual NIC and network security group), done “the old way” (in JSON, which was actually converted from Bicep… it’s easier than writing JSON from the scratch) and then done via Bicep (“the right way”? 😀).

Additionally, you’ll see that I’ve tried to break stuff into modules – with more or less sucess. 😀

The ARM/JSON way (could be done nicer/shorter, with parameters inside .parameters.json… if you know what you’re doing – this is converted from Bicep and serves just for illustrative purposes):

{
  "$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.4.451.19169",
      "templateHash": "6031449369953332876"
    }
  },
  "parameters": {
    "resourceGroupName": {
      "type": "string",
      "defaultValue": "Gym"
    },
    "location": {
      "type": "string",
      "defaultValue": "westeurope"
    }
  },
  "functions": [],
  "resources": [
    {
      "type": "Microsoft.Resources/resourceGroups",
      "apiVersion": "2021-04-01",
      "name": "[parameters('resourceGroupName')]",
      "location": "[parameters('location')]"
    },
    {
      "type": "Microsoft.Resources/deployments",
      "apiVersion": "2019-10-01",
      "name": "vm-module",
      "resourceGroup": "[parameters('resourceGroupName')]",
      "properties": {
        "expressionEvaluationOptions": {
          "scope": "inner"
        },
        "mode": "Incremental",
        "parameters": {
          "location": {
            "value": "[parameters('location')]"
          },
          "adminName": {
            "value": "tomica"
          },
          "adminSSHKey": {
            "value": "ssh-rsa <SOME SSH PUBLIC KEY>"
          },
          "machineName": {
            "value": "tkVM"
          },
          "machineSize": {
            "value": "Standard_A1_v2"
          },
          "networkName": {
            "value": "tkNet"
          },
          "networkSubnet": {
            "value": "default"
          },
          "networkSG": {
            "value": "tkNetSG"
          }
        },
        "template": {
          "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
          "contentVersion": "1.0.0.0",
          "metadata": {
            "_generator": {
              "name": "bicep",
              "version": "0.4.451.19169",
              "templateHash": "141276028935159012"
            }
          },
          "parameters": {
            "location": {
              "type": "string"
            },
            "adminName": {
              "type": "string"
            },
            "adminSSHKey": {
              "type": "string"
            },
            "machineName": {
              "type": "string"
            },
            "machineSize": {
              "type": "string"
            },
            "networkName": {
              "type": "string"
            },
            "networkSubnet": {
              "type": "string"
            },
            "networkSG": {
              "type": "string"
            }
          },
          "functions": [],
          "variables": {
            "networkNIC": "[format('{0}-nic', parameters('machineName'))]",
            "networkSubnetID": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('networkName'), parameters('networkSubnet'))]",
            "osConfiguration": {
              "disablePasswordAuthentication": true,
              "ssh": {
                "publicKeys": [
                  {
                    "path": "[format('/home/{0}/.ssh/authorized_keys', parameters('adminName'))]",
                    "keyData": "[parameters('adminSSHKey')]"
                  }
                ]
              }
            }
          },
          "resources": [
            {
              "type": "Microsoft.Network/networkSecurityGroups",
              "apiVersion": "2021-02-01",
              "name": "[parameters('networkSG')]",
              "location": "[parameters('location')]",
              "properties": {
                "securityRules": [
                  {
                    "name": "allow-ssh",
                    "properties": {
                      "priority": 1000,
                      "access": "Allow",
                      "direction": "Inbound",
                      "destinationPortRange": "22",
                      "protocol": "Tcp",
                      "sourceAddressPrefix": "*",
                      "sourcePortRange": "*",
                      "destinationAddressPrefix": "*"
                    }
                  }
                ]
              }
            },
            {
              "type": "Microsoft.Network/virtualNetworks",
              "apiVersion": "2021-02-01",
              "name": "[parameters('networkName')]",
              "location": "[parameters('location')]",
              "properties": {
                "addressSpace": {
                  "addressPrefixes": [
                    "10.0.0.0/16"
                  ]
                },
                "subnets": [
                  {
                    "name": "[parameters('networkSubnet')]",
                    "properties": {
                      "addressPrefix": "10.0.3.0/24",
                      "networkSecurityGroup": {
                        "id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"
                      }
                    }
                  }
                ]
              },
              "dependsOn": [
                "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"
              ]
            },
            {
              "type": "Microsoft.Network/networkInterfaces",
              "apiVersion": "2021-02-01",
              "name": "[variables('networkNIC')]",
              "location": "[parameters('location')]",
              "properties": {
                "ipConfigurations": [
                  {
                    "name": "ipConfig",
                    "properties": {
                      "privateIPAllocationMethod": "Dynamic",
                      "subnet": {
                        "id": "[variables('networkSubnetID')]"
                      }
                    }
                  }
                ]
              },
              "dependsOn": [
                "[resourceId('Microsoft.Network/virtualNetworks', parameters('networkName'))]"
              ]
            },
            {
              "type": "Microsoft.Compute/virtualMachines",
              "apiVersion": "2021-03-01",
              "name": "[parameters('machineName')]",
              "location": "[parameters('location')]",
              "properties": {
                "hardwareProfile": {
                  "vmSize": "[parameters('machineSize')]"
                },
                "osProfile": {
                  "computerName": "[parameters('machineName')]",
                  "adminUsername": "[parameters('adminName')]",
                  "adminPassword": "[parameters('adminSSHKey')]",
                  "linuxConfiguration": "[variables('osConfiguration')]"
                },
                "storageProfile": {
                  "imageReference": {
                    "publisher": "Canonical",
                    "offer": "UbuntuServer",
                    "sku": "18.04-LTS",
                    "version": "latest"
                  }
                },
                "networkProfile": {
                  "networkInterfaces": [
                    {
                      "id": "[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"
                    }
                  ]
                }
              },
              "dependsOn": [
                "[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"
              ]
            }
          ]
        }
      },
      "dependsOn": [
        "[subscriptionResourceId('Microsoft.Resources/resourceGroups', parameters('resourceGroupName'))]"
      ]
    }
  ]
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

{

"$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",

"contentVersion": "1.0.0.0",

"metadata": {

"_generator": {

"name": "bicep",

"version": "0.4.451.19169",

"templateHash": "6031449369953332876"

}

"parameters": {

"resourceGroupName": {

"type": "string",

"defaultValue": "Gym"

"location": {

"type": "string",

"defaultValue": "westeurope"

}

"functions": [],

"resources": [

{

"type": "Microsoft.Resources/resourceGroups",

"apiVersion": "2021-04-01",

"name": "[parameters('resourceGroupName')]",

"location": "[parameters('location')]"

{

"type": "Microsoft.Resources/deployments",

"apiVersion": "2019-10-01",

"name": "vm-module",

"resourceGroup": "[parameters('resourceGroupName')]",

"properties": {

"expressionEvaluationOptions": {

"scope": "inner"

"mode": "Incremental",

"parameters": {

"location": {

"value": "[parameters('location')]"

"adminName": {

"value": "tomica"

"adminSSHKey": {

"value": "ssh-rsa <SOME SSH PUBLIC KEY>"

"machineName": {

"value": "tkVM"

"machineSize": {

"value": "Standard_A1_v2"

"networkName": {

"value": "tkNet"

"networkSubnet": {

"value": "default"

"networkSG": {

"value": "tkNetSG"

}

"template": {

"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",

"contentVersion": "1.0.0.0",

"metadata": {

"_generator": {

"name": "bicep",

"version": "0.4.451.19169",

"templateHash": "141276028935159012"

}

"parameters": {

"location": {

"type": "string"

"adminName": {

"type": "string"

"adminSSHKey": {

"type": "string"

"machineName": {

"type": "string"

"machineSize": {

"type": "string"

"networkName": {

"type": "string"

"networkSubnet": {

"type": "string"

"networkSG": {

"type": "string"

}

"functions": [],

"variables": {

"networkNIC": "[format('{0}-nic', parameters('machineName'))]",

"networkSubnetID": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('networkName'), parameters('networkSubnet'))]",

"osConfiguration": {

"disablePasswordAuthentication": true,

"ssh": {

"publicKeys": [

{

"path": "[format('/home/{0}/.ssh/authorized_keys', parameters('adminName'))]",

"keyData": "[parameters('adminSSHKey')]"

}

]

}

"resources": [

{

"type": "Microsoft.Network/networkSecurityGroups",

"apiVersion": "2021-02-01",

"name": "[parameters('networkSG')]",

"location": "[parameters('location')]",

"properties": {

"securityRules": [

{

"name": "allow-ssh",

"properties": {

"priority": 1000,

"access": "Allow",

"direction": "Inbound",

"destinationPortRange": "22",

"protocol": "Tcp",

"sourceAddressPrefix": "*",

"sourcePortRange": "*",

"destinationAddressPrefix": "*"

}

]

}

{

"type": "Microsoft.Network/virtualNetworks",

"apiVersion": "2021-02-01",

"name": "[parameters('networkName')]",

"location": "[parameters('location')]",

"properties": {

"addressSpace": {

"addressPrefixes": [

"10.0.0.0/16"

]

"subnets": [

{

"name": "[parameters('networkSubnet')]",

"properties": {

"addressPrefix": "10.0.3.0/24",

"networkSecurityGroup": {

"id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"

}

]

"dependsOn": [

"[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"

]

{

"type": "Microsoft.Network/networkInterfaces",

"apiVersion": "2021-02-01",

"name": "[variables('networkNIC')]",

"location": "[parameters('location')]",

"properties": {

"ipConfigurations": [

{

"name": "ipConfig",

"properties": {

"privateIPAllocationMethod": "Dynamic",

"subnet": {

"id": "[variables('networkSubnetID')]"

}

]

"dependsOn": [

"[resourceId('Microsoft.Network/virtualNetworks', parameters('networkName'))]"

]

{

"type": "Microsoft.Compute/virtualMachines",

"apiVersion": "2021-03-01",

"name": "[parameters('machineName')]",

"location": "[parameters('location')]",

"properties": {

"hardwareProfile": {

"vmSize": "[parameters('machineSize')]"

"osProfile": {

"computerName": "[parameters('machineName')]",

"adminUsername": "[parameters('adminName')]",

"adminPassword": "[parameters('adminSSHKey')]",

"linuxConfiguration": "[variables('osConfiguration')]"

"storageProfile": {

"imageReference": {

"publisher": "Canonical",

"offer": "UbuntuServer",

"sku": "18.04-LTS",

"version": "latest"

}

"networkProfile": {

"networkInterfaces": [

{

"id": "[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"

}

]

}

"dependsOn": [

"[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"

]

}

]

}

"dependsOn": [

"[subscriptionResourceId('Microsoft.Resources/resourceGroups', parameters('resourceGroupName'))]"

]

}

]

}

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.4.451.19169",
      "templateHash": "141276028935159012"
    }
  },
  "parameters": {
    "location": {
      "type": "string"
    },
    "adminName": {
      "type": "string"
    },
    "adminSSHKey": {
      "type": "string"
    },
    "machineName": {
      "type": "string"
    },
    "machineSize": {
      "type": "string"
    },
    "networkName": {
      "type": "string"
    },
    "networkSubnet": {
      "type": "string"
    },
    "networkSG": {
      "type": "string"
    }
  },
  "functions": [],
  "variables": {
    "networkNIC": "[format('{0}-nic', parameters('machineName'))]",
    "networkSubnetID": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('networkName'), parameters('networkSubnet'))]",
    "osConfiguration": {
      "disablePasswordAuthentication": true,
      "ssh": {
        "publicKeys": [
          {
            "path": "[format('/home/{0}/.ssh/authorized_keys', parameters('adminName'))]",
            "keyData": "[parameters('adminSSHKey')]"
          }
        ]
      }
    }
  },
  "resources": [
    {
      "type": "Microsoft.Network/networkSecurityGroups",
      "apiVersion": "2021-02-01",
      "name": "[parameters('networkSG')]",
      "location": "[parameters('location')]",
      "properties": {
        "securityRules": [
          {
            "name": "allow-ssh",
            "properties": {
              "priority": 1000,
              "access": "Allow",
              "direction": "Inbound",
              "destinationPortRange": "22",
              "protocol": "Tcp",
              "sourceAddressPrefix": "*",
              "sourcePortRange": "*",
              "destinationAddressPrefix": "*"
            }
          }
        ]
      }
    },
    {
      "type": "Microsoft.Network/virtualNetworks",
      "apiVersion": "2021-02-01",
      "name": "[parameters('networkName')]",
      "location": "[parameters('location')]",
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "10.0.0.0/16"
          ]
        },
        "subnets": [
          {
            "name": "[parameters('networkSubnet')]",
            "properties": {
              "addressPrefix": "10.0.3.0/24",
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"
              }
            }
          }
        ]
      },
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"
      ]
    },
    {
      "type": "Microsoft.Network/networkInterfaces",
      "apiVersion": "2021-02-01",
      "name": "[variables('networkNIC')]",
      "location": "[parameters('location')]",
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipConfig",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[variables('networkSubnetID')]"
              }
            }
          }
        ]
      },
      "dependsOn": [
        "[resourceId('Microsoft.Network/virtualNetworks', parameters('networkName'))]"
      ]
    },
    {
      "type": "Microsoft.Compute/virtualMachines",
      "apiVersion": "2021-03-01",
      "name": "[parameters('machineName')]",
      "location": "[parameters('location')]",
      "properties": {
        "hardwareProfile": {
          "vmSize": "[parameters('machineSize')]"
        },
        "osProfile": {
          "computerName": "[parameters('machineName')]",
          "adminUsername": "[parameters('adminName')]",
          "adminPassword": "[parameters('adminSSHKey')]",
          "linuxConfiguration": "[variables('osConfiguration')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "Canonical",
            "offer": "UbuntuServer",
            "sku": "18.04-LTS",
            "version": "latest"
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"
            }
          ]
        }
      },
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"
      ]
    }
  ]
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

{

"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",

"contentVersion": "1.0.0.0",

"metadata": {

"_generator": {

"name": "bicep",

"version": "0.4.451.19169",

"templateHash": "141276028935159012"

}

"parameters": {

"location": {

"type": "string"

"adminName": {

"type": "string"

"adminSSHKey": {

"type": "string"

"machineName": {

"type": "string"

"machineSize": {

"type": "string"

"networkName": {

"type": "string"

"networkSubnet": {

"type": "string"

"networkSG": {

"type": "string"

}

"functions": [],

"variables": {

"networkNIC": "[format('{0}-nic', parameters('machineName'))]",

"networkSubnetID": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('networkName'), parameters('networkSubnet'))]",

"osConfiguration": {

"disablePasswordAuthentication": true,

"ssh": {

"publicKeys": [

{

"path": "[format('/home/{0}/.ssh/authorized_keys', parameters('adminName'))]",

"keyData": "[parameters('adminSSHKey')]"

}

]

}

"resources": [

{

"type": "Microsoft.Network/networkSecurityGroups",

"apiVersion": "2021-02-01",

"name": "[parameters('networkSG')]",

"location": "[parameters('location')]",

"properties": {

"securityRules": [

{

"name": "allow-ssh",

"properties": {

"priority": 1000,

"access": "Allow",

"direction": "Inbound",

"destinationPortRange": "22",

"protocol": "Tcp",

"sourceAddressPrefix": "*",

"sourcePortRange": "*",

"destinationAddressPrefix": "*"

}

]

}

{

"type": "Microsoft.Network/virtualNetworks",

"apiVersion": "2021-02-01",

"name": "[parameters('networkName')]",

"location": "[parameters('location')]",

"properties": {

"addressSpace": {

"addressPrefixes": [

"10.0.0.0/16"

]

"subnets": [

{

"name": "[parameters('networkSubnet')]",

"properties": {

"addressPrefix": "10.0.3.0/24",

"networkSecurityGroup": {

"id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"

}

]

"dependsOn": [

"[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSG'))]"

]

{

"type": "Microsoft.Network/networkInterfaces",

"apiVersion": "2021-02-01",

"name": "[variables('networkNIC')]",

"location": "[parameters('location')]",

"properties": {

"ipConfigurations": [

{

"name": "ipConfig",

"properties": {

"privateIPAllocationMethod": "Dynamic",

"subnet": {

"id": "[variables('networkSubnetID')]"

}

]

"dependsOn": [

"[resourceId('Microsoft.Network/virtualNetworks', parameters('networkName'))]"

]

{

"type": "Microsoft.Compute/virtualMachines",

"apiVersion": "2021-03-01",

"name": "[parameters('machineName')]",

"location": "[parameters('location')]",

"properties": {

"hardwareProfile": {

"vmSize": "[parameters('machineSize')]"

"osProfile": {

"computerName": "[parameters('machineName')]",

"adminUsername": "[parameters('adminName')]",

"adminPassword": "[parameters('adminSSHKey')]",

"linuxConfiguration": "[variables('osConfiguration')]"

"storageProfile": {

"imageReference": {

"publisher": "Canonical",

"offer": "UbuntuServer",

"sku": "18.04-LTS",

"version": "latest"

}

"networkProfile": {

"networkInterfaces": [

{

"id": "[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"

}

]

}

"dependsOn": [

"[resourceId('Microsoft.Network/networkInterfaces', variables('networkNIC'))]"

]

}

]

}

The Bicep way:

// main.bicep
targetScope = 'subscription'

param resourceGroupName string = 'Gym'
param location string = 'westeurope'

resource rg 'Microsoft.Resources/resourceGroups@2021-04-01' = {
  name: resourceGroupName
  location: location
}

module vm 'vm.bicep' = {
  name: 'vm-module'
  scope: rg
  params: {
    location: location
    adminName: 'tomica'
    adminSSHKey: 'ssh-rsa <SOME SSH PUBLIC KEY>'
    machineName: 'tkVM'
    machineSize: 'Standard_A1_v2'
    networkName: 'tkNet'
    networkSubnet: 'default'
    networkSG: 'tkNetSG'
  }
}

// main.bicep

targetScope = 'subscription'

param resourceGroupName string = 'Gym'

param location string = 'westeurope'

resource rg 'Microsoft.Resources/resourceGroups@2021-04-01' = {

location: location

}

module vm 'vm.bicep' = {

scope: rg

params: {

location: location

adminName: 'tomica'

adminSSHKey: 'ssh-rsa <SOME SSH PUBLIC KEY>'

machineName: 'tkVM'

machineSize: 'Standard_A1_v2'

networkName: 'tkNet'

networkSubnet: 'default'

networkSG: 'tkNetSG'

}

// vm.bicep
targetScope = 'resourceGroup'

param location string
param adminName string
param adminSSHKey string
param machineName string
param machineSize string
param networkName string
param networkSubnet string
param networkSG string

var networkNIC = '${machineName}-nic'
var networkSubnetID = resourceId('Microsoft.Network/virtualNetworks/subnets', networkName, networkSubnet)
var osConfiguration = {
  disablePasswordAuthentication: true
  ssh: {
    publicKeys: [
      {
        path: '/home/${adminName}/.ssh/authorized_keys'
        keyData: adminSSHKey
      }
    ]
  }
}

resource networkSecurityGroupName 'Microsoft.Network/networkSecurityGroups@2021-02-01' = {
  name: networkSG
  location: location
  properties: {
    securityRules: [
      {
        name: 'allow-ssh'
        properties: {
          priority: 1000
          access: 'Allow'
          direction: 'Inbound'
          destinationPortRange: '22'
          protocol: 'Tcp'
          sourceAddressPrefix: '*'
          sourcePortRange: '*'
          destinationAddressPrefix: '*'
        }
      }
    ]
  }
}

resource virtualNetworkName 'Microsoft.Network/virtualNetworks@2021-02-01' = {
  name: networkName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [
      {
        name: networkSubnet
        properties: {
          addressPrefix: '10.0.3.0/24'
          networkSecurityGroup: {
            id: networkSecurityGroupName.id
          }
        }
      }
    ]
  }
}

resource nicName 'Microsoft.Network/networkInterfaces@2021-02-01' = {
  name: networkNIC
  location: location
  properties: {
    ipConfigurations: [
      {
        name: 'ipConfig'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          subnet: {
            id: networkSubnetID
          }
        }
      }
    ]
  }
  dependsOn: [
    virtualNetworkName
  ]
}

resource vmName 'Microsoft.Compute/virtualMachines@2021-03-01' = {
  name: machineName
  location: location
  properties: {
    hardwareProfile: {
      vmSize: machineSize
    }
    osProfile: {
      computerName: machineName
      adminUsername: adminName
      adminPassword: adminSSHKey
      linuxConfiguration: osConfiguration
    }
    storageProfile: {
      imageReference: {
        publisher: 'Canonical'
        offer: 'UbuntuServer'
        sku: '18.04-LTS'
        version: 'latest'
      }
    }
    networkProfile: {
      networkInterfaces: [
        {
          id: nicName.id
        }
      ]
    }
  }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

// vm.bicep

targetScope = 'resourceGroup'

param location string

param adminName string

param adminSSHKey string

param machineName string

param machineSize string

param networkName string

param networkSubnet string

param networkSG string

var networkNIC = '${machineName}-nic'

var networkSubnetID = resourceId('Microsoft.Network/virtualNetworks/subnets', networkName, networkSubnet)

var osConfiguration = {

disablePasswordAuthentication: true

ssh: {

publicKeys: [

{

path: '/home/${adminName}/.ssh/authorized_keys'

keyData: adminSSHKey

}

]

}

resource networkSecurityGroupName 'Microsoft.Network/networkSecurityGroups@2021-02-01' = {

location: location

properties: {

securityRules: [

{

properties: {

priority: 1000

access: 'Allow'

direction: 'Inbound'

destinationPortRange: '22'

protocol: 'Tcp'

sourceAddressPrefix: '*'

sourcePortRange: '*'

destinationAddressPrefix: '*'

}

]

}

resource virtualNetworkName 'Microsoft.Network/virtualNetworks@2021-02-01' = {

location: location

properties: {

addressSpace: {

addressPrefixes: [

'10.0.0.0/16'

]

}

subnets: [

{

properties: {

addressPrefix: '10.0.3.0/24'

networkSecurityGroup: {

id: networkSecurityGroupName.id

}

]

}

resource nicName 'Microsoft.Network/networkInterfaces@2021-02-01' = {

location: location

properties: {

ipConfigurations: [

{

properties: {

privateIPAllocationMethod: 'Dynamic'

subnet: {

id: networkSubnetID

}

]

}

dependsOn: [

virtualNetworkName

]

}

resource vmName 'Microsoft.Compute/virtualMachines@2021-03-01' = {

location: location

properties: {

hardwareProfile: {

vmSize: machineSize

}

osProfile: {

computerName: machineName

adminUsername: adminName

adminPassword: adminSSHKey

linuxConfiguration: osConfiguration

}

storageProfile: {

imageReference: {

publisher: 'Canonical'

offer: 'UbuntuServer'

sku: '18.04-LTS'

version: 'latest'

}

networkProfile: {

networkInterfaces: [

{

id: nicName.id

}

]

}

Bicep seems a bit easier to read and shorter, right (while still doing basically the same thing)? 😀

If we deploy the .bicep files above (note that I’m deploying the “raw” .bicep file directly – which is cool!):

New-AzDeployment -TemplateFile ".\main.bicep" -DeploymentName "vm-example" -Location "West Europe"

1	New-AzDeployment -TemplateFile ".\main.bicep" -DeploymentName "vm-example" -Location "West Europe"

We finally get our resources:

So, where should you start if you’re new to Bicep?

I would certainly recommend starting with free and official Deploy and manage resources in Azure by using Bicep learning path on Microsoft Learn.

After that, you can probably pick up Freek Berson’s book Getting started with Bicep: Infrastructure as Code on Azure (first and only book on Bicep that I know of – really liked it because of the simple (yet effective) examples with storage accounts, it connects everything and flows naturally – building up “brick by brick” and not “jumping around”, just to show off what Bicep can do).

Another great resource are also the Bicep examples – there’s plenty to learn from them too!

Of course, you’ll also need to practice – install the Azure CLI or Azure PowerShell module, add Bicep and use Visual Studio Code for your first steps with creating, deleting, configuring and breaking stuff… powered by Bicep! 😀

Cheers!

Yet another “Kubernetes with Raspberry Pi” post

By Tomica Kaniski in Other 08/12/2020 4 Comments

There’s a ton of the tutorials on how to get Kubernetes installed onto your Raspberry Pi, so… let’s write another one. 😊

As mentioned in my last post, I’ve found my forgotten Raspberry Pi, and played around with installing and configuring Raspbian Buster on it.

Today, I wanted to check if it will be possible to install Kubernetes onto such small machine – they are many articles on the “widest of the world’s webs” that say “Yes, it can be done!“, so I’ve decided to give it a try! And I chose to follow one of them (seemed like a nice reference).

As you remember, I’m starting with a cleanly installed (and just slightly customized) Raspbian Buster and building it from there.

And I’ll be using kubeadm for installing my cluster.

So, once I had at least two machines (my Raspberry Pi for the “control plane” and Ubuntu 20.04 LTS Hyper-V virtual machine as the “node” – you can read more about it here), I prepared them like this:

install Docker (in my case)
change the default cgroups driver for Docker to systemd
add cgroups limit support (for my Raspberry Pi 3)
configure iptables
disable swap (this one was a bit challenging)
prepare for Kubernetes installation (source, keys, kubeadm)
install Kubernetes “control plane”
add flannel
add a node to the cluster
test with some workload

One thing that bothered me (on Buster) was disabling swap in a way that it also stays disabled after a reboot (I know, it’s the details that eventually get you) – after a while, I’ve stumbled on this forum post and the solution provided by “powerpete” did the trick! Thank you, @powerpete! 😊

And finally, details about the each step are here (outputs are commented and somewhat redacted/condensed):

# install Docker
sudo apt install -yqq docker.io
# The following additional packages will be installed:
#   cgroupfs-mount git git-man libcurl3-gnutls liberror-perl libintl-perl libintl-xs-perl libltdl7 libmodule-find-perl
#   libmodule-scandeps-perl libnspr4 libnss3 libproc-processtable-perl libsort-naturally-perl libterm-readkey-perl
#   needrestart runc tini
# ...
# The following NEW packages will be installed:
#   cgroupfs-mount docker.io git git-man libcurl3-gnutls liberror-perl libintl-perl libintl-xs-perl libltdl7
#   libmodule-find-perl libmodule-scandeps-perl libnspr4 libnss3 libproc-processtable-perl libsort-naturally-perl
#   libterm-readkey-perl needrestart runc tini
# 0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded.
# ...

# check docker info and change the default cgroups driver for Docker to systemd
sudo docker info
# ...
# Server Version: 18.09.1
# Storage Driver: overlay2
# ...
# Logging Driver: json-file
# Cgroup Driver: cgroupfs
# ...
# Kernel Version: 5.4.79-v7+
# Operating System: Raspbian GNU/Linux 10 (buster)
# OSType: linux
# Architecture: armv7l
# CPUs: 4
# Total Memory: 974.4MiB
# Name: pimaster
# ...
#
# WARNING: No memory limit support
# WARNING: No swap limit support
# WARNING: No kernel memory limit support
# WARNING: No oom kill disable support

# create/edit the /etc/docker/daemon.json file (with the following content, uncommented) and restart docker service
sudo nano /etc/docker/daemon.json
# {
#   "exec-opts": ["native.cgroupdriver=systemd"],
#   "log-driver": "json-file",
#   "log-opts": {
#     "max-size": "100m"
#   },
#   "storage-driver": "overlay2"
# }

sudo systemctl restart docker

# docker info now showing the changed cgroup driver
sudo docker info
# ...
# Cgroup Driver: systemd
# ...
# 
# WARNING: No memory limit support
# WARNING: No swap limit support
# WARNING: No kernel memory limit support
# WARNING: No oom kill disable support

# add cgroups limit support and reboot
sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/cmdline.txt
# console=tty1 root=PARTUUID=03f5b65e-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1

sudo reboot

# recheck docker info (warnings should be gone)
sudo docker info
# ...
# Storage Driver: overlay2
# ...
# Logging Driver: json-file
# Cgroup Driver: systemd
# ...
# Operating System: Raspbian GNU/Linux 10 (buster)
# ...

# configure iptables
cat << EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

# configure iptables (add lines to the file, uncommented) and apply the configuration
sudo nano /etc/sysctl.d/k8s.conf
# net.bridge.bridge-nf-call-ip6tables = 1
# net.bridge.bridge-nf-call-iptables = 1

sudo sysctl --system
* Applying /etc/sysctl.d/98-rpi.conf ...
# kernel.printk = 3 4 1 3
# vm.min_free_kbytes = 16384
# * Applying /etc/sysctl.d/99-sysctl.conf ...
# * Applying /etc/sysctl.d/k8s.conf ...
# net.bridge.bridge-nf-call-ip6tables = 1
# net.bridge.bridge-nf-call-iptables = 1
# * Applying /etc/sysctl.d/protect-links.conf ...
# fs.protected_hardlinks = 1
# fs.protected_symlinks = 1
# * Applying /etc/sysctl.conf ...

# disable swap
# check current swap utilization/settings
free -m
#               total        used        free      shared  buff/cache   available
# Mem:            974          82         709           6         182         834
# Swap:            99           0          99

sudo cat /etc/fstab
# proc            /proc           proc    defaults          0       0
# PARTUUID=03f5b65e-01  /boot           vfat    defaults          0       2
# PARTUUID=03f5b65e-02  /               ext4    defaults,noatime  0       1

# disable swap (note - disabling the service as well!)
sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove
sudo systemctl disable dphys-swapfile.service
# Synchronizing state of dphys-swapfile.service with SysV service script with /lib/systemd/systemd-sysv-install.
# Executing: /lib/systemd/systemd-sysv-install disable dphys-swapfile
# Removed /etc/systemd/system/multi-user.target.wants/dphys-swapfile.service.

# optionally - reboot & recheck
free -m
#               total        used        free      shared  buff/cache   available
# Mem:            974          79         745          12         149         832
# Swap:             0           0           0

# prepare for Kubernetes installation (source, keys, kubeadm) and "lock" the kubelet, kubeadm and kubectl versions
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
# OK

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
# deb https://apt.kubernetes.io/ kubernetes-xenial main

sudo apt -qq update
# All packages are up to date.

sudo apt install -yqq kubeadm
# The following additional packages will be installed:
#   conntrack cri-tools ebtables kubectl kubelet kubernetes-cni socat
# Suggested packages:
#   nftables
# The following NEW packages will be installed:
#   conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat
# 0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
# ...

sudo apt-mark hold kubelet kubeadm kubectl
# kubelet set on hold.
# kubeadm set on hold.
# kubectl set on hold.

# install Kubernetes control plane
# generate token for the installation
TOKEN=$(sudo kubeadm token generate)
echo $TOKEN
# rqyytr.f3pvvz2yepq51b7h

sudo kubeadm init --token=${TOKEN} --kubernetes-version=v1.19.4 --pod-network-cidr=10.244.0.0/16
# W1208 16:59:02.624049     781 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
# [init] Using Kubernetes version: v1.19.4
# [preflight] Running pre-flight checks
#         [WARNING SystemVerification]: missing optional cgroups: hugetlb
# [preflight] Pulling images required for setting up a Kubernetes cluster
# [preflight] This might take a minute or two, depending on the speed of your internet connection
# [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
# [certs] Using certificateDir folder "/etc/kubernetes/pki"
# [certs] Generating "ca" certificate and key
# [certs] Generating "apiserver" certificate and key
# [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local pimaster] and IPs [10.96.0.1 192.168.12.101]
# [certs] Generating "apiserver-kubelet-client" certificate and key
# [certs] Generating "front-proxy-ca" certificate and key
# [certs] Generating "front-proxy-client" certificate and key
# [certs] Generating "etcd/ca" certificate and key
# [certs] Generating "etcd/server" certificate and key
# [certs] etcd/server serving cert is signed for DNS names [localhost pimaster] and IPs [192.168.12.101 127.0.0.1 ::1]
# [certs] Generating "etcd/peer" certificate and key
# [certs] etcd/peer serving cert is signed for DNS names [localhost pimaster] and IPs [192.168.12.101 127.0.0.1 ::1]
# [certs] Generating "etcd/healthcheck-client" certificate and key
# [certs] Generating "apiserver-etcd-client" certificate and key
# [certs] Generating "sa" key and public key
# [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
# [kubeconfig] Writing "admin.conf" kubeconfig file
# [kubeconfig] Writing "kubelet.conf" kubeconfig file
# [kubeconfig] Writing "controller-manager.conf" kubeconfig file
# [kubeconfig] Writing "scheduler.conf" kubeconfig file
# [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
# [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
# [kubelet-start] Starting the kubelet
# [control-plane] Using manifest folder "/etc/kubernetes/manifests"
# [control-plane] Creating static Pod manifest for "kube-apiserver"
# [control-plane] Creating static Pod manifest for "kube-controller-manager"
# [control-plane] Creating static Pod manifest for "kube-scheduler"
# [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
# [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
# [kubelet-check] Initial timeout of 40s passed.
# [kubelet-check] It seems like the kubelet isn't running or healthy.
# [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
# [kubelet-check] It seems like the kubelet isn't running or healthy.
# [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
# [kubelet-check] It seems like the kubelet isn't running or healthy.
# [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
# [apiclient] All control plane components are healthy after 212.227971 seconds
# [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
# [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
# [upload-certs] Skipping phase. Please see --upload-certs
# [mark-control-plane] Marking the node pimaster as control-plane by adding the label "node-role.kubernetes.io/master=''"
# [mark-control-plane] Marking the node pimaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
# [bootstrap-token] Using token: rqyytr.f3pvvz2yepq51b7h
# [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
# [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
# [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
# [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
# [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
# [addons] Applied essential addon: CoreDNS
# [addons] Applied essential addon: kube-proxy
#
# Your Kubernetes control-plane has initialized successfully!
# 
# To start using your cluster, you need to run the following as a regular user:
#   mkdir -p $HOME/.kube
#   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#   sudo chown $(id -u):$(id -g) $HOME/.kube/config
#
# You should now deploy a pod network to the cluster.
# Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
#   https://kubernetes.io/docs/concepts/cluster-administration/addons/
#
# Then you can join any number of worker nodes by running the following on each as root:
#
# kubeadm join 192.168.12.101:6443 --token rqyytr.f3pvvz2yepq51b7h \
#     --discovery-token-ca-cert-hash sha256:5b04368aaee148fa3eb8c6aed3c3bc8041248590e2055a81617d16d8f796bb77

# start using the cluster
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# check node status (something's missing - flannel)
kubectl get nodes
# NAME       STATUS     ROLES    AGE    VERSION
# pimaster   NotReady   master   14m    v1.19.4

# install flannel network plugin
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# podsecuritypolicy.policy/psp.flannel.unprivileged created
# clusterrole.rbac.authorization.k8s.io/flannel created
# clusterrolebinding.rbac.authorization.k8s.io/flannel created
# serviceaccount/flannel created
# configmap/kube-flannel-cfg created
# daemonset.apps/kube-flannel-ds created

# and now we're ready (check under STATUS)
kubectl get nodes
# NAME       STATUS     ROLES    AGE     VERSION
# pimaster   Ready      master   17m     v1.19.4

# add node to the cluster (pinode, 192.168.12.102)
sudo kubeadm join 192.168.12.101:6443 --token rqyytr.f3pvvz2yepq51b7h --discovery-token-ca-cert-hash sha256:5b04368aaee148fa3eb8c6aed3c3bc8041248590e2055a81617d16d8f796bb77
# [preflight] Running pre-flight checks
# [preflight] Reading configuration from the cluster...
# [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
# [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
# [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
# [kubelet-start] Starting the kubelet
# [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
#
# This node has joined the cluster:
# * Certificate signing request was sent to apiserver and a response was received.
# * The Kubelet was informed of the new secure connection details.
#
# Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

# check nodes in the cluster (check the details... nice)
kubectl get nodes -o wide
# NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
# pimaster   Ready    master   18m     v1.19.4   192.168.12.101           Raspbian GNU/Linux 10 (buster)   5.4.79-v7+         docker://18.9.1
# pinode     Ready       6m15s   v1.19.4   192.168.12.102           Ubuntu 20.04.1 LTS               5.4.0-56-generic   docker://19.3.8

# test with some workload
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-deployment.yaml
# deployment.apps/redis-master created

kubectl get pods
# NAME                           READY   STATUS    RESTARTS   AGE
# redis-master-f46ff57fd-9m9g9   1/1     Running   0          37s

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-service.yaml
# service/redis-master created

kubectl get service
# NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
# kubernetes     ClusterIP   10.96.0.1                443/TCP    22m
# redis-master   ClusterIP   10.100.245.207           6379/TCP   4s

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-slave-deployment.yaml
# deployment.apps/redis-slave created

kubectl get pods
# NAME                           READY   STATUS    RESTARTS   AGE
# redis-master-f46ff57fd-9m9g9   1/1     Running   0          2m30s
# redis-slave-bbc7f655d-dz895    1/1     Running   0          12s
# redis-slave-bbc7f655d-slnhv    1/1     Running   0          12s

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-slave-service.yaml
# service/redis-slave created

kubectl get services
# NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
# kubernetes     ClusterIP   10.96.0.1                443/TCP    23m
# redis-master   ClusterIP   10.100.245.207           6379/TCP   91s
# redis-slave    ClusterIP   10.101.165.188           6379/TCP   13s

kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-deployment.yaml
# deployment.apps/frontend created

kubectl get pods -l app=guestbook -l tier=frontend
# NAME                        READY   STATUS    RESTARTS   AGE
# frontend-6c6d6dfd4d-945hd   1/1     Running   0          7s
# frontend-6c6d6dfd4d-mm7vm   1/1     Running   0          7s
# frontend-6c6d6dfd4d-wllpx   1/1     Running   0          7s

kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-service.yaml
# service/frontend created

kubectl get services
# NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
# frontend       NodePort    10.101.140.217           80:31597/TCP   4s
# kubernetes     ClusterIP   10.96.0.1                443/TCP        24m
# redis-master   ClusterIP   10.100.245.207           6379/TCP       2m45s
# redis-slave    ClusterIP   10.101.165.188           6379/TCP       87s

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

# install Docker

sudo apt install -yqq docker.io

# The following additional packages will be installed:

# cgroupfs-mount git git-man libcurl3-gnutls liberror-perl libintl-perl libintl-xs-perl libltdl7 libmodule-find-perl

# libmodule-scandeps-perl libnspr4 libnss3 libproc-processtable-perl libsort-naturally-perl libterm-readkey-perl

# needrestart runc tini

# ...

# The following NEW packages will be installed:

# cgroupfs-mount docker.io git git-man libcurl3-gnutls liberror-perl libintl-perl libintl-xs-perl libltdl7

# libmodule-find-perl libmodule-scandeps-perl libnspr4 libnss3 libproc-processtable-perl libsort-naturally-perl

# libterm-readkey-perl needrestart runc tini

# 0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded.

# ...

# check docker info and change the default cgroups driver for Docker to systemd

sudo docker info

# ...

# Server Version: 18.09.1

# Storage Driver: overlay2

# ...

# Logging Driver: json-file

# Cgroup Driver: cgroupfs

# ...

# Kernel Version: 5.4.79-v7+

# Operating System: Raspbian GNU/Linux 10 (buster)

# OSType: linux

# Architecture: armv7l

# CPUs: 4

# Total Memory: 974.4MiB

# Name: pimaster

# ...

# WARNING: No memory limit support

# WARNING: No swap limit support

# WARNING: No kernel memory limit support

# WARNING: No oom kill disable support

# create/edit the /etc/docker/daemon.json file (with the following content, uncommented) and restart docker service

sudo nano /etc/docker/daemon.json

# {

# "exec-opts": ["native.cgroupdriver=systemd"],

# "log-driver": "json-file",

# "log-opts": {

# "max-size": "100m"

# },

# "storage-driver": "overlay2"

# }

sudo systemctl restart docker

# docker info now showing the changed cgroup driver

sudo docker info

# ...

# Cgroup Driver: systemd

# ...

# WARNING: No memory limit support

# WARNING: No swap limit support

# WARNING: No kernel memory limit support

# WARNING: No oom kill disable support

# add cgroups limit support and reboot

sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/cmdline.txt

# console=tty1 root=PARTUUID=03f5b65e-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1

sudo reboot

# recheck docker info (warnings should be gone)

sudo docker info

# ...

# Storage Driver: overlay2

# ...

# Logging Driver: json-file

# Cgroup Driver: systemd

# ...

# Operating System: Raspbian GNU/Linux 10 (buster)

# ...

# configure iptables

cat << EOF | sudo tee /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

# configure iptables (add lines to the file, uncommented) and apply the configuration

sudo nano /etc/sysctl.d/k8s.conf

# net.bridge.bridge-nf-call-ip6tables = 1

# net.bridge.bridge-nf-call-iptables = 1

sudo sysctl --system

* Applying /etc/sysctl.d/98-rpi.conf ...

# kernel.printk = 3 4 1 3

# vm.min_free_kbytes = 16384

# * Applying /etc/sysctl.d/99-sysctl.conf ...

# * Applying /etc/sysctl.d/k8s.conf ...

# net.bridge.bridge-nf-call-ip6tables = 1

# net.bridge.bridge-nf-call-iptables = 1

# * Applying /etc/sysctl.d/protect-links.conf ...

# fs.protected_hardlinks = 1

# fs.protected_symlinks = 1

# * Applying /etc/sysctl.conf ...

# disable swap

# check current swap utilization/settings

free -m

# total used free shared buff/cache available

# Mem: 974 82 709 6 182 834

# Swap: 99 0 99

sudo cat /etc/fstab

# proc /proc proc defaults 0 0

# PARTUUID=03f5b65e-01 /boot vfat defaults 0 2

# PARTUUID=03f5b65e-02 / ext4 defaults,noatime 0 1

# disable swap (note - disabling the service as well!)

sudo dphys-swapfile swapoff

sudo dphys-swapfile uninstall

sudo update-rc.d dphys-swapfile remove

sudo systemctl disable dphys-swapfile.service

# Synchronizing state of dphys-swapfile.service with SysV service script with /lib/systemd/systemd-sysv-install.

# Executing: /lib/systemd/systemd-sysv-install disable dphys-swapfile

# Removed /etc/systemd/system/multi-user.target.wants/dphys-swapfile.service.

# optionally - reboot & recheck

free -m

# total used free shared buff/cache available

# Mem: 974 79 745 12 149 832

# Swap: 0 0 0

# prepare for Kubernetes installation (source, keys, kubeadm) and "lock" the kubelet, kubeadm and kubectl versions

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

# OK

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# deb https://apt.kubernetes.io/ kubernetes-xenial main

sudo apt -qq update

# All packages are up to date.

sudo apt install -yqq kubeadm

# The following additional packages will be installed:

# conntrack cri-tools ebtables kubectl kubelet kubernetes-cni socat

# Suggested packages:

# nftables

# The following NEW packages will be installed:

# conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat

# 0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.

# ...

sudo apt-mark hold kubelet kubeadm kubectl

# kubelet set on hold.

# kubeadm set on hold.

# kubectl set on hold.

# install Kubernetes control plane

# generate token for the installation

TOKEN=$(sudo kubeadm token generate)

echo $TOKEN

# rqyytr.f3pvvz2yepq51b7h

sudo kubeadm init --token=${TOKEN} --kubernetes-version=v1.19.4 --pod-network-cidr=10.244.0.0/16

# W1208 16:59:02.624049 781 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]

# [init] Using Kubernetes version: v1.19.4

# [preflight] Running pre-flight checks

# [WARNING SystemVerification]: missing optional cgroups: hugetlb

# [preflight] Pulling images required for setting up a Kubernetes cluster

# [preflight] This might take a minute or two, depending on the speed of your internet connection

# [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

# [certs] Using certificateDir folder "/etc/kubernetes/pki"

# [certs] Generating "ca" certificate and key

# [certs] Generating "apiserver" certificate and key

# [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local pimaster] and IPs [10.96.0.1 192.168.12.101]

# [certs] Generating "apiserver-kubelet-client" certificate and key

# [certs] Generating "front-proxy-ca" certificate and key

# [certs] Generating "front-proxy-client" certificate and key

# [certs] Generating "etcd/ca" certificate and key

# [certs] Generating "etcd/server" certificate and key

# [certs] etcd/server serving cert is signed for DNS names [localhost pimaster] and IPs [192.168.12.101 127.0.0.1 ::1]

# [certs] Generating "etcd/peer" certificate and key

# [certs] etcd/peer serving cert is signed for DNS names [localhost pimaster] and IPs [192.168.12.101 127.0.0.1 ::1]

# [certs] Generating "etcd/healthcheck-client" certificate and key

# [certs] Generating "apiserver-etcd-client" certificate and key

# [certs] Generating "sa" key and public key

# [kubeconfig] Using kubeconfig folder "/etc/kubernetes"

# [kubeconfig] Writing "admin.conf" kubeconfig file

# [kubeconfig] Writing "kubelet.conf" kubeconfig file

# [kubeconfig] Writing "controller-manager.conf" kubeconfig file

# [kubeconfig] Writing "scheduler.conf" kubeconfig file

# [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

# [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

# [kubelet-start] Starting the kubelet

# [control-plane] Using manifest folder "/etc/kubernetes/manifests"

# [control-plane] Creating static Pod manifest for "kube-apiserver"

# [control-plane] Creating static Pod manifest for "kube-controller-manager"

# [control-plane] Creating static Pod manifest for "kube-scheduler"

# [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

# [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

# [kubelet-check] Initial timeout of 40s passed.

# [kubelet-check] It seems like the kubelet isn't running or healthy.

# [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

# [kubelet-check] It seems like the kubelet isn't running or healthy.

# [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

# [kubelet-check] It seems like the kubelet isn't running or healthy.

# [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

# [apiclient] All control plane components are healthy after 212.227971 seconds

# [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

# [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster

# [upload-certs] Skipping phase. Please see --upload-certs

# [mark-control-plane] Marking the node pimaster as control-plane by adding the label "node-role.kubernetes.io/master=''"

# [mark-control-plane] Marking the node pimaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

# [bootstrap-token] Using token: rqyytr.f3pvvz2yepq51b7h

# [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles

# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes

# [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

# [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

# [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

# [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

# [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key

# [addons] Applied essential addon: CoreDNS

# [addons] Applied essential addon: kube-proxy

# Your Kubernetes control-plane has initialized successfully!

# To start using your cluster, you need to run the following as a regular user:

# mkdir -p $HOME/.kube

# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

# sudo chown $(id -u):$(id -g) $HOME/.kube/config

# You should now deploy a pod network to the cluster.

# Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

# https://kubernetes.io/docs/concepts/cluster-administration/addons/

# Then you can join any number of worker nodes by running the following on each as root:

# kubeadm join 192.168.12.101:6443 --token rqyytr.f3pvvz2yepq51b7h \

# --discovery-token-ca-cert-hash sha256:5b04368aaee148fa3eb8c6aed3c3bc8041248590e2055a81617d16d8f796bb77

# start using the cluster

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

# check node status (something's missing - flannel)

kubectl get nodes

# NAME STATUS ROLES AGE VERSION

# pimaster NotReady master 14m v1.19.4

# install flannel network plugin

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# podsecuritypolicy.policy/psp.flannel.unprivileged created

# clusterrole.rbac.authorization.k8s.io/flannel created

# clusterrolebinding.rbac.authorization.k8s.io/flannel created

# serviceaccount/flannel created

# configmap/kube-flannel-cfg created

# daemonset.apps/kube-flannel-ds created

# and now we're ready (check under STATUS)

kubectl get nodes

# NAME STATUS ROLES AGE VERSION

# pimaster Ready master 17m v1.19.4

# add node to the cluster (pinode, 192.168.12.102)

sudo kubeadm join 192.168.12.101:6443 --token rqyytr.f3pvvz2yepq51b7h --discovery-token-ca-cert-hash sha256:5b04368aaee148fa3eb8c6aed3c3bc8041248590e2055a81617d16d8f796bb77

# [preflight] Running pre-flight checks

# [preflight] Reading configuration from the cluster...

# [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

# [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

# [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

# [kubelet-start] Starting the kubelet

# [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

# This node has joined the cluster:

# * Certificate signing request was sent to apiserver and a response was received.

# * The Kubelet was informed of the new secure connection details.

# Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

# check nodes in the cluster (check the details... nice)

kubectl get nodes -o wide

# NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME

# pimaster Ready master 18m v1.19.4 192.168.12.101 Raspbian GNU/Linux 10 (buster) 5.4.79-v7+ docker://18.9.1

# pinode Ready 6m15s v1.19.4 192.168.12.102 Ubuntu 20.04.1 LTS 5.4.0-56-generic docker://19.3.8

# test with some workload

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-deployment.yaml

# deployment.apps/redis-master created

kubectl get pods

# NAME READY STATUS RESTARTS AGE

# redis-master-f46ff57fd-9m9g9 1/1 Running 0 37s

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-service.yaml

# service/redis-master created

kubectl get service

# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

# kubernetes ClusterIP 10.96.0.1 443/TCP 22m

# redis-master ClusterIP 10.100.245.207 6379/TCP 4s

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-slave-deployment.yaml

# deployment.apps/redis-slave created

kubectl get pods

# NAME READY STATUS RESTARTS AGE

# redis-master-f46ff57fd-9m9g9 1/1 Running 0 2m30s

# redis-slave-bbc7f655d-dz895 1/1 Running 0 12s

# redis-slave-bbc7f655d-slnhv 1/1 Running 0 12s

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-slave-service.yaml

# service/redis-slave created

kubectl get services

# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

# kubernetes ClusterIP 10.96.0.1 443/TCP 23m

# redis-master ClusterIP 10.100.245.207 6379/TCP 91s

# redis-slave ClusterIP 10.101.165.188 6379/TCP 13s

kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-deployment.yaml

# deployment.apps/frontend created

kubectl get pods -l app=guestbook -l tier=frontend

# NAME READY STATUS RESTARTS AGE

# frontend-6c6d6dfd4d-945hd 1/1 Running 0 7s

# frontend-6c6d6dfd4d-mm7vm 1/1 Running 0 7s

# frontend-6c6d6dfd4d-wllpx 1/1 Running 0 7s

kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-service.yaml

# service/frontend created

kubectl get services

# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

# frontend NodePort 10.101.140.217 80:31597/TCP 4s

# kubernetes ClusterIP 10.96.0.1 443/TCP 24m

# redis-master ClusterIP 10.100.245.207 6379/TCP 2m45s

# redis-slave ClusterIP 10.101.165.188 6379/TCP 87s

Seems to be working (😊):

# cleanup
kubectl delete deployment -l app=redis
# deployment.apps "redis-master" deleted
# deployment.apps "redis-slave" deleted

kubectl delete service -l app=redis
# service "redis-master" deleted
# service "redis-slave" deleted

kubectl delete deployment -l app=guestbook
# deployment.apps "frontend" deleted

kubectl delete service -l app=guestbook
# service "frontend" deleted

kubectl get pods
# No resources found in default namespace.

# cleanup

kubectl delete deployment -l app=redis

# deployment.apps "redis-master" deleted

# deployment.apps "redis-slave" deleted

kubectl delete service -l app=redis

# service "redis-master" deleted

# service "redis-slave" deleted

kubectl delete deployment -l app=guestbook

# deployment.apps "frontend" deleted

kubectl delete service -l app=guestbook

# service "frontend" deleted

kubectl get pods

# No resources found in default namespace.

Cheers!

P.S. I’ve read about some having issues with flannel and using other network options (didn’t have this one). Also, if you’ll have issues with iptables (v1.8+), maybe you’ll need to switch to legacy version (didn’t have this one either).