Using a self-hosted runner with GitHub Actions

As I was going through the excellent short course called Azure Infrastructure as Code with GitHub (by fellow MVP, Barbara Forbes), a thought appeared – what do I need to do to use my custom runner machine inside a pipeline for… I don’t know… security/privacy concerns, isolation, special requirements, different OS, control, price… or just to complicate things a bit?

Of course, GitHub supports this and it’s called a self-hosted runner.

So, what do I need to do to use this self-hosted runner with my GitHub Actions?

It’s relatively simple – there is an application package, which will be installed on your runner machine, and which will listen for and eventually do all the work defined in your workflow!

But first, let’s introduce my environment.

I have a simple GitHub Action (workflow), which creates a simple storage account on my Azure environment (there is actually no need to convert Bicep to ARM before deployment, but it seemed cool 😀). It’s currently using the „ubuntu-latest“ runner, provided by GitHub… which has also all the needed components inside (like Azure CLI, Azure PowerShell, …).

And it works fine. When there is a push to my GitHub repository, GitHub Actions starts and does what is needed on my Azure environment via this workflow:

And the mighty Bicep file (😀) it’s using for the deployment is:

Of course, this runs just fine on a standard (hosted) runner:

To run this workflow (successfully) not that much is needed.

First, I’ve created a new virtual machine (I’ll use a simple Ubuntu Hyper-V VM, no autoscaling, no… nothing) called hermes (god of speed 😀), with freshly installed Ubuntu 22.04.1-LTS (minimized).

After that, I went to the Settings of my GitHub repository and got the download and install scripts for the x64 Linux runner:

As you can see, I’ll be using crontab later to automatically (re)start my self-hosted runner.

If everything went well, you should see your runner “up and running” (😀) in the GitHub portal:

Next, I’ll use the following script to install all prerequisites for my workflow (like Azure CLI, Azure PowerShell, etc. – it really depends on your workflow and things you use):

Once this is done, my self-hosted runner hermes should be ready to run the workflow.

To try this, I need to make a slight update to my workflow file – line 12 inside the job configuration should be updated from “runs-on: ubuntu-latest” to “runs-on: self-hosted“.

So, my workflow YAML file now looks like this:

And once I push the configuration to my GitHub, my workflow automatically starts and runs on hermes, my self-hosted runner:

If we prepared our runner right, all is good! 😊

Of course, our resources are deployed successfully:

So, this is how you can use your own, self-hosted runner, to execute your GitHub Actions (workflows).

Cheers!

What about this Bicep?

You’ve probably heard about Azure Resource Manager (ARM) – the deployment and management service/layer of Azure, which enables you to manage (create, configure, delete) your Azure resources. Also, you are probably aware that ARM uses so called ARM templates – basically, JSON files that actually define the infrastructure and configuration you want to deploy to Azure (think Infrastructure as Code, IaC).

So, if you have dealt with ARM/JSON in the past, you may have been finding it difficult to start with, and somewhat complex.

Bicep is here to help.

Here is a short overview of Bicep – basically, it’s a language which enables you easier deployment of Azure resources, without messing around (too much) with JSON. To be frank, it somehow reminds of Terraform, but it’s also different. It has many cool features, immediately supports all new Azure features and APIs, can be built (converted) into .json and deployed as such or it can be deployed straight away as .bicep, doesn’t require state file, it’s open and free, has great support in Visual Studio Code and much more. And it’s still in active development!

If you’re dealing with IaC and Azure, try it.

To show you the power (and simplicity) of Bicep, here is a short example of deploying Linux virtual machine in Azure (together with a resource group, virtual network, virtual network subnet, virtual NIC and network security group), done “the old way” (in JSON, which was actually converted from Bicep… it’s easier than writing JSON from the scratch) and then done via Bicep (“the right way”? 😀).

Additionally, you’ll see that I’ve tried to break stuff into modules – with more or less sucess. 😀

The ARM/JSON way (could be done nicer/shorter, with parameters inside .parameters.json… if you know what you’re doing – this is converted from Bicep and serves just for illustrative purposes):

 

The Bicep way:



Bicep seems a bit easier to read and shorter, right (while still doing basically the same thing)? 😀

If we deploy the .bicep files above (note that I’m deploying the “raw” .bicep file directly – which is cool!):


We finally get our resources:

So, where should you start if you’re new to Bicep?

I would certainly recommend starting with free and official Deploy and manage resources in Azure by using Bicep learning path on Microsoft Learn.

After that, you can probably pick up Freek Berson’s book Getting started with Bicep: Infrastructure as Code on Azure (first and only book on Bicep that I know of – really liked it because of the simple (yet effective) examples with storage accounts, it connects everything and flows naturally – building up “brick by brick” and not “jumping around”, just to show off what Bicep can do).

Another great resource are also the Bicep examples – there’s plenty to learn from them too!

Of course, you’ll also need to practice – install the Azure CLI or Azure PowerShell module, add Bicep and use Visual Studio Code for your first steps with creating, deleting, configuring and breaking stuff… powered by Bicep! 😀

Cheers!

Having fun with Helm and file encoding

Had some spare time, so I’ve tried to learn a bit more about Helm, the package manager for Kubernetes.

I’ve decided to follow the relatively new Pluralsight course called – Kubernetes Package Administration with Helm, done by my MVP colleague Andrew Pruski. And it was great – not too long, clear and easy to follow, with only a handful of prerequisites if you want to follow along! Great job!

Of course, there is also the nice, official documentation.

But why am I writing this post?

I was normally following this course on my Windows 10 laptop, using Visual Studio Code, as suggested, and also using PowerShell terminal, with Helm v3.3.1.

It all went well until the part when we are creating our Helm Chart, more specifically – when we’re filling up our deployment.yaml and service.yaml files. Suggested (and simplest) method is to use the simple output redirection (with “>“), like this:

But, this gave me the following error when trying to deploy the chart:

It’s quite obvious – Helm works with UTF-8, and my .yaml files seem to be encoded differently. Quick look at the bottom of my VSCode confirms it:

How can I fix it?

As I’m using PowerShell, it’s pretty easy – instead of doing the simple output redirection (“>“), I pipe output to Out-File cmdlet with -Encoding UTF8 option, in all cases, which takes care of the encoding (and sets it to UTF-8 with BOM, which is just fine for Helm):

So, long story short – if you run into the error above (Error: unable to build kubernetes objects from release manifest: error parsing : error converting YAML to JSON: yaml: invalid leading UTF-8 octet”), remember to check your file’s encoding (and change it to UTF-8, if needed)! 🙂

Cheers!

P.S. Thanks to good people at Pluralsight for providing me a complimentary subscription!

Creating a function in Azure (Packt)

Functions in a serverless architecture consist of logic that serves a single, well-defined purpose. They are executed using an ephemeral compute service and can be scaled automatically based on demand. Azure Functions is Microsoft’s solution for serverless functions.

— post by Joseph Ingeno, provided by Packt —

In my book, Software Architect’s Handbook, I discuss serverless architecture and the use of functions. In this blog post, we will create a new function in Azure. Functions in Azure provide you with a choice of programming language (C#, JavaScript, Java, F#, with others coming in the future) and allow you to bring in dependencies from the NuGet and npm package managers. The runtime that powers Azure Functions can be found on GitHub.

There are different ways that a function for Azure can be created, such as using Visual Studio, Visual Studio Code, or the Azure command line interface (Azure CLI). However, so as to make this demo not require the installation of any tools, we will create a function through the Azure portal.

The following steps will be necessary to create a function in the Azure portal:

  • Logging in to the Azure portal
  • Creating a function app
  • Creating a function in the new function app
  • Testing the function
  • Customizing the function
  • Cleaning up resources

 

Logging in to the Azure portal

If you do not already have one, the first step is to create a free Azure account by navigating to https://azure.microsoft.com. Each account receives some free credit to allow you to try out Azure services.

Once you have an account, you can log in to the Azure portal by going to http://portal.azure.com.

 

Creating a function app

A function app in Azure is a container that hosts the execution of individual functions. Before we create a function, we must create a function app that will host it.

  1. In the top left corner, select the Create a resource option, followed by Compute.

  1. Select Function App.

  1. Create a function app by providing values for the various settings:
    • App Name: A unique name for the function app
    • Subscription: The subscription under which the function app will be created
    • Resource Group: A resource group is a container that holds resources related to the Azure solution. Create a new resource group or use an existing one for the function
    • Hosting Plan: The hosting plan for the function app; The consumption plan is charged on a pay-per-execution basis and dynamically allows resources based on the app’s load. The App Service Plan lets you define a capacity allocation with predictable costs and scale.
    • Location: The region where your function app will execute; Select one near you or near other services that your function will need to access.
    • Storage: An Azure storage account is used to store and access your Azure Storage data objects. Create a new storage account or select an existing one to be used by the function

  1. Click the Create button to deploy the new function app. Once the application has been deployed, you will receive a notification.

 

Creating a function

Once your new function app has been created, we can create a function to be used in the container. We’ll be creating an HTTP triggered function that can be executed via an HTTP call.

  1. If Function Apps is already under your Favorites, simply select it.

  1. If it isn’t already in your favorites, select All services. Find Function Apps and click the star next to it to make it one of your favorites. After doing so, select Function Apps.

  1. You should see the function app that you created previously. Click the Create New icon next to Functions under your function app in order to create a new function.

  1. To create a new function, we can either select a premade one or create one on our own. In this case, we will use one of the premade ones.

Select Webhook + API and select one of the languages. In this example, I have elected to use C#. Click the Create this function button when you are finished, which will create the function.

If C# was selected as the language, the code for the premade function is as follows:

 

Testing the function

Now that the function has been created, we can test it to confirm that it works properly.

  1. Your function has now been created and should be visible in the Functions section underneath your function app.

  1. With your function selected, click the Get function URL link to get the URL for your function.

  1. The previous step will cause the Get function URL dialog box to be displayed. Select default (Function key) for the Key drop-down, and then click Copy to copy the URL.

The URL contains a key that is required, by default, to access the function over HTTP.

  1. Paste the URL into a browser. As we saw when looking at the code, there is a query string parameter for a name Add &name=<yourname> to the end of the URL, replacing <yourname> with your actual name. Press the Enter key to execute the request.

If the name that you passed was “Joe,” depending on the browser that you used you would either see the string “Hello Joe” or you would see it in XML format.

Trace information is written to logs during function execution. With the focus being placed on your function in the Azure portal, the contents of the logs can be viewed at the bottom of the page.

 

Customizing the function

You may want to customize the function that you just created in terms of things like the HTTP methods that are supported by the function and the routing template. Under your specific function, select the Integrate option.

There are various options for the HTTP trigger that can be customized.

Allowed HTTP methods

You can configure your function to support all HTTP methods (in which case the Selected HTTP methods section with its checkboxes is hidden) or selected HTTP methods. In the Selected HTTP methods section, you can choose which HTTP methods you want the function to support (GET, POST, DELETE, HEAD, PATCH, PUT, OPTIONS, and TRACE).

Authorization level

You can choose between the following authorization levels:

  • Function
  • Admin
  • Anonymous

With Function selected as the authorization level, an API key is required to access the function. This key was being passed as part of the URL:

If we set the authorization level to Anonymous, our function will be accessible without an API key. This makes the URL look as follows:

Route template

You can change the route that is used in order to invoke the function.  For example, if you were to enter “/HelloTrigger” in the Route template textbox, instead of using the function name in the URL as is the case by default, the enter route template value will be used instead. The resulting URL is as follows:

The “api” part of the URL is part of the base path prefix and is handled by a global setting.

Cleaning up resources

Resource groups in Azure are containers that hold resources such as function apps, functions, and storage accounts. Deleting a resource group deletes everything that it contains. Once you are done with the demo, you may want to delete the resource group that was used so you are not billed for any resources.

  1. In the Azure portal, go to the relevant resource group page. One of the ways that this can be done is by selecting Resource groups and then selecting a resource group.

  1. An alternative approach is to select our function app as we had done previously and select its resource group from there.

  1. With the resource group in focus, select the Delete resource group

  1. Follow the instructions and click the Delete

It can take a few minutes, but eventually, the resource group and its contents will be deleted. A notification will appear to let you know when it is done.

Joseph Ingeno is a software architect who has designed and developed many different software applications. During his career, he has worked on projects for a number of different business domains, using a variety of technologies, programming languages, and frameworks. His book, Software Architect’s Handbook, was published by Packt and is now available. The Software Architect’s Handbook is a comprehensive guide to help developers, architects, and senior programmers advance their career in the software architecture domain.

Cheers!

Creating, Debugging and Deploying an Azure Function (Packt)

You can learn how to create, debug, and deploy an Azure Function by reading this tutorial by Daniel Bass, a developer who develops complex backend systems entirely on Azure, making heavy use of event-driven Azure Functions and Azure Data Lake.

Serverless programming has been a buzzword in technology for a while now, first implemented for arbitrary code by Amazon on Amazon Web Services (AWS) in 2014. The term normally refers to snippets of backend code running in environments that are wholly managed by the cloud provider, totally invisible to developers. This approach has some astounding benefits, enabling an entirely new paradigm of computing architecture.

This article will focus on Microsoft’s serverless product, Azure Functions. In this article, you’ll create an Azure Function in Visual Studio, debug it locally, and deploy it to an Azure cloud instance. You can refer to https://github.com/TrainingByPackt/Serverless-Architectures-with-Azure/tree/master/Lesson%201 to access the complete code for this article.

To develop Azure Functions for production, you need a computer running Windows and Visual Studio 2015 or later; however, the smoothest experience is present in Visual Studio 2017, version 15.4 or later. If your computer can run Visual Studio, it can handle the Azure Function development.

Creating Your First Function to Receive and Process Data from an HTTP Request

Before you begin, confirm that you have Visual Studio 2017 version 15.4 installed; if not, download and install it. Visual Studio 2017 has a comprehensive suite of Azure tools, including Azure Function development. To do so, perform the following steps:

1. Open the Visual Studio Installer, which will show you the version of Visual Studio that you have installed and allow you to select the Azure Workflow and install it; if it is missing, then update Visual Studio, if required, to the latest version:

2. Click on Modify, select the Azure development workload, and click on Modify again:

Now, you can create a new Azure Function as a part of your serverless architecture that listens to HTTP requests to a certain address as its trigger. Begin by implementing the following steps:

1. Create a new solution. The example is called BeginningAzureServerlessArchitecture, which is a logical wrapper for several functions that will get deployed to this namespace.

2. Use the Visual C# Cloud | Azure Function Select the Empty trigger type and leave the default options, but set storage to None. This will create a Function App, which is a logical wrapper for several functions that will get deployed and scaled together:

3. You now have a solution with two files in it: host.json and local.settings.json. The local.settings.json file is used solely for local development, where it stores all details on connections to other Azure services.

It is important to note that when uploading something to a public repository, be very careful not to commit unencrypted connection settings – by default they will be unencrypted. host.json is the only file required to configure any functions running as a part of your Function App. This file can have settings that control the function timeout, security settings for every function, and a lot more.

4. Now, right-click on the project and select Add New Item. Once again, choose the Azure Function template:

5. On the next screen, select Http trigger with parameters, and set the Access rights to Anonymous. Right-click on your solution and select Enable NuGet Package Restore:

6. You will now have a C# file called PostTransactions.cs. It consists of a single method, Run, with an awful lot in the method signature: an attribute and an annotation. Some of this will be familiar to you if you are an experienced C# developer, and it is important to understand this signature.

Configuration as code is an important modern development practice. Rather than having servers reconfigured or configured manually by developers before code is deployed to them, configuration as code dictates that the entire configuration required to deploy an application to production is included in the source code.

This allows for variable replacement by your build/release agent, as you will (understandably) want slightly different settings, depending on your environment. Azure Functions implement this principle, with a configuration split between the host.json file for app-wide configurations and app settings, and the Run method signature for individual functions. Therefore, you can deploy an Azure Function to production with only the code that you find in the GitHub repository:

Outcome

You created an Azure Function, understood the roles the different files play, and learned about configuration as code.

The FunctionName annotation defines the name of the function within the Function App. This can be used for triggering your function, or it can be kept separate. The first parameter is an HttpRequestMessage object with an HttpTrigger attribute. This is what varies when you choose different triggers (for example, a timer trigger will have an object with a TimerTrigger attribute).

This attribute has several arguments. The first is the authorization level. Do you remember setting this when you created the function? It was called Access rights in the template. This defines the level of authorization that the function will demand of HTTP requests. The five levels are shown in the following table:

Authorization level Required information
Anonymous No key required; anyone with the path can call it an unlimited number of times.
User Need a valid token, generated by a user that has AD permission to trigger the Function App. Useful for high-security environments, where each service needs to manage its own security. Generally, token-based authentication is much more desirable than key-based.
Function Need the function key—a unique key created for each function in a Function App upon deployment. Any host key will also work. The most common form of authorization for basic deployments.
System Need the master key—a key at the Function App level (called a host key) that cannot be deleted but can be renewed.
Admin Need any host key.

One thing to bear in mind is that if you set a function to be high security and use System or Admin authorization, then any client that you give that key to will also be able to access any other functions in the Function App (if they can work out the path). Make sure that you separate high-security functions into different apps.

The next parameters are GET and POST, which define the HTTP verbs that will activate the function. Generally, from a microservices architecture point of view, you should only have one verb to prevent you from having to do bug-prone switching logic inside of the function. You can simply create four separate functions if you want GET,  POST,  PUT, and  DELETE on an artifact.

Finally, there is a string assigned to the property route. This is the only bit of routing logic that the function itself can see, and it simply defies the subpath from the Function App. It accepts WebAPI syntax, which you can see in the curly braces, / {name}. This will assign any text that appears where the curly braces are to a parameter called name.

This completes the HttpTrigger object. The three parameters left in the method signature are an HttpRequestMessage object, which allows you to access the HttpRequestMessage that triggered the function; a string parameter called name, which is what the string in the curly braces in the path will get bound to; and a TraceWriter for logging.

The current logic of the Function App can be seen in the following example, and you should see that it will take whatever name is put into it and send back an HTTP response saying Hello to that name.

Debugging an Azure Function

You now have a working Azure Function that can be deployed to Azure or run locally. You’ll first host and debug the function locally, to show the development cycle in action.

Debug an Azure Function

In this section, you’ll run an Azure Function locally and debug it. You can develop new functions and test the functionality before deploying to the public cloud. And to ensure that it happens correctly, you’ll require the single function created directly from the HTTP trigger with the parameters template.

Currently, your machine does not have the correct runtime to run an Azure Function, so you need to download it:

1. Click on the Play button in Visual Studio, and a dialog box should ask you if you want to download Azure Functions Core Tools – click on Yes. A Windows CMD window will open, with the lightning bolt logo of Azure Functions. It will bootstrap the environment and attach the debugger from Visual Studio. It will then list the endpoints the Function App is listening on.

2. Open the Postman app and copy and paste the endpoint into it, selecting either a POST or GET verb. You should get the response Hello {name}. Try changing the {name} in the path to your name, and you will see a different response. You can download Postman at https://www.getpostman.com/.

3. Create a debug point in the Run method by clicking in the margin to the left of the code:

4. Use Postman to send the request.

5. You are now able to use the standard Visual Studio debugging features and inspect the different objects as shown in the following screenshot:

6. Set your verb to POST, and add a message in the payload. See if you can find the verb in the HttpRequestMessage object in debug mode. It should be in the method property.

Outcome

You have debugged an Azure Function and tested it using Postman. As you can see from running the function locally, you, the developer, do not need to write any of the usual boilerplate code for message handling or routing. You don’t even need to use ASP.NET controllers, or set up middleware. The Azure Functions container handles absolutely everything, leaving your code to simply do the business logic.

Activity: Improving Your Function

In this activity, you’ll add a JSON payload to the request and write code to parse that message into a C# object.

Prerequisites

You’ll require a function created from the HTTP trigger with the parameters template.

Scenario

You are creating a personal finance application that allows users to add their own transactions, integrate with other applications, and perhaps allow their credit card to directly log transactions. It will be able to scale elastically to any number of users, saving money when you don’t have any users.

Aim

Parse a JSON payload into a C# object, starting your RESTful API.

Steps for Completion

1. Change the Route to transactions.

2. Remove the get Remove the String parameter called name:

3. Add the Newtonsoft.json package, if it isn’t already present. You can do this by right-clicking on Solution Manage NuGet packages | Browse | Newtonsoft.json.

4. Right-click on the project and add a folder called Models, and then add a C# class called Transaction. Add two properties to this class: a DateTime property called ExecutionTime, and a Decimal property called Amount:

5. Use DeserializeObject<Transaction>(message).Result() to de-serialize the HttpRequestMessage into an instantiation of this class. To do this, you need to import the Models namespace and Newtonsoft.json. This will parse the JSON payload and use the Amount property to file the corresponding property on the Transaction object:

6. Change the return message to use a property of the new Transaction object, for example, You entered a transaction of £47.32!. Go to Postman and open the Body tab and select raw.

7. Enter the following JSON object:

8. Run locally to test. Make sure that you change the endpoint to /transactions in Postman.

Outcome

You learned how to access the HttpRequestMessage, and you will have a function that can read a JSON message and turn it into a C# object. During this subtopic, you debugged an Azure Function. Visual Studio only allows this through downloading azure-functions-core-tools. Unfortunately, it doesn’t make it available on the general command line—only through command windows started in Visual Studio. If you want to use it independently, then you have to download it using npm. If you need to download azure-functions-core-tools separately, you can use npm to get it – npm install -g azure-functions-core-tools for version 1 (fully supported) and npm install -g azure-functions-core-tools@core for version 2 (beta). You can then use the debug setup to set Visual Studio to call an external program with the command func host start when you click on the Debug button.

This package is a lot more than just a debug environment, however; it actually has a CLI for everything you could possibly need in the Azure Function development. Open up a command window (in Visual Studio, if you haven’t downloaded it independently) and type func help; you should see a full list of everything the CLI can do. Notable commands are func host start, which starts the local debug environment, and func azure {functionappname} fetch-app-settings, which lets you download the app settings of a function deployed to Azure so that you can test integration locally, as well. These need to be run in the same folder as the host.json file.

Deploying an Azure Function

An Azure Function is obviously geared towards being hosted on the Azure cloud, rather than locally or on your own computer. Visual Studio comes with a complete suite of tools to deploy and manage Azure services, and this includes full support for Azure Functions. The azure-functions-core-tools CLI that you downloaded to provide a local debug environment also has a set of tools for interacting with Azure Functions in the cloud, if you prefer CLIs.

It is possible to run Azure Functions on your own servers, using the Azure Functions runtime. This is a good way to utilize the sunk cost that you have already spent on servers, combined with the unlimited scale that Azure offers (if demand exceeds your server capacity). It’s probably only worth it in terms of cost if you have a significant amount of unused Windows server time because this solution inevitably requires more management than normal Azure Function deployments. To deploy to Azure, you’ll need an Azure login with a valid Azure subscription.

Deploying to Azure

In this section, you’ll deploy your first function to the public cloud and learn how to call it. You’ll go live with your Azure Function start creating your serverless architecture. And to ensure that it happens correctly, you’ll need a function project and a valid Azure subscription. You can begin by implementing the following steps:

1. Right-click on your project and select Publish…. Now select Azure Function App Create New as shown in the following screenshot:

2. Enter a memorable name and create a resource group and a consumption app service plan to match the following:

3. Click on the Publish button to publish your function.

4. Open a browser, navigate to http://portal.azure.com, and find your function. You can use the search bar and search the name of your function. Click on your Function App, and then click on the function name. Click on Get function URL in the upper-right corner, paste the address of your function in Postman, and test it. If you have a paid subscription, these executions will cost a small amount of money – you are only charged for the compute resources that you actually use. On a free account, you get a million executions for free.

Outcome

You now have a fully deployed and working Azure Function in the cloud.

This is not the recommended way to deploy to production. Azure Resource Manager (ARM) templates are the recommended way to deploy to production. ARM templates are JavaScript Object Notation (JSON) files. The resources that you want to deploy are declaratively described within JSON. An ARM template is idempotent, which means it can be run as many times as required, and the output will be the same each and every time. Azure handles the execution and targets the changes that need to be run.

If you found this article interesting, explore Daniel Bass’ Beginning Serverless Architectures with Microsoft Azure to quickly get up and running with your own serverless development on Microsoft Azure. This book will provide you with the context you need to get started on a larger project of your own, leaving you equipped with everything you need to migrate to a cloud-first serverless solution.

Cheers!

P.S. This post was provided by good people at Packt Publishing. Thank you, Ron!