Bad Request for url (error 400) in AKS

By Tomica Kaniski in Other 2 Comments

I’ve decided to go through the **awesome** AKS Workshop on Microsoft Learn and had some issues (with my setup), which I wanted to share, in case someone else hits them.

It was all good until I got to the part of creating the AKS cluster with Azure CLI – I was using Windows Terminal with WSL (Ubuntu 20.04) instead of using Azure Cloud Shell as suggested. I’ve gone through the steps of preparing variables needed for creating the cluster as it says, and when I tried to finally create the cluster by using “az aks create” command, I’ve got an error:

Error states that something is wrong with our request and neither –verbose or –debug options were giving me any useful details (actually, it was in front of me all the time, but I didn’t see it ?). I’ve rechecked/reset the variables, tried once more and once more… it was all the same. As Google was conveniently down at the time (who would say, right?!), I’ve had to try and figure it out by myself. So, I’ve looked at the error once again:

Operation failed with status: ‘Bad Request’. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/aks-workshop/providers/Microsoft.Network/virtualNetworks/aks-vnet/subnets/aks-subnet%0D/providers/Microsoft.Authorization/roleAssignments?$filter=atScope%28%29&api-version=2018-09-01-preview

… and then it struck me!

There’s some trash in the URL (more precisely – my AKS subnet ID was having “%0D” added to the end)!

And if we check what “%0D” exactly stands for, it says “carriage return” (which I’ve obviously didn’t want to be a part of my subnet ID) – so, even it all seemed fine when looking at the variable content, now I know it wasn’t.

Easy-peasy, we can fix the part where we’re extracting this subnet ID or we can just replace the variable’s value with the right one (without the %0D at its end, that is).

That got me going… towards the next error. This one was actually more descriptive (yes, and the first one is descriptive enough, if you read it carefully ?) – it said that I’ve got additional content inside my Kubernetes version variable:

Operation failed with status: ‘Bad Request’. Details: Error to parse agent pool version “1.19.3\r”: Invalid character(s) found in patch number “3\r”

You can see the extra “\r“, which again, is here because of bad value assigned to the variable $VERSION.

Which can also be easily fixed.

One other funny thing I’ve observed was, when getting my Kubernetes cluster credentials, as you can see below, they were actually merged to C:\Users\tomica\.kube\config:

This was funny because I’m inside WSL… which doesn’t actually have C:\Users\tomica\.kube\config, right? (and no, credentials weren’t merged to /home/tomica/.kube/config, which kubectl there uses by default, so… they are actually at /mnt/c/Users/tomica/.kube/config – funny, will check with the MS folks) ?

Fair enough – we can merge them manually or just select the right file and we’re good to go:

There you go – if you get stuck on similar things, maybe this can help you. ?

Cheers!

Yet another “Kubernetes with Raspberry Pi” post

By Tomica Kaniski in Other 4 Comments

There’s a ton of the tutorials on how to get Kubernetes installed onto your Raspberry Pi, so… let’s write another one. ?

As mentioned in my last post, I’ve found my forgotten Raspberry Pi, and played around with installing and configuring Raspbian Buster on it.

Today, I wanted to check if it will be possible to install Kubernetes onto such small machine – they are many articles on the “widest of the world’s webs” that say “Yes, it can be done!“, so I’ve decided to give it a try! And I chose to follow one of them (seemed like a nice reference).

As you remember, I’m starting with a cleanly installed (and just slightly customized) Raspbian Buster and building it from there.

And I’ll be using kubeadm for installing my cluster.

So, once I had at least two machines (my Raspberry Pi for the “control plane” and Ubuntu 20.04 LTS Hyper-V virtual machine as the “node” – you can read more about it here), I prepared them like this:

  • install Docker (in my case)
  • change the default cgroups driver for Docker to systemd
  • add cgroups limit support (for my Raspberry Pi 3)
  • configure iptables
  • disable swap (this one was a bit challenging)
  • prepare for Kubernetes installation (source, keys, kubeadm)
  • install Kubernetes “control plane”
  • add flannel
  • add a node to the cluster
  • test with some workload

One thing that bothered me (on Buster) was disabling swap in a way that it also stays disabled after a reboot (I know, it’s the details that eventually get you) – after a while, I’ve stumbled on this forum post and the solution provided by “powerpete” did the trick! Thank you, @powerpete! ?

And finally, details about the each step are here (outputs are commented and somewhat redacted/condensed):

Seems to be working (?):

Cheers!

P.S. I’ve read about some having issues with flannel and using other network options (didn’t have this one). Also, if you’ll have issues with iptables (v1.8+), maybe you’ll need to switch to legacy version (didn’t have this one either).

Deploying Kubernetes on top of Azure Stack (Development Kit)

By Tomica Kaniski in Other 0 Comment

If you had a chance to deploy Azure Stack or Azure Stack Development Kit (ASDK) in your environment, maybe you’ve asked yourself “OK, but what should I do with it now?“.

Well, one of many things you “can do with it” is offer your users to deploy Kubernetes clusters on top of it (at least, that was what I did the other day… on my ASDK deployment) – in short, official documentation has you pretty much covered. I know, Azure enables it as well… and the process here is similar, or – the same.

The main thing you have to decide at the beginning, is if you’ll use Azure AD or ADFS for identity management (the same as with Azure Stack deployment, if you remember, from my previous posts). Why – because the installation steps differ a bit.

Once you decide it (or you ask your Azure Stack administrator how it’s done in your case), you can proceed with the installation – I assume you have your Azure Stack/ASDK up and running.

Next, in the admin portal (https://adminportal.local.azurestack.external/), you’ll need to add the prerequisites from Azure Marketplace (for this, if you remember, your Azure Stack/ASDK has to be registered):

Once done, you’re ready to set up the service principal, to which you’ll then assign the required permissions on both – the Azure side and on the Azure Stack side! (don’t forget this detail… it is well documented, but easy to overlook)

In case you don’t give your service principal the required permissions on both “sides”, you’ll probably get the “error 12” and your deployment will fail:

And you can see details in the log:

So… be careful with service principal and permissions! 🙂

Next thing you’ll need to make sure of is that you create a plan and an offer, but set your quotas right! It depends on your Kubernetes cluster deployment settings, but if you’ll go with the defaults, the default quotas (disk, in particular) need to be expanded!

If not, you’ll probably get this error:

If you were careful while reading the official docs (with a few “lessons learned” in this post), and you’ve made it to here… you’re probably set to deploy your first Kubernetes cluster on top of your Azure Stack/ASDK.

In the user portal (https://portal.local.azurestack.external/), you now have the option to deploy something called Kubernetes Cluster (preview):

Here you really can’t miss much – you’ll give your deployment a brand new (or empty) resource group, user details (together with your public SSH key, of course), DNS prefix, number and size of nodes and service principal details:

After that, your deployment starts and runs for some time (it, again, depends on your hardware and settings you’ve chosen for your cluster). Hopefully, it will end with this message:

If all is good, you can SSH into one of your master nodes and see the details of your cluster:

One other thing that would be nice to have is the Kubernetes dashboard – the process of enabling it is well documented here:

And – you’re done!

You now have your own Kubernetes cluster deployment on top of your Azure Stack/ASDK! How cool is that?! 🙂

One last thing to note – currently, this is in preview (as it says on the template), but… it works. 🙂

Cheers!