Beware of the proxy!

By Tomica Kaniski in Other 0 Comment

Had a (somewhat) interesting case the other day – after (finally) upgrading my Windows Admin Center (WAC) gateway machine to the new Windows Server 2022, my WAC suddenly stopped working. I couldn’t connect to any of the servers from within the console, couldn’t add new ones, … nothing.

When tried adding new servers, nothing happened – wizard stays at “Searching for…“:

Even PowerShell couldn’t connect anymore (which is actually the root cause of the above).

So, what happened?

Everything worked before and I wasn’t aware of other changes… other than upgrading my OS (in-place upgrade, Windows Server 2019 to Windows Server 2022), that is.

Let’s try and make sense of all this.

Test-NetConnection says everything is fine, Test-WSMan from another machine works:

However, Test-WSMan from this (WAC) machine simply doesn’t work:

Tried checking the logs next – two errors inside Applications and Services Logs -> Microsoft -> Windows Remote Management -> Operational log caught my eye:

  • Error 138: The client got a timeout from the network layer (ERROR_WINHTTP_TIMEOUT)
  • Error 142: WSMan operation Identify failed, error code 2150859046

So, it’s something with the network after all – more specifically, seems like there is some issue on the HTTP/S part!

After some thinking, I remembered that we have a HTTP/S proxy in our network – maybe my PowerShell session actually tries to go through it?! 😀

Checking if proxy is set (with netsh winhttp show proxy) – it is! This could be the issue.

Now I’m resetting the proxy settings (with netsh winhttp reset proxy, of course):

And then trying Test-WSMan again:

It finally works! And WAC works as well! 😀

Hope this helps!

Cheers!

What about this Bicep?

By Tomica Kaniski in Other 0 Comment

You’ve probably heard about Azure Resource Manager (ARM) – the deployment and management service/layer of Azure, which enables you to manage (create, configure, delete) your Azure resources. Also, you are probably aware that ARM uses so called ARM templates – basically, JSON files that actually define the infrastructure and configuration you want to deploy to Azure (think Infrastructure as Code, IaC).

So, if you have dealt with ARM/JSON in the past, you may have been finding it difficult to start with, and somewhat complex.

Bicep is here to help.

Here is a short overview of Bicep – basically, it’s a language which enables you easier deployment of Azure resources, without messing around (too much) with JSON. To be frank, it somehow reminds of Terraform, but it’s also different. It has many cool features, immediately supports all new Azure features and APIs, can be built (converted) into .json and deployed as such or it can be deployed straight away as .bicep, doesn’t require state file, it’s open and free, has great support in Visual Studio Code and much more. And it’s still in active development!

If you’re dealing with IaC and Azure, try it.

To show you the power (and simplicity) of Bicep, here is a short example of deploying Linux virtual machine in Azure (together with a resource group, virtual network, virtual network subnet, virtual NIC and network security group), done “the old way” (in JSON, which was actually converted from Bicep… it’s easier than writing JSON from the scratch) and then done via Bicep (“the right way”? 😀).

Additionally, you’ll see that I’ve tried to break stuff into modules – with more or less sucess. 😀

The ARM/JSON way (could be done nicer/shorter, with parameters inside .parameters.json… if you know what you’re doing – this is converted from Bicep and serves just for illustrative purposes):

 

The Bicep way:



Bicep seems a bit easier to read and shorter, right (while still doing basically the same thing)? 😀

If we deploy the .bicep files above (note that I’m deploying the “raw” .bicep file directly – which is cool!):


We finally get our resources:

So, where should you start if you’re new to Bicep?

I would certainly recommend starting with free and official Deploy and manage resources in Azure by using Bicep learning path on Microsoft Learn.

After that, you can probably pick up Freek Berson’s book Getting started with Bicep: Infrastructure as Code on Azure (first and only book on Bicep that I know of – really liked it because of the simple (yet effective) examples with storage accounts, it connects everything and flows naturally – building up “brick by brick” and not “jumping around”, just to show off what Bicep can do).

Another great resource are also the Bicep examples – there’s plenty to learn from them too!

Of course, you’ll also need to practice – install the Azure CLI or Azure PowerShell module, add Bicep and use Visual Studio Code for your first steps with creating, deleting, configuring and breaking stuff… powered by Bicep! 😀

Cheers!

Capturing network trace in Windows

By Tomica Kaniski in Other 0 Comment

Do you need to capture some network traffic on a Windows box for further analysis, but don’t want to install additional software just… everywhere?

I usually do.

If you didn’t know, Windows has built-in tool with which you can do just that – (among other things) capture network trace to a file for further analysis. The tool is called netsh.

So, how do you capture traffic with netsh?

It’s fairly easy (for more options, filters and such, you can always check the accompanying help content – netsh trace start ?):

If you look at the location where you’ve saved your trace, you’ll see two files – of those two files, MyTrace.etl is the one you want:

OK, but what do you do with it?

If you try to open it with, for example, WireShark, you’ll see it doesn’t work:

So… we have a trace file with which we can’t really do anything?!?

Not exactly!

If you have Microsoft Network Monitor (now archived, but can be found… on the Internet) or Microsoft Message Analyzer (now retired), you can open up and analyze your trace as you normally would:

If you already have WireShark on, let’s say, your workstation, and want to continue using it for the analysis, this trace needs to be converted to a format which WireShark understands (hope that one day we’ll have WireShark which opens such .etl files natively).

You can convert it by using the free tool called etl2pcapng.

It doesn’t require installation, and if you want to use the pre-compiled binaries, they are available under etl2pcapng releases.

So, convert your (netsh) MyTrace.etl to (WireShark’s) MyTrace.pcapng with this command:

Once converted, you can open the new file (MyTrace.pcapng) in WireShark, and do what you would usually do to analyze it:

Hope this helps!

Cheers!

Patch Tuesday with PowerShell

By Tomica Kaniski in Other 0 Comment

It’s actually Wednesday here, but yesterday was another “Patch Tuesday” and Microsoft released its update packages.

So – it’s time to patch! 😊

Not sure how you’re patching your machines, but just wanted to make you aware of the nice PowerShell module called PSWindowsUpdate.

You can read more about it on the official PowerShell Gallery page and also here.

In short – this module takes care of controlling Windows Update from within PowerShell on your local and also remote machines.

And… I know it’s not nice like the Windows Update screen, but it does its job! 😊

To make use of it, you’ll have to take care of some minor prerequisites and install it via PowerShell Gallery.

Once done, you can use it to control Windows Update:

To make it easier for you, here is the installation script which takes care of… everything:

And that’s it – we’re done!

Happy patching!

Cheers!

Bad Request for url (error 400) in AKS

By Tomica Kaniski in Other 2 Comments

I’ve decided to go through the **awesome** AKS Workshop on Microsoft Learn and had some issues (with my setup), which I wanted to share, in case someone else hits them.

It was all good until I got to the part of creating the AKS cluster with Azure CLI – I was using Windows Terminal with WSL (Ubuntu 20.04) instead of using Azure Cloud Shell as suggested. I’ve gone through the steps of preparing variables needed for creating the cluster as it says, and when I tried to finally create the cluster by using “az aks create” command, I’ve got an error:

Error states that something is wrong with our request and neither –verbose or –debug options were giving me any useful details (actually, it was in front of me all the time, but I didn’t see it 😊). I’ve rechecked/reset the variables, tried once more and once more… it was all the same. As Google was conveniently down at the time (who would say, right?!), I’ve had to try and figure it out by myself. So, I’ve looked at the error once again:

Operation failed with status: ‘Bad Request’. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/aks-workshop/providers/Microsoft.Network/virtualNetworks/aks-vnet/subnets/aks-subnet%0D/providers/Microsoft.Authorization/roleAssignments?$filter=atScope%28%29&api-version=2018-09-01-preview

… and then it struck me!

There’s some trash in the URL (more precisely – my AKS subnet ID was having “%0D” added to the end)!

And if we check what “%0D” exactly stands for, it says “carriage return” (which I’ve obviously didn’t want to be a part of my subnet ID) – so, even it all seemed fine when looking at the variable content, now I know it wasn’t.

Easy-peasy, we can fix the part where we’re extracting this subnet ID or we can just replace the variable’s value with the right one (without the %0D at its end, that is).

That got me going… towards the next error. This one was actually more descriptive (yes, and the first one is descriptive enough, if you read it carefully 😊) – it said that I’ve got additional content inside my Kubernetes version variable:

Operation failed with status: ‘Bad Request’. Details: Error to parse agent pool version “1.19.3\r”: Invalid character(s) found in patch number “3\r”

You can see the extra “\r“, which again, is here because of bad value assigned to the variable $VERSION.

Which can also be easily fixed.

One other funny thing I’ve observed was, when getting my Kubernetes cluster credentials, as you can see below, they were actually merged to C:\Users\tomica\.kube\config:

This was funny because I’m inside WSL… which doesn’t actually have C:\Users\tomica\.kube\config, right? (and no, credentials weren’t merged to /home/tomica/.kube/config, which kubectl there uses by default, so… they are actually at /mnt/c/Users/tomica/.kube/config – funny, will check with the MS folks) 😊

Fair enough – we can merge them manually or just select the right file and we’re good to go:

There you go – if you get stuck on similar things, maybe this can help you. 😊

Cheers!

Yet another “Kubernetes with Raspberry Pi” post

By Tomica Kaniski in Other 4 Comments

There’s a ton of the tutorials on how to get Kubernetes installed onto your Raspberry Pi, so… let’s write another one. 😊

As mentioned in my last post, I’ve found my forgotten Raspberry Pi, and played around with installing and configuring Raspbian Buster on it.

Today, I wanted to check if it will be possible to install Kubernetes onto such small machine – they are many articles on the “widest of the world’s webs” that say “Yes, it can be done!“, so I’ve decided to give it a try! And I chose to follow one of them (seemed like a nice reference).

As you remember, I’m starting with a cleanly installed (and just slightly customized) Raspbian Buster and building it from there.

And I’ll be using kubeadm for installing my cluster.

So, once I had at least two machines (my Raspberry Pi for the “control plane” and Ubuntu 20.04 LTS Hyper-V virtual machine as the “node” – you can read more about it here), I prepared them like this:

  • install Docker (in my case)
  • change the default cgroups driver for Docker to systemd
  • add cgroups limit support (for my Raspberry Pi 3)
  • configure iptables
  • disable swap (this one was a bit challenging)
  • prepare for Kubernetes installation (source, keys, kubeadm)
  • install Kubernetes “control plane”
  • add flannel
  • add a node to the cluster
  • test with some workload

One thing that bothered me (on Buster) was disabling swap in a way that it also stays disabled after a reboot (I know, it’s the details that eventually get you) – after a while, I’ve stumbled on this forum post and the solution provided by “powerpete” did the trick! Thank you, @powerpete! 😊

And finally, details about the each step are here (outputs are commented and somewhat redacted/condensed):

Seems to be working (😊):

Cheers!

P.S. I’ve read about some having issues with flannel and using other network options (didn’t have this one). Also, if you’ll have issues with iptables (v1.8+), maybe you’ll need to switch to legacy version (didn’t have this one either).

Found my forgotten Raspberry Pi

By Tomica Kaniski in Other 2 Comments

And, naturally, decided to put it to use (although, for exactly what… is currently unclear). 😊

So… how?

As there was already a micro SD card inside my Raspberry Pi, I was all set!

Basically, what I had to do:

  • download the OS image (Raspberry Pi OS Lite)
  • download imaging software (Etcher)
  • extract the OS onto micro SD card
  • enable SSH by adding an empty file called “ssh” (yes, without any extension) to the boot volume
  • boot it up
  • set it up as I like

Extracting the OS image onto micro SD card is a “breeze” with right tools – select OS image, select where do you want to put it and click Flash:

After it’s finished, don’t forget to enable yourself the SSH access (it’s easier that way):

Done.

Let’s put the card back into Raspberry Pi and boot it up.

Few seconds later, you can use (e.g.) Windows Terminal and included SSH client to access your Raspberry Pi (default networking option is DHCP, with default username of pi and password raspberry):

I wanted to “tweak” my installation a bit (with the provided raspi-config script), so I’ve used the following for disabling unnecessary devices, custom network settings, etc.:

And after a while, my Raspberry Pi is finally ready:

Cheers!

Utilman.exe to cmd.exe and back

By Tomica Kaniski in Other 0 Comment

Let’s say you have a Windows (virtual) machine, for which you’ve forgotten your login info, but you want to enter it anyway, because of… reasons. 😀

How can you do it?

Note – if the disk/VM is encrypted, you’ll need the decryption key, of course (if you don’t have it, well… I’m sorry, the following won’t really help you).

Ok, if it’s a virtual machine and you only need to grab some data from it, it’s relatively easy – you’ll just mount the virtual disk, extract the data needed and done.

If you need access to the OS instead, you can maybe use the old trick with replacing the Utilman.exe with cmd.exe, which essentially gives you command prompt with local system permissions, which then gives you… well, everything you need.

One minor obstacle with doing this “hack” would be the fact that the owner of Utilman.exe is actually the TrustedInstaller, so your workflow would be like this:

  • (e.g. turn off the VM, mount the disk, …)
  • replace the owner of Utilman.exe
  • add yourself the needed permissions
  • replace the Utilman.exe with cmd.exe
  • do what you need (e.g. change the local Administrator’s password, set this account as active, …)
  • cleanup (replace the replaced Utilman.exe with the original one)

And we can do this with PowerShell:

And now you can login as local Administrator again and do the work you wanted to do in the first place. 😊

To leave things in (somewhat) the way we found them, we can use the following PowerShell:

Cheers!

Network share feeds in WAC

By Tomica Kaniski in Other 0 Comment

You know about (and actively using) the Windows Admin Center (WAC), right?! 😉

While it’s great for managing your Microsoft infrastructure, it can also be extended with different extensions. You can even write and use your internal, custom extensions, which do… well, whatever you make them do. And you can read all about that here.

But let’s go back to the subject of today’s post – extensions can be installed via different feeds, either official or unofficial, provided by Microsoft or 3rd-party. You can easily add new feeds or remove existent by providing the feed location, which can be either a NuGet feed URL or a file share location, as stated in the official docs.

Using a file share location is easy:

  • you choose/create a folder:

  • share it (\\<my_server_name>\WACExtensions in my case):

  • and add it to your feeds – I’ll use the “PowerShell way”:

But no!

My feed seems to be added successfully, but it’s not showing in the list!

You can try the same through the web interface – it’s almost the same (OK, you’ll get the errors):

And permissions are fine, don’t worry. 😉

Why’s that?!

The catch here is that we added an empty folder/share – when adding this share, WAC intelligently looked into the folder, found nothing and (successfully) didn’t add our share to the feed list, as it’s empty. And yes, it also forgot to mention it when using PowerShell.

So, what can be done?

The workaround/solution is rather simple – just make sure you don’t add an empty feed/folder.

Just for fun – I’ve downloaded the HPE Extension for WAC, moved it into the WACExtensions shared folder and tried to add the feed again:

And – it worked! 😊

Cheers!

Having fun with Helm and file encoding

By Tomica Kaniski in Other 6 Comments

Had some spare time, so I’ve tried to learn a bit more about Helm, the package manager for Kubernetes.

I’ve decided to follow the relatively new Pluralsight course called – Kubernetes Package Administration with Helm, done by my MVP colleague Andrew Pruski. And it was great – not too long, clear and easy to follow, with only a handful of prerequisites if you want to follow along! Great job!

Of course, there is also the nice, official documentation.

But why am I writing this post?

I was normally following this course on my Windows 10 laptop, using Visual Studio Code, as suggested, and also using PowerShell terminal, with Helm v3.3.1.

It all went well until the part when we are creating our Helm Chart, more specifically – when we’re filling up our deployment.yaml and service.yaml files. Suggested (and simplest) method is to use the simple output redirection (with “>“), like this:

But, this gave me the following error when trying to deploy the chart:

It’s quite obvious – Helm works with UTF-8, and my .yaml files seem to be encoded differently. Quick look at the bottom of my VSCode confirms it:

How can I fix it?

As I’m using PowerShell, it’s pretty easy – instead of doing the simple output redirection (“>“), I pipe output to Out-File cmdlet with -Encoding UTF8 option, in all cases, which takes care of the encoding (and sets it to UTF-8 with BOM, which is just fine for Helm):

So, long story short – if you run into the error above (“Error: unable to build kubernetes objects from release manifest: error parsing : error converting YAML to JSON: yaml: invalid leading UTF-8 octet”), remember to check your file’s encoding (and change it to UTF-8, if needed)! 🙂

Cheers!

P.S. Thanks to good people at Pluralsight for providing me a complimentary subscription!