Playing around with Azure Stack HCI

Decided to have some fun with (nested) Microsoft Azure Stack HCI in my lab.

If you want to do the same, I’ve scripted most the stuff you need, so… maybe it will be useful.

Steps to prepare a brand new, shiny, nested Azure Stack HCI lab are (roughly):

  • prepare your (parent) Windows Server 2022 Hyper-V host (ensure enough resources are available)
    • it already hosts my Active Directory, DNS, DHCP, router, …,  VMs
    • everything will be saved locally to D:\AzureStackHCI
  • (optional) install Windows Admin Center (WAC) for easier management
    • download it and install with simple command:

  • obtain the Azure Stack HCI 60-day trial ISO image from here
  • make VHD(X) from the obtained ISO image:

    • note that I’m using Convert-WindowsImage.ps1 available here
    • which gives me nice, generalized Azure Stack HCI VHD(X), which we will “upgrade with things” and later use for VM creation
  • install prerequisites into VHD(X)
    • this one is fairly easy – install Windows roles and features directly to the VHD(X) itself:

    • NOTE: If you try to install the Hyper-V role later, it may fail as we’re running Azure Stack HCI on “normal” Windows Server, so it get’s confused with nested virtualization availability. With preinstaling it, we make sure it just works.
  • update VHD(X) with latest patches:

    • I have previously downloaded all the Azure Stack HCI patches available to D:\AzureStackHCI\Updates
  • add Unattend.xml to handle the “set password at first login issue”
    • it annoys me that I need to set up the initial password, so… simple Unattend.xml file, injected into VHD(X) should take care of this:

    • NOTE: Make sure you don’t use clear-text passwords in Unattend.xml file!
  • create Azure Stack HCI VMs
    • I’m creating two VMs from our prepared VHD(X), with a couple of additional data disks, few network adapters for different purposes, nested virtualization enabled, etc.:

  • set node networking, join them to the domain, prepare for cluster (by using PowerShell Direct):

  • create the Azure Stack HCI cluster:

  • register (optional) the Azure Stack HCI cluster:

  • create CSV(s) and virtual switches for child workloads (but add nodes/cluster to WAC before, if not using PowerShell)

  • play around with your new cluster
  • (optional) clean all/redeploy if needed:

And now you have fully functional, nested, 2-node Azure Stack HCI cluster – nothing too fancy, but you can extend it how you wish! 😊

You can begin exploring the Azure Stack HCI itself, use it with Azure Arc, or perhaps install AKS on Azure Stack HCI and play around with it. Or something else.

Cheers!

P.S. You can use these scripts also for stuff other than Azure Stack HCI, of course! 😉
P.P.S. Code is also available on my GitHub page.

Create a self-signed certificate for your web server with PowerShell

Sometimes you may need SSL certificate just for testing your (local) web application. Of course, for public and trusted purposes, you’ll probably use free Let’s Encrypt certificate or something similar (or, of course, any of the paid options).

And this is OK as long as you have publicly resolvable domain name.

But what if you need certificate for, let’s say, “localhost” or “webserver.local”?

Then you’ll probably use your internal PKI infrastructure or a simple self-signed certificate.

Second one can be easily achieved with PowerShell, by using the New-SelfSignedCertificate cmdlet (or with OpenSSL, yes 🙂).

So, let me show you how.

We have a simple IIS setup hosting a single (default) website, responding to http://localhost/:

We’ll issue a new self-signed certificate, make it trusted (important!) and then attach it to our test website, with following:

If everything goes well, we will see another binding created in our IIS console:

And if we open https://localhost/, all should be good as well:

Cheers!

Add a route to your VPN connection via PowerShell

I’m sure that you’re using some VPN somewhere, and you’re having “trouble” with split tunneling and routing, right?

Well, I had. 😀

As I’m “here and there” most of the time, I’ve setup an “anchor” location (no, it’s not in the cloud… yet) which is always available via VPN, and which has few machines that I’m, more or less, using regularly. When I’m not there, I connect there via my precious Windows 10/11 laptop and work as I’m there locally. I know – you know what VPNs are used for… bear with me a bit longer. 😀

So, all good – I have a VPN client (Windows built-in), a VPN server and Internet connection, and I can work.

One thing that I like to have is Internet access which is not routed via my “anchor” location, so that “the work stuff” goes through VPN and “the fun stuff” not.

It’s really easy to set this up – in properties of your VPN connection, just untick the “Use default gateway on remote network” checkbox:

But then you’ll have an issue with connecting to “the work stuff” – your current default gateway doesn’t know where “the work stuff” network is and how to get there.

It needs a route.

No problem, it’s easy to add a route in Windows (my “the work stuff” network is 192.168.13.0/24 and my VPN gateway is 192.168.14.1, or publicly 141.138.55.154):

And now you have access to “the work stuff” network again! And Internet access works as it should (not via the “anchor” location)!

Great.

But then you disconnect. And reconnect. And route you’ve added is gone. So, you repeat the procedure. Or script it. Or…

What if I tell you there is actually a better way?

I’m not really sure in which release this came out, but now you have an updated set of PowerShell cmdlets in (Windows 10/11) (which is cool!). For this story, the one we’re interested the most is Add-VpnConnectionRoute.

“So, doest that mean that, with it, I can configure my VPN connection to always have the route I need, whenever I connect to VPN? No more adding routes manually?!”

Exactly.

If I use the discussed Add-VpnConnectionRoute on my existing VPN connection, I can add the route I need and it will be written in the connection configuration and made active when the tunnel comes up, while still using the split tunneling.

Let’s see:

  • connected to “the work stuff” VPN and this is (part of) routing table prior the route configuration:

  • adding route configuration:

  • checking routes again:

As you can see, I’ve got new routes in my route table (it would be the same by using route add command above) and now I can access “the work stuff” without any issue:

And if I disconnect and connect again – it still works! 😊

Hope it helps someone!

Cheers!

Beware of the proxy!

Had a (somewhat) interesting case the other day – after (finally) upgrading my Windows Admin Center (WAC) gateway machine to the new Windows Server 2022, my WAC suddenly stopped working. I couldn’t connect to any of the servers from within the console, couldn’t add new ones, … nothing.

When tried adding new servers, nothing happened – wizard stays at “Searching for…“:

Even PowerShell couldn’t connect anymore (which is actually the root cause of the above).

So, what happened?

Everything worked before and I wasn’t aware of other changes… other than upgrading my OS (in-place upgrade, Windows Server 2019 to Windows Server 2022), that is.

Let’s try and make sense of all this.

Test-NetConnection says everything is fine, Test-WSMan from another machine works:

However, Test-WSMan from this (WAC) machine simply doesn’t work:

Tried checking the logs next – two errors inside Applications and Services Logs -> Microsoft -> Windows Remote Management -> Operational log caught my eye:

  • Error 138: The client got a timeout from the network layer (ERROR_WINHTTP_TIMEOUT)
  • Error 142: WSMan operation Identify failed, error code 2150859046

So, it’s something with the network after all – more specifically, seems like there is some issue on the HTTP/S part!

After some thinking, I remembered that we have a HTTP/S proxy in our network – maybe my PowerShell session actually tries to go through it?! 😀

Checking if proxy is set (with netsh winhttp show proxy) – it is! This could be the issue.

Now I’m resetting the proxy settings (with netsh winhttp reset proxy, of course):

And then trying Test-WSMan again:

It finally works! And WAC works as well! 😀

Hope this helps!

Cheers!

Capturing network trace in Windows

Do you need to capture some network traffic on a Windows box for further analysis, but don’t want to install additional software just… everywhere?

I usually do.

If you didn’t know, Windows has built-in tool with which you can do just that – (among other things) capture network trace to a file for further analysis. The tool is called netsh.

So, how do you capture traffic with netsh?

It’s fairly easy (for more options, filters and such, you can always check the accompanying help content – netsh trace start ?):

If you look at the location where you’ve saved your trace, you’ll see two files – of those two files, MyTrace.etl is the one you want:

OK, but what do you do with it?

If you try to open it with, for example, WireShark, you’ll see it doesn’t work:

So… we have a trace file with which we can’t really do anything?!?

Not exactly!

If you have Microsoft Network Monitor (now archived, but can be found… on the Internet) or Microsoft Message Analyzer (now retired), you can open up and analyze your trace as you normally would:

If you already have WireShark on, let’s say, your workstation, and want to continue using it for the analysis, this trace needs to be converted to a format which WireShark understands (hope that one day we’ll have WireShark which opens such .etl files natively).

You can convert it by using the free tool called etl2pcapng.

It doesn’t require installation, and if you want to use the pre-compiled binaries, they are available under etl2pcapng releases.

So, convert your (netsh) MyTrace.etl to (WireShark’s) MyTrace.pcapng with this command:

Once converted, you can open the new file (MyTrace.pcapng) in WireShark, and do what you would usually do to analyze it:

Hope this helps!

Cheers!

Utilman.exe to cmd.exe and back

Let’s say you have a Windows (virtual) machine, for which you’ve forgotten your login info, but you want to enter it anyway, because of… reasons. 😀

How can you do it?

Note – if the disk/VM is encrypted, you’ll need the decryption key, of course (if you don’t have it, well… I’m sorry, the following won’t really help you).

Ok, if it’s a virtual machine and you only need to grab some data from it, it’s relatively easy – you’ll just mount the virtual disk, extract the data needed and done.

If you need access to the OS instead, you can maybe use the old trick with replacing the Utilman.exe with cmd.exe, which essentially gives you command prompt with local system permissions, which then gives you… well, everything you need.

One minor obstacle with doing this “hack” would be the fact that the owner of Utilman.exe is actually the TrustedInstaller, so your workflow would be like this:

  • (e.g. turn off the VM, mount the disk, …)
  • replace the owner of Utilman.exe
  • add yourself the needed permissions
  • replace the Utilman.exe with cmd.exe
  • do what you need (e.g. change the local Administrator’s password, set this account as active, …)
  • cleanup (replace the replaced Utilman.exe with the original one)

And we can do this with PowerShell:

And now you can login as local Administrator again and do the work you wanted to do in the first place. 😊

To leave things in (somewhat) the way we found them, we can use the following PowerShell:

Cheers!

Fixing Hyper-V virtual machine import with Compare-VM

Well, I was rearranging some stuff the other day, and come to an interesting “lesson learned”, which I’ll share. 🙂

In my lab, I’ve had a Hyper-V server running Windows 2012 R2, which I finally wanted to upgrade to something newer. I’ve decided to go with the latest Windows Server Insider Preview (SA 20180), just for fun.

When trying to do an in-place upgrade, I was presented with the message “it can’t be done“, which is fine – my existing installation is with GUI, the new one will be Core.

So, evacuate everything and reinstall.

In the process, I’ve also reorganized some stuff (machines were moved to another disk, not all files were on the same place, etc.).

Installed Windows, installed Hyper-V, created VM switches, but when I tried to import it all back (from PowerShell… because I had no GUI anymore), I was presented with an error.

Error during virtual machine import was (I know – could’ve used more specific Import-VM command, which will select all the right folders and required options, but… learned something new by doing it this way!):

So, the error says it all – “Please use Compare-VM to repair the virtual machine.” 🙂

But how?! 🙂

If you go to the docs page of Compare-VM, you can see how it’s used.

And, in my case, the whole process of repairing this virtual machine looks like this:

Hope this helps you as well!

Cheers!

Backing up Office 365 to S3 storage (Exoscale SOS) with Veeam

Are you backing up your Office 365? And… why not? 🙂

I’m not going into the lengthy and exhausting discussion of why you should take care of your data, even if it’s stored in something unbreakable like “the cloud”, at least not in this post. I would like to focus on one of the features of the new Veeam Backup for Office 365 v4, which was released just the other day. This feature is “object storage support“, as you may have guessed it already from the title of this fine post!

So, this means that you can take Amazon S3, Microsoft Azure Blob Storage or even IBM Cloud Object Storage and use it for your Veeam Backup for Office 365. And even better – you can use any S3-compatible storage to do the same! How cool is that?!

To test this, I decided to use the Exoscale SOS (also S3-compatible) storage for backups of my personal Office 365 via Veeam Backup for Office 365.

I’ve created a small environment to support this test (and later production, if it works as it should) and basically done the following:

  • created a standard Windows Server 2019 VM on top of Microsoft Azure, to hold my Veeam Backup for Office 365 installation
    (good people at Microsoft provided me Azure credits, so… why not?!)
  • downloaded Veeam Backup for Office 365
    (good people at Veeam provided me NFR license for it, so I’ve used it instead of Community Edition)
  • created an Exoscale SOS bucket for my backups
    (good people at Exoscale/A1TAG/A1.digital/A1HR provided me credits, so… why not?!)
  • installed Veeam Backup for Office 365
    (it’s a “Next-Next-Finish” type of installation, hard to get it wrong)
  • configured Veeam Backup for Office 365 (not so hard, if you know what you are doing and you’ve read the official docs)
    • added a new Object Storage Repository
    • added a new Backup Repository which offloads the backup data to the previously created Object Storage Repository
    • configured a custom AAD app (with the right permissions)
    • added a new Office 365 organization with AAD app and Global Admin account credentials (docs)
    • created a backup job for this Office 365 organization
    • started backing it all up

Now, a few tips on the “configuration part”:

  • Microsoft Azure:
    • no real prerequisites and tips here – simple Windows VM, on which I’m installing the downloaded software (there is a list of system requirements if want to make sure it’s all “by the book”)
  • Exoscale:
    • creating the Exoscale SOS bucket is relatively easy, once you have your account (you can request a trial here) – you choose the bucket name and zone in which data will be stored and… voilà:

    • if you need to make adjustments to the ACL of the bucket, you can (quick ACL with private setting is just fine for this one):

    • to access your bucket from Veeam, you’ll need your API keys, which you can find in the Account – Profile – API keys section:

    • one other thing you’ll need from this section is the Storage API Endpoint, which depends on the zone you’ve created your bucket in (mine was created inside AT-VIE-1 zone, so my endpoint is https://sos-at-vie-1.exo.io):

  • Office 365:
    • note: I’m using the Modern authentication option because of MFA on my tenant and… it’s the right way to do it!
    • for this, I created a custom application in Azure Active Directory (AAD) (under App registrations – New registration) (take a note of the Application (client) ID, as you will need it when configuring Veeam):

    • I’ve added a secret (which you should also take a note of, because you’ll need it later) to this app:

    • then, I’ve added the minimal required API permissions to this app (as per the official docs) – but note that the official docs have an error (at this time), which I reported to Veeam – you’ll need the SharePoint Online API access permissions even if you don’t use the certificate based authentication(!) – so, the permissions which work for me are:

    • UPDATE: Got back the word from Veeam development – additional SharePoint permissions may not be necessary after all, maybe I needed to wait a bit longer… will retry next time without those permissions. 🙂
    • after that, I’ve enabled the “legacy authentication protocols”, which is still a requirement (you can do it in Office 365 admin center – SharePoint admin center – Access Control – Apps that don’t use modern authentication – Allow access or via PowerShell command “Set-SPOTenant -LegacyAuthProtocolsEnabled $True”):

    • lastly, I’ve created an app password for my (global admin) account (which will also be required for Veeam configuration):

  • Veeam Backup for Office 365:
    • add a new Object Storage Repository:

    • add a new Backup Repository (connected to the created Object Storage Repository; this local repository will only store metadata – backup data will be offloaded to the object storage and can be encrypted, if needed):

    • add a new Office 365 organization:

    • create a backup job:

    • start backing up your Office 365 data:

Any questions/difficulties with your setup?
Leave them in the comments section, I’ll be happy to help (if I can).

Cheers!

Figuring out your public IP address with PowerShell

Sometimes, you need to know your public IP address because of… reasons. My particular reason was creating firewall rule to limit SSH only from my current public IP address, to a machine on the Internet. And how to do it?

You can always use free services like What Is My IP?, which shows you your public IP address in a nice form:

But there are also other ways – if you’re running Linux (or WSL) and do a Google search for the command that can help you, you’ll probably get this (https://askubuntu.com/questions/95910/command-for-determining-my-public-ip?noredirect=1&lq=1):

And if you’re using Windows, PowerShell is here to help you! I like “oneliners”, even if they are not always easy to read:

I’m sure that my friend Aleksandar (PowerShell guru & Microsoft MVP) has a better way, but for me, this works just fine. 🙂

Hope it helps!

Cheers!

Counters missing when machines accessed remotely

Not so long ago, we observed an issue with remotely accessing the PhysicalDisk counters on several machines, more specifically – there were none. 🙂

To be clear – if you opened up the Performance Monitor (perfmon.exe) on the affected machine, you can see all the counters, including the PhysicalDisk counters. But, if you opened up the Performance Monitor on a different machine and tried to access PhysicalDisk counters of the first machine over network, they aren’t shown anymore… but others (like CPU and Memory) are still there and can be used!

Counters shown normally on local computer and in local Performance Monitor

The same counters not visible from remote machine’s Performance Monitor

So… why? 🙂

At first, we thought that our monitoring software went berserk, but no – the PhysicalDisk counters on a remote machine were missing even we were using the built-in Performance Monitor tool (PhysicalDisk counters weren’t shown).

Next – maybe it’s something on the network? Of course, network is never the issue, but still… (wasn’t an issue here as well, because other counters worked without any issues)

Next, we thought, it’s related to the version of Windows accessing from, or the version at the destination – as we found out, too many different versions were impacted to hold that theory, so… no.

One thing we are not sure is if it’s caused by some of the “not so recent security patches”.

As we found the solution for our issue, what exactly caused it in the first place is not so important right now… Solution is simple – you actually need to run one command to re-register the system performance libraries with WMI (winmgmt /resyncperf) and then reboot the affected machine.

So, the commands you need are:

After that, we can access all the needed counters (PhysicalDisk) remotely again:

Counters shown normally from remote computer and in local Performance Monitor

Cheers!

P.S. Don’t forget to reboot the affected machine! 🙂