blog.kaniski.eu I just wanna learn!

8Sep/200

Having fun with Helm and file encoding

Had some spare time, so I've tried to learn a bit more about Helm, the package manager for Kubernetes.

I've decided to follow the relatively new Pluralsight course called - Kubernetes Package Administration with Helm, done by my MVP colleague Andrew Pruski. And it was great - not too long, clear and easy to follow, with only a handful of prerequisites if you want to follow along! Great job!

Of course, there is also the nice, official documentation.

But why am I writing this post?

I was normally following this course on my Windows 10 laptop, using Visual Studio Code, as suggested, and also using PowerShell terminal, with Helm v3.3.1.

It all went well until the part when we are creating our Helm Chart, more specifically - when we're filling up our deployment.yaml and service.yaml files. Suggested (and simplest) method is to use the simple output redirection (with ">"), like this:

But, this gave me the following error when trying to deploy the chart:

It's quite obvious - Helm works with UTF-8, and my .yaml files seem to be encoded differently. Quick look at the bottom of my VSCode confirms it:

How can I fix it?

As I'm using PowerShell, it's pretty easy - instead of doing the simple output redirection (">"), I pipe output to Out-File cmdlet with -Encoding UTF8 option, in all cases, which takes care of the encoding (and sets it to UTF-8 with BOM, which is just fine for Helm):

So, long story short - if you run into the error above ("Error: unable to build kubernetes objects from release manifest: error parsing : error converting YAML to JSON: yaml: invalid leading UTF-8 octet"), remember to check your file's encoding (and change it to UTF-8, if needed)! 🙂

Cheers!

P.S. Thanks to good people at Pluralsight for providing me a complimentary subscription!

30Jul/200

Fixing Hyper-V virtual machine import with Compare-VM

Well, I was rearranging some stuff the other day, and come to an interesting "lesson learned", which I'll share. 🙂

In my lab, I've had a Hyper-V server running Windows 2012 R2, which I finally wanted to upgrade to something newer. I've decided to go with the latest Windows Server Insider Preview (SA 20180), just for fun.

When trying to do an in-place upgrade, I was presented with the message "it can't be done", which is fine - my existing installation is with GUI, the new one will be Core.

So, evacuate everything and reinstall.

In the process, I've also reorganized some stuff (machines were moved to another disk, not all files were on the same place, etc.).

Installed Windows, installed Hyper-V, created VM switches, but when I tried to import it all back (from PowerShell... because I had no GUI anymore), I was presented with an error.

Error during virtual machine import was (I know - could've used more specific Import-VM command, which will select all the right folders and required options, but... learned something new by doing it this way!):

So, the error says it all - "Please use Compare-VM to repair the virtual machine." 🙂

But how?! 🙂

If you go to the docs page of Compare-VM, you can see how it's used.

And, in my case, the whole process of repairing this virtual machine looks like this:

Hope this helps you as well!

Cheers!

29Jun/200

Fixing things with… terraform destroy

I like Terraform, because it's so clean, fast and elegant; OK, I also suck at it, but hey - I'm trying! 🙂

The long story

Usually, Terraform and its providers are very good at doing things in the order they should be done. But sometimes people do come up with silly ideas, and mine was such (of course) - I've decided to rename something and it broke things. Just a little.

I have a simple lab in Azure, with a couple of virtual machines behind the Azure Load Balancer, no big deal. All this is being deployed (and redeployed) via Terraform, using the official azurerm provider. It's actually a Standard SKU Azure Load Balancer (don't ask why), with a single backend pool and a few probes and rules. Nothing special.

I've deployed this lab a few days ago (thanks to good people at Microsoft, I have some free credits to spare), everything worked just fine, but today I've got the wild idea - I've decided to rename my backend pool.

With all the automation in place, this shouldn't be a problem... one would think. 🙂

So, as I've updated my code and during the make it so phase (terraform apply), I've got some errors (truncated, with only the useful stuff):

Issue

After going through these errors, I've realized that my resources are indeed in the same region, but existing rules are referencing the current backend pool by name and actually blocking Terraform in renaming the backend pool.

There are a couple of options at this stage - you can destroy your deployment and run it again (as I normally would) and it should all be fine. Or you can try to fix only the dependent resources and make it work as part of the existing deployment.

With some spare time at my hands, I've tried to fix it using the second one and it actually worked.

Resolution

Terraform has a nice option for destroying just some parts of the deployment.

If you look at help for the terraform destroy command, you can see the target option:

And if you run it to fix your issues, you'll get a nice red warning saying that this is only for exceptional situations (and they mean it!):

 

So... BE CAREFUL! (can't stress this enough!)

 

Anyhow, I've destroyed rules which were preventing my rename operation:

And then another terraform apply --auto-approve recreated everything that was needed, and finally - my backend pool got renamed:

Another idea I've had was to taint the resources (terraform taint -help), which would probably be a lot nicer. Oh, well... maybe next time. 🙂

As things are constantly improving, it shouldn't be long until this is fixed (should it even be fixed?!). Until then, hope this will help you with similar issues!

Cheers!

1May/200

CRC – OpenShift 4 cluster in your pocket

... but only if you have large pockets, that is! 🙂

I suppose that, by now, you've already heard of minishift - tool which helps you run a single-node OpenShift 3.x cluster inside a virtual machine (on your preferred hypervisor), which is extremely convenient if you need to do some local development, testing, etc. Of course, you can always deploy what you need on top of Azure or any other cloud provider, but having OpenShift cluster locally can have its benefits.

And what if you need, let's say, an OpenShift 4.x cluster "to go"?

Don't worry, there is a solution for you as well! It's called CodeReady Containers (CRC) (version 1.9 at the time of writing), and is basically a single node OpenShift 4.x cluster, inside a virtual machine (sounds a lot like minishift, doesn't it?).

So, how can you get your hands on CRC and make it work?

There are a couple of steps involved:

  • download the OpenShift tools (as I'm using Windows, so I'll use this one):

  • unzip all to a location you like (mine is C:\PocketOpenShift):

  • (optional) add this location to PATH for easier usage (as already mentined, I'm using Windows & PowerShell):

  • run "crc setup" which will prepare the environment:

  • run "crc start" (with one or more options, if needed - as you can see, I'll be using custom number of vCPUs, amount of RAM and a custom nameserver):

  • at any time, you can check the status of your cluster by using "crc status" command:

  • once it is up, you can use "oc login" or console (bring it up with "crc console") to connect to it, either as an (kube)admin or a developer, and continue working as you would normally with any other OpenShift cluster:

  • one other thing I like to do is to enable monitoring and other disabled stuff (note though - your VM should have 12+ GB RAM) - you can do it with two commands - first one lists all that is disabled, and the second one, with the index at the end, enables it (also note that in the official documentation there is an issue with "" and '' (they are switched), if you're working in PowerShell):

  • monitoring should now be working as well:

And that's it - you're ready to work on your own "pocket OpenShift cluster"! 🙂

Of course, don't forget that there is also the official documentation, including the Getting Started guide and Release Notes and Known Issues document. So... take a look!

Cheers!

27Nov/190

Backing up Office 365 to S3 storage (Exoscale SOS) with Veeam

Are you backing up your Office 365? And... why not? 🙂

I'm not going into the lengthy and exhausting discussion of why you should take care of your data, even if it's stored in something unbreakable like "the cloud", at least not in this post. I would like to focus on one of the features of the new Veeam Backup for Office 365 v4, which was released just the other day. This feature is "object storage support", as you may have guessed it already from the title of this fine post!

So, this means that you can take Amazon S3, Microsoft Azure Blob Storage or even IBM Cloud Object Storage and use it for your Veeam Backup for Office 365. And even better - you can use any S3-compatible storage to do the same! How cool is that?!

To test this, I decided to use the Exoscale SOS (also S3-compatible) storage for backups of my personal Office 365 via Veeam Backup for Office 365.

I've created a small environment to support this test (and later production, if it works as it should) and basically done the following:

  • created a standard Windows Server 2019 VM on top of Microsoft Azure, to hold my Veeam Backup for Office 365 installation
    (good people at Microsoft provided me Azure credits, so... why not?!)
  • downloaded Veeam Backup for Office 365
    (good people at Veeam provided me NFR license for it, so I've used it instead of Community Edition)
  • created an Exoscale SOS bucket for my backups
    (good people at Exoscale/A1TAG/A1.digital/A1HR provided me credits, so... why not?!)
  • installed Veeam Backup for Office 365
    (it's a "Next-Next-Finish" type of installation, hard to get it wrong)
  • configured Veeam Backup for Office 365 (not so hard, if you know what you are doing and you've read the official docs)
    • added a new Object Storage Repository
    • added a new Backup Repository which offloads the backup data to the previously created Object Storage Repository
    • configured a custom AAD app (with the right permissions)
    • added a new Office 365 organization with AAD app and Global Admin account credentials (docs)
    • created a backup job for this Office 365 organization
    • started backing it all up

Now, a few tips on the "configuration part":

  • Microsoft Azure:
    • no real prerequisites and tips here - simple Windows VM, on which I'm installing the downloaded software (there is a list of system requirements if want to make sure it's all "by the book")
  • Exoscale:
    • creating the Exoscale SOS bucket is relatively easy, once you have your account (you can request a trial here) - you choose the bucket name and zone in which data will be stored and... voilà:

    • if you need to make adjustments to the ACL of the bucket, you can (quick ACL with private setting is just fine for this one):

    • to access your bucket from Veeam, you'll need your API keys, which you can find in the Account - Profile - API keys section:

    • one other thing you'll need from this section is the Storage API Endpoint, which depends on the zone you've created your bucket in (mine was created inside AT-VIE-1 zone, so my endpoint is https://sos-at-vie-1.exo.io):

  • Office 365:
    • note: I'm using the Modern authentication option because of MFA on my tenant and... it's the right way to do it!
    • for this, I created a custom application in Azure Active Directory (AAD) (under App registrations - New registration) (take a note of the Application (client) ID, as you will need it when configuring Veeam):

    • I've added a secret (which you should also take a note of, because you'll need it later) to this app:

    • then, I've added the minimal required API permissions to this app (as per the official docs) - but note that the official docs have an error (at this time), which I reported to Veeam - you'll need the SharePoint Online API access permissions even if you don't use the certificate based authentication(!) - so, the permissions which work for me are:

    • UPDATE: Got back the word from Veeam development - additional SharePoint permissions may not be necessary after all, maybe I needed to wait a bit longer... will retry next time without those permissions. 🙂
    • after that, I've enabled the "legacy authentication protocols", which is still a requirement (you can do it in Office 365 admin center - SharePoint admin center - Access Control - Apps that don't use modern authentication - Allow access or via PowerShell command "Set-SPOTenant -LegacyAuthProtocolsEnabled $True"):

    • lastly, I've created an app password for my (global admin) account (which will also be required for Veeam configuration):

  • Veeam Backup for Office 365:
    • add a new Object Storage Repository:

    • add a new Backup Repository (connected to the created Object Storage Repository; this local repository will only store metadata - backup data will be offloaded to the object storage and can be encrypted, if needed):

    • add a new Office 365 organization:

    • create a backup job:

    • start backing up your Office 365 data:

Any questions/difficulties with your setup?
Leave them in the comments section, I'll be happy to help (if I can).

Cheers!

15Mar/190

Creating some virtual machines in Azure with PowerShell

The other day I was creating some Linux virtual machines (I know, I know...) and, with Azure being my preferred hosting platform, I've decided to create this machines by using a simple PowerShell script. Not because I'm so good at PowerShell, but because I like it... and sometimes I really don't like clicking through the wizard to create multiple machines.

I wanted to create multiple machines with ease, each with "static" IP address from the provided subnet, accessible via the Internet (SSH, HTTP) and running the latest Ubuntu Linux, of course.

So, I was browsing through the official documentation (a.k.a. docs.com, more specifically https://docs.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-powershell), and I've come up with this (my version of the official docs):

If this helps you with similar task - you're welcome.

Cheers!

17Dec/180

Renewing the expired Office Online/Web Apps Server farm certificate

Certificates sometimes expire... it happens! 🙂

But what happens if the certificate for your Office Online Server (OOS) or Office Web Apps Server (OWAS) farm expires and your farm is not available anymore?

Obviously, OOS farm and your Skype for Business, Exchange & SharePoint integration stops working. Next thing to do will be to renew the expired certificate.

But how?

My MVP colleague Andi Krüger did a nice blog post on updating the farm certificate, and it's fairly simple - Set-OfficeWebAppsFarm -CertificateName "RenewedOOSInternalCertificate" should do the trick... if your farm is running.

If things got out of hand and your farm is not running anymore and you cannot use the Set-OfficeWebAppsFarm cmdlet (you'll see that Office Online (WACSM) service is Stopped and cannot be brought back up with the expired certificate and your machine is showing that it's no longer part of the farm), you'll need to take a different approach, because you'll be getting errors when running the above mentioned command (like "It does not appear this machine is part of an Office Online Server farm." or similar).

WACSM Service is Stopped and and your machine is showing that it's no longer part of the farm

One of the possible solutions would be:

  • make a note of the Friendly Name of your old (expired) certificate (MMC or PowerShell) (in my case it's called "OOSInternalCertificate")
  • remove the expired certificate
  • renew/request/install the new certificate
  • change the Friendly Name of a new certificate to match the previous one
  • start the Office Online (WACSM) service or restart the machine
  • (copy the certificate/do the procedure on other farm members, if needed)

Everything is back normal

Your farm operations should now be restored and you can run Get-OfficeWebAppsFarm cmdlet normally:

Or you can open up the farm's discovery URL - if it's rendering again, everything should be OK (in my case "https://oos.myfarm.local/hosting/discovery"):

Even the discovery works

Cheers!

12Dec/180

Counters missing when machines accessed remotely

Not so long ago, we observed an issue with remotely accessing the PhysicalDisk counters on several machines, more specifically - there were none. 🙂

To be clear - if you opened up the Performance Monitor (perfmon.exe) on the affected machine, you can see all the counters, including the PhysicalDisk counters. But, if you opened up the Performance Monitor on a different machine and tried to access PhysicalDisk counters of the first machine over network, they aren't shown anymore... but others (like CPU and Memory) are still there and can be used!

Counters shown normally on local computer and in local Performance Monitor

The same counters not visible from remote machine's Performance Monitor

So... why? 🙂

At first, we thought that our monitoring software went berserk, but no - the PhysicalDisk counters on a remote machine were missing even we were using the built-in Performance Monitor tool (PhysicalDisk counters weren't shown).

Next - maybe it's something on the network? Of course, network is never the issue, but still... (wasn't an issue here as well, because other counters worked without any issues)

Next, we thought, it's related to the version of Windows accessing from, or the version at the destination - as we found out, too many different versions were impacted to hold that theory, so... no.

One thing we are not sure is if it's caused by some of the "not so recent security patches".

As we found the solution for our issue, what exactly caused it in the first place is not so important right now... Solution is simple - you actually need to run one command to re-register the system performance libraries with WMI (winmgmt /resyncperf) and then reboot the affected machine.

So, the commands you need are:

After that, we can access all the needed counters (PhysicalDisk) remotely again:

Counters shown normally from remote computer and in local Performance Monitor

Cheers!

P.S. Don't forget to reboot the affected machine! 🙂

5Dec/180

Software Management in Linux (Packt)

Learn software management with advanced Linux administration in this tutorial by Frederik Vos, a Linux trainer and evangelist and a senior technical trainer of virtualization technologies, such as Citrix XenServer and VMware vSphere.

-- post by Frederik Vos, provided by Packt --

Software management

In the old days, installing software was a matter of extracting an archive to a filesystem. There were several problems with this approach:

  • It was difficult to remove the software if the files were copied into directories that were also used by another software
  • It was difficult to upgrade software, maybe because the files were still in use or were renamed
  • It was difficult to handle shared libraries

That's why Linux distributions invented software managers.

The RPM software manager

In 1997, Red Hat released the first version of their package manager, RPM. Other distributions such as SUSE adopted this package manager. RPM is the name of the rpm utility, as well as the name of the format and the filename extension.

The RPM package contains the following:

  • A CPIO archive
  • Metadata with information about the software, such as a description and dependencies
  • Scriptlets for pre and post-installation scripts

In the past, Linux administrators used the rpm utility to install/update and remove software on a Linux system. If there was a dependency, the rpm command was able to tell exactly which other packages you needed to install. However, the rpm utility couldn’t fix the dependencies or possible conflicts between packages.

Nowadays, the rpm utility isn’t used any longer to install or remove software; instead, you use more advanced software installers. After the installation of software with yum (Red Hat/CentOS) or zypper (SUSE), all the metadata goes into a database. Querying this rpm database with the rpm command can be very handy.

A list of the most common rpm query parameters are as follows:

Parameter Description
-qa List all the installed packages.
-qi <software> List information.
-qc <software> List the installed configuration files.
-qd <software> List the installed documentation and examples.
-ql <software> List all the installed files.
-qf <filename> Shows the package that installed this file
-V <software> Verifies the integrity/changes after the installation of a package; use -va to do it for all installed software.
-qp Use this parameter together with other parameters if the package is not already installed. It's especially useful if you combine this parameter with --script to investigate the pre and post-installation scripts in the package.

The following screenshot is an example of getting information about the installed SSH server package:

The output of the -V parameter indicates that the modification time has changed since the installation. Now, make another change in the sshd_config file:

If you verify the installed package again, there is an S added to the output, indicating that the file size is different, and a T, indicating that the modification time has changed:

Other possible characters in the output are as follows:

S File size
M Mode (permissions)
5 Checksum
D Major/minor on devices
L Readlink mismatch
U User ownership
G Group ownership
T Modification time
P Capabilities

For text files, the diff command can help show the differences between the backup in the /tmp directory and the configuration in /etc/ssh:

You can also restore the original file as follows:

The DPKG software manager

The Debian distribution doesn't use the RPM format; instead, it uses the DEB format invented in 1995. The format is in use on all Debian and Ubuntu-based distributions.

A DEB package contains:

  • A file, debian-binary, with the version of the package
  • An archive file, control.tar, with metadata (package name, version, dependencies, and maintainer)
  • An archive file, data.tar, containing the actual software

Management of DEB packages can be done with the dpkg utility. Like rpm, the utility is not in use any longer to install software. Instead, the more advanced apt command is used. All the metadata goes into a database, which can be queried with dpkg or dpkg-query.

The important parameters of dpkg-query are as follows:

-l Lists all the packages without parameters, but you can use wildcards, for example, dpkg -l *ssh*
-L <package> Lists files in an installed package
-p <package> Shows information about the package
-s <package> Shows the state of the package

The first column from the output of dpkg -l also shows a status as follows:

The first character in the first column is the desired action, the second is the actual state of the package, and a possible third character indicates an error flag (R). ii means that the package is installed.

The possible desired states are as follows:

  • (u) unknown
  • (h) hold
  • (r) remove
  • (p) urge

The important package states are as follows:

  • n(ot) installed
  • H(a)lf installed
  • Hal(F) configured

Software management with YUM

Your Update Manager or Yellowdog Updater Modified (YUM) is a modern software management tool that was introduced by Red Hat in Enterprise Linux version 5, replacing the up2date utility. It is currently in use in all Red Hat-based distributions but will be replaced with dnf, which is used by Fedora. The good news is that dnf is syntax-compatible with yum.

Yum is responsible for:

  • Installing software, including dependencies
  • Updating software
  • Removing software
  • Listing and searching for software

The important basic parameters are as follows:

Command Description 
yum search Search for software based on package name/summary
yum provides Search for software based on a filename in a package
yum install Install software
yum info Information and status
yum update Update all software
yum remove Remove software

You can also install patterns of software, for instance, the pattern or group File and Print Server is a convenient way to install the NFS and Samba file servers together with the Cups print server:

Command Description
yum groups list List the available groups.
yum groups install Install a group.
yum groups info Information about a group, including the group names that are in use by the Anaconda installer. This information is important for unattended installations.
yum groups update Update software within a group.
yum groups remove Remove the installed group.

Another nice feature of yum is working with history:

Command Description
yum history list List the tasks executed by yum
yum history info <number> List the content of a specific task
yum history undo <number> Undo the task; a redo is also available

The yum command uses repositories to be able to do all the software management. To list the currently configured repositories, use:

To add another repository, you'll need the yum-config-manager tool, which creates and modifies the configuration files in /etc/yum.repos.d. For instance, if you want to add a repository to install Microsoft SQL Server, use the following:

The yum functionality can be extended with plugins, for instance, to select the fastest mirror, enabling the filesystem / LVM snapshots and running yum as a scheduled task (cron).

Software management with Zypp

SUSE, like Red Hat, uses RPM for package management. But instead of using yum, they use another toolset with Zypp (also known as libZypp) as backend. Software management can be done with the graphical configuration software YaST or the command-line interface tool Zypper. The important basic parameters are as follows:

Command Description
zypper search Search for software
zypper install Install software
zypper remove Remove software
zypper update Update software
zypper dist-upgrade Perform a distribution upgrade
zypper info Show information

There is a search option to search for a command, what-provides, but it's very limited. If you don't know the package name, there is a utility called cnf instead. Before you can use cnf, you'll need to install scout; this way, the package properties can be searched:

After this, you can use cnf:

If you want to update your system to a new distribution version, you have to modify the repositories first. For instance, if you want to update from SUSE LEAP 42.3 to version 15.0, execute the following procedure:

  1. First, install the available updates for your current version:

  1. Update to the latest version in the 42.3.x releases:

  1. Modify the repository configuration:

  1. Initialize the new repositories:

  1. Install the new distribution:

  1. Now, reboot after the distribution upgrade.

Besides installing packages, you can also install the following:

  • patterns: Groups of packages, for instance, to install a complete web server including PHP and MySQL (also known as a lamp)
  • patches: Incremental updates for a package
  • products: Installation of an additional product

To list the available patterns, use:

To install them, use:

The same procedure applies to patches and products. Zypper uses online repositories to view the currently configured repositories:

You can add repositories with the addrepo parameter, for instance, to add a community repository for the latest PowerShell version on LEAP 15.0:

If you add a repository, you must always refresh the repositories:

Software management with apt

In Debian/Ubuntu-based distributions, software management is done via the apt utility, which is a recent replacement for the utilities, apt-get and apt-cache.

The most-used commands include:

Command Description
apt list List packages
apt search Search in descriptions
apt install Install a package
apt show Show package details
apt remove Remove a package
apt update Update catalog of available packages
apt upgrade Upgrade the installed software
apt edit-sources Edit the repository configuration

Repositories are configured in /etc/apt/sources.list and files in the /etc/apt/sources.list.d/ directory. Alternatively, there is a command, apt-add-repository, available:

The apt repositories have the concept of release classes:

  • Old stable, tested in the previous version of a distribution
  • Stable
  • Testing
  • Unstable

They also have the concept of components:

  • Main: Tested and provided with support and updates
  • Contrib: Tested and provided with support and updates, but there are dependencies that are not in main, but for instance, in non-free
  • Non-free: Software that isn't compliant with the Debian Social Contract Guidelines (https://www.debian.org/social_contract#guidelines)

Ubuntu adds some extra components:

  • Universe: Community provided, no support, updates possible
  • Restricted: Proprietary device drivers
  • Multiverse: Software restricted by copyright or legal issues

If you found this article interesting, you can explore Frederik Vos’ Hands-On Linux Administration on Azure to administer Linux on Azure. Hands-On Linux Administration on Azure will help you efficiently run Linux-based workloads in Azure and make the most of the important tools required for deployment.

Cheers!