terraform | Tom's Notes

I like Terraform! 🙂

Had an idea the other day – I wanted to make myself an Ubuntu virtual machine on Microsoft Azure and also create a corresponding DNS record inside my kaniski.eu domain (currently hosted at CloudFlare… like, I guess, half of the Internet). Of course, I can do all that by using both of the portals, but this time I wanted to do it with Terraform, so it can be easily brought up and down, reusable, etc.

For this, you’ll need:

Microsoft Azure subscription
CloudFlare account (with a domain attached to its DNS servers, of course)
Terraform (v0.13)
(optional) Visual Studio Code
(or you can all do it inside Azure Cloud Shell)

To configure Terraform with your Azure subscription, you’ll need:

Azure Subscription ID
Tenant ID
Client ID
Client secret

There’s a nice document about setting it all up – https://docs.microsoft.com/en-us/azure/developer/terraform/overview.

For accessing CloudFlare API via Terraform, you’ll need the following info from your CloudFlare account page:

e-mail address
CloudFlare zone ID
CloudFlare account ID

CloudFlare API key

Be really careful with using your Azure/CloudFlare keys (can’t be stressed enough!)!

Also note that the following code is only one way of doing it, one that suits me for this purpose, not best practice or the prettiest code ever! 🙂

The whole example (main.tf) looks like this (it can/should be improved, but that’s not really the point here and now):

# providers
provider "azurerm" {
  version = "~> 2.24"

  subscription_id = var.subscription_id
  tenant_id       = var.tenant_id
  client_id       = var.client_id
  client_secret   = var.client_secret

  features {}
}

provider "cloudflare" {
  version = "~> 2.10.0"

  email      = var.cf_email
  api_key    = var.cf_apikey
  account_id = var.cf_accountid
}

# resource group
resource "azurerm_resource_group" "example-rg" {
  name     = "example-rg"
  location = "West Europe"
}

# network
resource "azurerm_virtual_network" "example-net" {
  name                = "example-net"
  location            = azurerm_resource_group.example-rg.location
  resource_group_name = azurerm_resource_group.example-rg.name
  address_space       = ["10.11.0.0/16"]
}

# subnet
resource "azurerm_subnet" "example-sub" {
  name                 = "example-sub"
  resource_group_name  = azurerm_resource_group.example-rg.name
  virtual_network_name = azurerm_virtual_network.example-net.name
  address_prefixes     = ["10.11.12.0/24"]
}

# nics
resource "azurerm_network_interface" "myvm-nic" {
  name                 = "myvm-nic"
  location             = azurerm_resource_group.example-rg.location
  resource_group_name  = azurerm_resource_group.example-rg.name
  enable_ip_forwarding = true

  ip_configuration {
    name                          = "myvm-nic-ip-config"
    subnet_id                     = azurerm_subnet.example-sub.id
    private_ip_address_allocation = "Static"
    private_ip_address            = "10.11.12.4"
    public_ip_address_id          = azurerm_public_ip.myvm-ip.id
  }
}

# public ips
resource "azurerm_public_ip" "myvm-ip" {
  name                = "myvm-ip"
  location            = azurerm_resource_group.example-rg.location
  resource_group_name = azurerm_resource_group.example-rg.name
  allocation_method   = "Static"
}

# network security group
resource "azurerm_network_security_group" "example-nsg" {
  name                = "example-nsg"
  location            = azurerm_resource_group.example-rg.location
  resource_group_name = azurerm_resource_group.example-rg.name

  security_rule {
    name                       = "SSH"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "10.11.12.4"
  }
}

resource "azurerm_subnet_network_security_group_association" "example-sub-nsg-assoc" {
  subnet_id                 = azurerm_subnet.example-sub.id
  network_security_group_id = azurerm_network_security_group.example-nsg.id
}

# vms
resource "azurerm_virtual_machine" "myvm" {
  name                          = "myvm"
  location                      = azurerm_resource_group.example-rg.location
  resource_group_name           = azurerm_resource_group.example-rg.name
  network_interface_ids         = [azurerm_network_interface.myvm-nic.id]
  vm_size                       = "Standard_DS1_v2"
  delete_os_disk_on_termination = true

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "19_10-daily-gen2"
    version   = "latest"
  }

  storage_os_disk {
    name              = "myvm-osdisk"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
  }

  os_profile {
    computer_name  = "myvm"
    admin_username = var.admin_username
  }

  os_profile_linux_config {
    disable_password_authentication = true
    ssh_keys {
      key_data = file("~/.ssh/id_rsa.pub")
      path     = "/home/${var.admin_username}/.ssh/authorized_keys"
    }
  }
}

# cloudflare dns
resource "cloudflare_record" "myvm-dns" {
  zone_id = var.cf_zoneid
  name    = "myvm"
  value   = azurerm_public_ip.myvm-ip.ip_address
  type    = "A"
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

# providers

provider "azurerm" {

version = "~> 2.24"

subscription_id = var.subscription_id

tenant_id = var.tenant_id

client_id = var.client_id

client_secret = var.client_secret

features {}

}

provider "cloudflare" {

version = "~> 2.10.0"

email = var.cf_email

api_key = var.cf_apikey

account_id = var.cf_accountid

}

# resource group

resource "azurerm_resource_group" "example-rg" {

name = "example-rg"

location = "West Europe"

}

# network

resource "azurerm_virtual_network" "example-net" {

name = "example-net"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

address_space = ["10.11.0.0/16"]

}

# subnet

resource "azurerm_subnet" "example-sub" {

name = "example-sub"

resource_group_name = azurerm_resource_group.example-rg.name

virtual_network_name = azurerm_virtual_network.example-net.name

address_prefixes = ["10.11.12.0/24"]

}

# nics

resource "azurerm_network_interface" "myvm-nic" {

name = "myvm-nic"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

enable_ip_forwarding = true

ip_configuration {

name = "myvm-nic-ip-config"

subnet_id = azurerm_subnet.example-sub.id

private_ip_address_allocation = "Static"

private_ip_address = "10.11.12.4"

public_ip_address_id = azurerm_public_ip.myvm-ip.id

}

# public ips

resource "azurerm_public_ip" "myvm-ip" {

name = "myvm-ip"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

allocation_method = "Static"

}

# network security group

resource "azurerm_network_security_group" "example-nsg" {

name = "example-nsg"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

security_rule {

name = "SSH"

priority = 100

direction = "Inbound"

access = "Allow"

protocol = "Tcp"

source_port_range = "*"

destination_port_range = "22"

source_address_prefix = "*"

destination_address_prefix = "10.11.12.4"

}

resource "azurerm_subnet_network_security_group_association" "example-sub-nsg-assoc" {

subnet_id = azurerm_subnet.example-sub.id

network_security_group_id = azurerm_network_security_group.example-nsg.id

}

# vms

resource "azurerm_virtual_machine" "myvm" {

name = "myvm"

location = azurerm_resource_group.example-rg.location

resource_group_name = azurerm_resource_group.example-rg.name

network_interface_ids = [azurerm_network_interface.myvm-nic.id]

vm_size = "Standard_DS1_v2"

delete_os_disk_on_termination = true

storage_image_reference {

publisher = "Canonical"

offer = "UbuntuServer"

sku = "19_10-daily-gen2"

version = "latest"

}

storage_os_disk {

name = "myvm-osdisk"

caching = "ReadWrite"

create_option = "FromImage"

managed_disk_type = "Standard_LRS"

}

os_profile {

computer_name = "myvm"

admin_username = var.admin_username

}

os_profile_linux_config {

disable_password_authentication = true

ssh_keys {

key_data = file("~/.ssh/id_rsa.pub")

path = "/home/${var.admin_username}/.ssh/authorized_keys"

}

# cloudflare dns

resource "cloudflare_record" "myvm-dns" {

zone_id = var.cf_zoneid

name = "myvm"

value = azurerm_public_ip.myvm-ip.ip_address

type = "A"

}

Variables used in this simple example are (terraform.tfvars):

subscription_id = ""
tenant_id       = ""
client_id       = ""
client_secret   = ""
admin_username  = ""
cf_zoneid       = ""
cf_email        = ""
cf_apikey       = ""
cf_accountid    = ""

subscription_id = ""

tenant_id = ""

client_id = ""

client_secret = ""

admin_username = ""

cf_zoneid = ""

cf_email = ""

cf_apikey = ""

cf_accountid = ""

Just note one more thing – I’m using the current version of Terraform (v0.13) and version 2 of the CloudFlare provider (notable changes between the v1 and v2 are listed at https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/guides/version-2-upgrade).

As always, running terraform init will take care of the missing providers and install them as needed. Also, running terraform plan can give us an estimation of what exactly will be done and terraform fmt will make our code look nice! 🙂

When I run this code (terraform apply), it completes successfully (of course) and provisions 9 items and, more important – gives me info about the DNS name and IP address of the provisioned VM (which I already know, but still):

Next is a series of checks:

DNS record really exists in CloudFlare DNS?

resources are really provisioned in Azure?

SSH into the VM works?

And that’s it – when you’re done, you can use terraform destroy and it’s all gone! 🙂

Cheers!

I like Terraform, because it’s so clean, fast and elegant; OK, I also suck at it, but hey – I’m trying! 🙂

The long story

Usually, Terraform and its providers are very good at doing things in the order they should be done. But sometimes people do come up with silly ideas, and mine was such (of course) – I’ve decided to rename something and it broke things. Just a little.

I have a simple lab in Azure, with a couple of virtual machines behind the Azure Load Balancer, no big deal. All this is being deployed (and redeployed) via Terraform, using the official azurerm provider. It’s actually a Standard SKU Azure Load Balancer (don’t ask why), with a single backend pool and a few probes and rules. Nothing special.

I’ve deployed this lab a few days ago (thanks to good people at Microsoft, I have some free credits to spare), everything worked just fine, but today I’ve got the wild idea – I’ve decided to rename my backend pool.

With all the automation in place, this shouldn’t be a problem… one would think. 🙂

So, as I’ve updated my code and during the make it so phase (terraform apply), I’ve got some errors (truncated, with only the useful stuff):

Error:
Error Creating/Updating LoadBalancer: network.LoadBalancersClient
CreateOrUpdate:
Failure sending request:
StatusCode=400
--
Original Error:
Code="InvalidResourceReference... .../k8sthw-lb/backendAddressPools/k8sthw-lb-pool referenced by resource
/subscriptions/fake-azure-subscription-id/resourceGroups/k8sthw-rg/providers/Microsoft.Network/... ... or verify that both resources are in the same region." Details=[]

Error:

Error Creating/Updating LoadBalancer: network.LoadBalancersClient

CreateOrUpdate:

Failure sending request:

StatusCode=400

Original Error:

Code="InvalidResourceReference... .../k8sthw-lb/backendAddressPools/k8sthw-lb-pool referenced by resource

/subscriptions/fake-azure-subscription-id/resourceGroups/k8sthw-rg/providers/Microsoft.Network/... ... or verify that both resources are in the same region." Details=[]

Issue

After going through these errors, I’ve realized that my resources are indeed in the same region, but existing rules are referencing the current backend pool by name and actually blocking Terraform in renaming the backend pool.

There are a couple of options at this stage – you can destroy your deployment and run it again (as I normally would) and it should all be fine. Or you can try to fix only the dependent resources and make it work as part of the existing deployment.

With some spare time at my hands, I’ve tried to fix it using the second one and it actually worked.

Resolution

Terraform has a nice option for destroying just some parts of the deployment.

If you look at help for the terraform destroy command, you can see the target option:

terraform destroy -help
# ...
# -target=resource Resource to target. Operation will be limited to this
#                  resource and its dependencies. This flag can be used
#                  multiple times.
# ...

terraform destroy -help

# ...

# -target=resource Resource to target. Operation will be limited to this

# resource and its dependencies. This flag can be used

# multiple times.

# ...

And if you run it to fix your issues, you’ll get a nice red warning saying that this is only for exceptional situations (and they mean it!):

terraform destroy -target azurerm_lb_outbound_rule.k8sthw-lb-out
# Warning: Resource targeting is in effect
#
# You are creating a plan with the -target option, which means that the result
# of this plan may not represent all of the changes requested by the current
# configuration.
# 
# The -target option is not for routine use, and is provided only for
# exceptional situations such as recovering from errors or mistakes, or when
# Terraform specifically suggests to use it as part of an error message.
# 
# Do you really want to destroy all resources?
# Terraform will destroy all your managed infrastructure, as shown above.
# There is no undo. Only 'yes' will be accepted to confirm.
# 
# Enter a value: yes
# ...

terraform destroy -target azurerm_lb_outbound_rule.k8sthw-lb-out

# Warning: Resource targeting is in effect

# You are creating a plan with the -target option, which means that the result

# of this plan may not represent all of the changes requested by the current

# configuration.

# The -target option is not for routine use, and is provided only for

# exceptional situations such as recovering from errors or mistakes, or when

# Terraform specifically suggests to use it as part of an error message.

# Do you really want to destroy all resources?

# Terraform will destroy all your managed infrastructure, as shown above.

# There is no undo. Only 'yes' will be accepted to confirm.

# Enter a value: yes

# ...

So… BE CAREFUL! (can’t stress this enough!)

Anyhow, I’ve destroyed rules which were preventing my rename operation:

terraform destroy -target azurerm_lb_outbound_rule.k8sthw-lb-out -target azurerm_lb_rule.k8sthw-lb-80-rule -target azurerm_lb_rule.k8sthw-lb-443-rule --auto-approve
# ...
#
# Destroy complete! Resources: 3 destroyed.

terraform destroy -target azurerm_lb_outbound_rule.k8sthw-lb-out -target azurerm_lb_rule.k8sthw-lb-80-rule -target azurerm_lb_rule.k8sthw-lb-443-rule --auto-approve

# ...

# Destroy complete! Resources: 3 destroyed.

And then another terraform apply –auto-approve recreated everything that was needed, and finally – my backend pool got renamed:

# ...
# azurerm_lb_backend_address_pool.k8sthw-controllers-lb-pool: Refreshing state... [id=/subscriptions/fake-azure-subscription-id/resourceGroups/k8sthw-rg/providers/Micro...
# ...
# 
# Apply complete! Resources: 2 added, 0 changed, 3 destroyed.

# ...

# azurerm_lb_backend_address_pool.k8sthw-controllers-lb-pool: Refreshing state... [id=/subscriptions/fake-azure-subscription-id/resourceGroups/k8sthw-rg/providers/Micro...

# ...

# Apply complete! Resources: 2 added, 0 changed, 3 destroyed.

Another idea I’ve had was to taint the resources (terraform taint -help), which would probably be a lot nicer. Oh, well… maybe next time. 🙂

As things are constantly improving, it shouldn’t be long until this is fixed (should it even be fixed?!). Until then, hope this will help you with similar issues!

Cheers!

Tom's Notes

Tag Archives: terraform

Azure VM with CloudFlare DNS… by Terraform

Fixing things with… terraform destroy