Terraform module debug

Lessons learned

The terraform module change I deployed seemed to have valid syntax, the module initialized properly with terraform init. The terraform plan created a valid plan, no errors or syntax issues were raised. When I started to execute the plan with terraform deploy it returned the following error:

Error: Error launching source instance: VPCIdNotSpecified: No default VPC for this user
        status code: 400, request id: xxx

This error pointed me towards a bug report on the Terraform github issues page, this stated:

I believe the Instance actually derives it's VPC from the subnet, 
can you verify that specifying a subnet_id in a VPC works as designed?

In my setup I was not using a Default VPC, I was creating VPC’s with modules and I was using their output, subnets as an input value for other resources and modules. I had just changed a string variable that passed a subnet_id into a string list variable. This string list variable contained a list of subnet ID’s. I blamed the error on this change and started looking how I could see the context of the variable I just created. Ansible has the -vvvv flag to raise verbosity, this often returns the context of filled in variables. I hoped I would find a way to raise verbosity in Terraform. On Google I found options to raise Terraform logs, this however was for Terraform crash logs. My Google-fu was failing me that day and I could not find a way to debug Terraform modules.

After a while I decided to raise the following question on the Terraform gitter:

Hi all, how do you debug things in Terraform?

Of course I explained the scenario I was working on and the channel fell silent, after waiting a weekend I asked again. And received the following answer: I use the terraform console to check interpolation behavior. It looks like you’re having a problem with the actual value you’re using for your resource arguments.

This made me try the Terraform console and see what results I would get. Unfortunately this would not give any result:

echo "module.subnets.private_subnet_ids" |terraform console -var-file=variables.tfvars
Error: Result depends on values that cannot be determined until after "terraform apply".

Perhaps I used the Terraform console wrong, my syntax was incorrect or I simply didn’t understand the Terraform console. It was not working and I did not get any further. So I decided to start debugging as if I was using Go; forward the values inside the module to output.

Terraform code

The problematic Terraform module:

locals {
  additional_ips_count        = var.associate_public_ip_address && var.instance_enabled && var.additional_ips_count > 0 ? var.additional_ips_count : 0

resource "aws_instance" "default" {
  count                       = var.instance_count
  ami                         = data.aws_ami.info.id
  availability_zone           = var.availability_zone
  instance_type               = var.instance_type
  ebs_optimized               = var.ebs_optimized
  disable_api_termination     = var.disable_api_termination
  user_data                   = var.user_data
  iam_instance_profile        = join("", aws_iam_instance_profile.default.*.name)
  associate_public_ip_address = var.associate_public_ip_address
  key_name                    = var.ssh_key_pair
  monitoring                  = var.monitoring
  private_ip                  = concat(var.private_ips, [""])[min(length(var.private_ips), count.index)]
  source_dest_check           = var.source_dest_check

  vpc_security_group_ids      = compact(
        var.create_default_security_group ? join("", aws_security_group.default.*.id) : ""

  root_block_device {
    volume_type               = var.root_volume_type
    volume_size               = var.root_volume_size
    iops                      = var.root_iops
    delete_on_termination     = var.delete_on_termination

resource "aws_network_interface" "extra_nic" {
  count                       = local.additional_ips_count * var.instance_count
  subnet_id                   = element(var.subnet_ids, (count.index % length(var.subnet_ids)))

  security_groups             = compact(
        var.create_default_security_group ? join("", aws_security_group.default.*.id) : ""

  depends_on                   = [aws_instance.default]

resource "aws_network_interface_attachment" "extra_nic" {
  count                = local.additional_ips_count * var.instance_count
  instance_id          = aws_instance.default.*.id[count.index % var.instance_count]
  network_interface_id = aws_network_interface.extra_nic.*.id[count.index]
  device_index         = 1 + count.index
  depends_on           = [aws_instance.default]

And the variable that caused the issues was:

variable "subnet_ids" {
  type        = list(string)
  description = "List of subnet IDs created in this network"

The alert Terraform users reading this could probably already see that I was missing something. The error however did not tell me and since I changed the subnet variable to a string list variable with key subnet_ids, the error was not helping at all, I decided to add output blocks to get some more information:

output "subnet_ids" {
  description = "Lists all the subnet IDs passed to the module."
  value       = var.subnet_ids

The output was this:

master_subnet_ids = [

Using the terraform output command now gave me the list of subnet id’s I was searching for, there was nothing wrong with the output. This made me realize that the values passed to the string list variable was correct and the issue was not with the input. Something else was wrong, but I knew now it was not with the variable itself. This made me decide to use the ec2 resource directly. Here I got the same error as before. After reading up on the documentation I found out that the subnet_id field was missing from the ec2 resource. A silly mistake, could have probably spent less time in finding it but apparently this was one of those days.

Lessons learned

Main lesson learned should be to RTFM. But who reads the manual anyway? What I took away from this experience is that terraform is far less sophisticated than Ansible in verbosity and debugging. You however have the option to add debugging to modules yourself. It will not tell you what caused the issue, but it will tell you what works and gives you a direction where to go from there.