Terraform Data Source and for_each – Manage Multiple Resources

Do you want to target multiple resources in your Terraform configuration code? If yes, your best bet is to use the powerful combination of Terraform data source and for_each looping.

While data sources in Terraform allow you to create dynamic infrastructure, the for_each keyword helps you extend the flexibility to multiple resources with a single statement.

In this post, I will dive into the ins and outs of using for_each with Terraform data sources to streamline your infrastructure management. To make things useful, I will be explaining the majority of aspects using code examples.

1 – Terraform Data Source for_each Example

Using for_each with Terraform data sources helps you target a list of resources and iterate over them. For example, you may have a list of EC2 instances, RDS instances or S3 buckets.

Check the below example of a Terraform configuration:

provider "aws" {
    region = "us-west-2"
    profile = "terraform-user"
}

variable "instances" {
  type = map(string)
  default = {
    "i-0f6834cf2cd8ecd10" = "web-server-1"
    "i-07b110fdd78605e2b" = "web-server-2"
  }
}

data "aws_instance" "example" {
  for_each = var.instances
  instance_id = each.key
}

output "ip_addresses" {
  value = [for instance in data.aws_instance.example : instance.public_ip]
}

output "tags" {
  value = [for instance in data.aws_instance.example : instance.tags]
}

This example uses the aws_instance data source with the for_each argument to reference multiple EC2 instances in a single data source block.

The instances variable is defined as a map that contains the instances that you want to reference. The key is the instance ID and the value is the name of the instance.

Next, you have the aws_instance data source that uses the for_each argument to iterate over the instances defined in the instances variable. For each instance, the instance_id argument is set to each.key. Basically, the key is the instance id.

At the end of the configuration, you declare a couple of outputs – one for the public IP address and the other for the tags associated with the instance. The for argument is used to iterate over the instance information.

If you run the above configuration (using terraform init followed by terraform apply) after creating a couple of EC2 instances in the AWS configuration, you should see a similar response as below:

Outputs:

ip_addresses = [
  "34.221.168.142",
  "18.237.73.45",
]
tags = [
  tomap({
    "Name" = "web-server-2"
  }),
  tomap({
    "Name" = "web-server-1"
  }),
]

2 – Practical usage of for_each with Terraform Data Source

The previous example was good to clear the basics about the using for_each argument with a Terraform data source.

However, you don’t usually end up only displaying the instance details as output. You would want to perform some meaningful action on multiple resources with for_each.

For example, let’s say you have a couple of EC2 instances and you want to attach EBS volumes to each of those instances.

Here’s how the configuration may look like:

provider "aws" {
    region = "us-west-2"
    profile = "terraform-user"
}

variable "instance_ids" {
  type = set(string)
  default = ["i-04c6c45ee7f12449f","i-0c8b17c4014283e16"]
}

variable "volume_size" {
    type = number
    default = "8"
}

data "aws_instance" "example" {
  for_each = var.instance_ids
  instance_id = each.value
}

resource "aws_ebs_volume" "example" {
  for_each = data.aws_instance.example
  availability_zone = data.aws_instance.example[each.key].availability_zone
  size = var.volume_size
}

resource "aws_volume_attachment" "example" {
  for_each = data.aws_instance.example
  device_name = "/dev/sdb"
  volume_id = aws_ebs_volume.example[each.key].id
  instance_id = data.aws_instance.example[each.key].id
}

You have a variable block that contains a list of EC2 instance IDs. The data block uses the for_each argument to reference all the instances with those IDs.

The first resource block uses for_each argument to create an EBS volume for each instance. In this block, you also set the availability zone of the volume to the availability zone of the instance. Also, you set the size of the volume.

The second resource block creates the attachment between the volume and the instance. It references the data source as well as the EBS volume resource.

You can execute the plan using the command terraform apply.

3 – Pitfalls of using Terraform Data Source with for_each

While for_each is extremely useful in a Terraform data source, you need to be careful about a few pitfalls:

• Performance – Using for_each can significantly increase the number of API calls made to the provider. This can slow down Terraform and lead to longer execution times.
• Data Mismatch – When using for_each with data sources, you need to make sure that the data source and the resource has the same key/value pairs. In case of mismatch, Terraform might not be able to create the resource correctly.
• Missing Data Attribute – You might be using data source with for_each to fetch a bunch of records and using some attribute in those records. It is important to make sure that the attribute is actually present in the object.
• Memory Usage – You need to be mindful of the amount of data being pulled from the data source when using for_each. If the data is too large, it can cause memory issues or even crash the Terraform process.

Conclusion

In conclusion, using for_each with Terraform data sources is a powerful technique for handling multiple resource in a single process and streamlining your infrastructure management.

However, the ability of for_each to iterate over multiple resources also leads to potential pitfalls such as performance, data mismatch and memory issues.

Want to explore more data source features? You can read about Terraform conditional data source.

Alternatively, you can also manage multiple resources using Terraform count meta-argument.

If you found this post useful, consider sharing it with friends and colleagues.