Terraform code project and use of Terragrunt (2023)

Terraform allows you to organize your code however you like.

This gives you a lot of flexibility and makes it easy to get started by just putting some resources in a file and running it.apply terraforming

But as your environment grows, you must be more disciplined in structuring your code.

This post is about how we started with a simple layout of freeform files in a single folder and then moved to Terragrunt to solve some of the issues we were having.

The first thing you need when working with Terraform is to manage the state file.

In the Terraform state, Terraform stores information about the features you create with Terraform.

If you are an individual developer, you can keep themterraform.tfstateArchive it locally on your machine (not logged to Git as it may contain potentially sensitive information), but if you're working in a team, you'll need two things:

1. A way to securely share this state file
2. A way to prevent multiple developers from modifying the state file at the same time

fortunately thes3The backend type solves all these problems:https://www.terraform.io/docs/language/settings/backends/s3.html

This saves theterraform.tfstatein an S3 bucket and uses DynamoDB to acquire an exclusive lock before performing any action that requires the state file.

However, this raises some new issues, how do you create the S3 bucket and DynamoDB table first?

There are numerous solutions, you can do it with cloud formation or manually.

But we wanted to keep everything in Terraform, so we have a module that creates an S3 bucket and a DynamoDB table.

This is instantiated once in each environment on aOhrThe folder that gives us the environment-specific S3 buckets and DynamoDB tables for the "main" Terraform state and the Terraform state for these two startup resources is stored in Git.

These 2 initial resources are never modified after initial creation, so we don't care about locking to protect this state from concurrent modifications, and it doesn't contain any sensitive data, so we're happy to push it to Git.

Now that state file management is taken care of, you can define some of the resources you need to create.

A starting point is here:

terraformar/
Developer/
Ohr/
bootstrap.tf
terraform.tfstatestaging/
Ohr/
bootstrap.tf
terraform.tfstatepuncture/
Ohr/
bootstrap.tf
terraform.tfstate

Each of these folders is a separate Terraform state, so in case you make a mistakeTerraforming-Appin one environment does not affect others.

In each folder, you must decide how to split the files that contain the module and feature definitions.

You could only have a giantterraform.tffile and paste each definition into it.

But you will have a lot of merge conflicts and find it almost impossible to figure out where certain resources are defined.

So a reasonable layout looks like this:

/terraformar
Developer/
application_a.tf
-> Resources/modules for application A
application_b.tf
-> Resources/module for application B
vpc.tf
-> VPC/sub-redes/gateways NAT etc.
eks.tf
-> Cluster K8s

Brace yourself for your exact set of technologies, but you get the point.

variables

The next thing you'll want is a way to define variables. Modules for App A should probably take App A's name so they can tag resources/add IDs so you can insert a local block.application_a.tf

local people {
app_a_name = "A"
}

And then reference it in the module definition.

Module app_a_resources {
app_name = local.app_a_name
}

Note that each file in theDeveloper/The directory accesseslocal peopleblock so you can't just call the variableApplication Name

You probably also have some variables that need to be the same across all your applications, for example your organization may have agreements on when maintenance windows apply to RDS instances.

So either copy and paste the window into the module definition for every database in every application, or set it to a location somewhere and refer to the location in every module that needs it.

problems with this approach

This will probably work fine for a while, but you'll find that you'll have some issues.

1. Because your entire environment is a single terraform stateterraformaroperations likethe planjto useThey will take longer and longer to process all of the area's resources.
2. like youterraformarrepo grows, the layout becomes more inconsistent.

People will name filese_resources.tfinstead ofapplication_e.tf

I'll find the local definition for forgotten/notwindow_maintenance="mon12:13"and create new local definitions likemaint_windowor just encode a string as a variable.

People will create Terraform modules with subtle variations in variable names,Application Nameit will beApplicationin some places,subnet_idit will besub-antiredepositionin others.

This isn't a big problem, you can just remap the local name to match the module definition.

a_resources-Modul {
app = local.app_name
}

But when people try to read/modify and extend the code, errors occur because people don't notice the subtle changes in the variable name.

Eventually you will have a different AWS Region and now you need itMaintenance windowbe different to account for time zones and your clients that are active at different times.

Don't worry, believe meeu_maintenance_windowand in everythingUEWrite module definitions

eu_a_resources-Modul {
Maintenance_window = local.eu_maintenance_window
}

One day someone will copy and pasteMaintenance windowSetting when creating a UE module and you forget to change it so that the database is restarted during peak hours for your EU customers.

The ideal would be to change the nameMaintenance windowAus_maintainence_windoweverywhere, but in reality, time pressure often means that these refactors are not quite finished.

3. Your list of.tfFiles in a single directory now get very large, you have togrepoften find what you are looking for.

4. One day, someone quickly defines an additional feature toapp_cEmdeveloperofapp_c_resourcesmodule and directly intodev/app_c.tffile so they can test something quickly.

You forget to move the resource to the module and so onapp_cis turned on, a feature is missing, or changes were not copied and pasteddev/app_c.tfAprod/app_c.tf

Most of these issues can be addressed, rigorous code reviews and refactoring commits when variable names get lost can keep your Terraform repository clean and tidy.

However, increasing the size of your terraform state will slow things down.

Also remember that only one person can get the lock to change the state file, so you might be stuck waiting for a round to runapply terraforming

Finally, if you accidentally corrupt state, it can affect a lot of resources (make sure you have S3 versioning turned on!).

To solve these problems, we started using itground gruntthat worked really well.

I have long struggled to understand what Terragrunt's purpose was or what exactly he did.

The best way to explain it is that Terragrunt places restrictions on how you can organize your Terraform code, forcing you to use directory structure hierarchies and shared variable definition files to organize your code.

These limitations force your code to be more consistent and make it harder to make mistakes, limiting the flexibility you have.

I don't think I would have really appreciated Terragrunt without first going through the above and seeing the problems of just being able to freestyle.

Logical organization of your infrastructure

To use Terragrunt, you must decide how to logically divide your infrastructure into smaller and smaller groups.

Ex. , database, cache, S3 bucket).

And then you have something like a folder structure

Developer/
-> wir-ost-1/
-> Applications/
-> Application-a
-> Database/
-> terragrunt.hcl
-> hidden/
-> terragrunt.hcl
-> cube s3/
-> terragrunt.hcl

Your directory structure represents how your infrastructure is organized.

ANDterragrunt.hclThe files are what Terragrunt reads to understand which Terraform module to apply, more on that later.

Now, what if we add another application that just needs an IAM role?

Developer/
-> wir-ost-1/
-> Applications/
-> Application-a
-> Database/
-> terragrunt.hcl
-> hidden/
-> terragrunt.hcl
-> cube s3/
-> terragrunt.hcl
-> app-b/
-> ya-roll/
-> terragrunt.hcl

That's great, but what about shared resources across a region like an EKS cluster?

You can place folders wherever you like, so let's make some at the environment level.

Developer/
-> wir-ost-1/
-> Ex-Cluster/
-> terragrunt.hcl-> Applications/
-> Application-a
-> Database/
-> terragrunt.hcl
-> hidden/
-> terragrunt.hcl
-> cube s3/
-> terragrunt.hcl
-> app-b/
-> ya-roll
-> terragrunt.hcl

Well why do we putapplication-ajapp-bin a folder calledforms?

Couldn't they be among them?we-east-1?

You could have it, but if you have a lot of applications in your company, it might be better to group them all in a subfolder for argument and give you the opportunity to have a single common settings file for all your applications. forms

inventory files

Let's talk about shared values ​​files, another great feature of Terragrunt.

Each Terraform module receives input variables that control the precise details of the features it creates.

Some of these input variables are likely to be the same for all resources deployed in an environment, e.g.environment namefor things like lyrics.

instead of writingenvironment name=developmentIn eachterragrunt.hclfile, we set all these environment level variables in a file calledenvironment.yaml

Developer/
 -> Environment.yaml-> wir-ost-1/
-> Ex-Cluster/
-> terragrunt.hcl-> Applications/
-> application-a/
-> Database/
-> terragrunt.hcl
-> hidden/
-> terragrunt.hcl
-> cube s3/
-> terragrunt.hcl
-> app-b/
-> ya-roll/
-> terragrunt.hcl

Yenvironment.yamlThis will look like this:

Environment name: dev

Next we have some region-level settings, for example inwe-east-1we can want ourMaintenance windowtherefore, RDS instances duringNoon 5am to 7ama peaceful time for our US customers.

Developer/
-> Environment.yaml-> wir-ost-1/
 -> region.yaml -> Ex-Cluster/
-> terragrunt.hcl-> Applications/
-> application-a/
-> Database/
-> terragrunt.hcl
-> hidden/
-> terragrunt.hcl
-> cube s3/
-> terragrunt.hcl
-> app-b/
-> ya-roll/
-> terragrunt.hcl

Yregion.yamlthis is how it will look

Maintenance window: Mon 05:00-07:00

Now when we expand to EU and it was the quiet time for our EU customersMarch 21st at 10pmwe could do something like this:

Developer/
-> Environment.yaml-> wir-ost-1/
 ->region.yaml<SNIP>
 -> eu-west-1/
-> region.yaml
-> Applications/
-> app-w/
-> Database/
-> terragrunt.hcl

Yeu-west-1/region.yamlwould seem

Maintenance window: Tue 21:00-22:00

I'm sure by now you're starting to get the idea that you can easily propagate the shared variable between subtrees, allowing you to easily customize your infrastructure without having to copy and paste values ​​everywhere.

The last shared variable we create will be for each app, the app name will likely appear somewhere in the resources for that.

Developer/
-> Environment.yaml-> wir-ost-1/
-> region.yaml
-> Ex-Cluster/
-> terragrunt.hcl-> Applications/
-> application-a/
 -> Application.yaml
-> Database/
-> terragrunt.hcl
-> hidden/
-> terragrunt.hcl
-> cube s3/
-> terragrunt.hcl
-> app-b/
 -> Application.yaml
-> ya-roll/
-> terragrunt.hcl

and everyoneapp.yamlIt will look something like this:

Application name: <NAME>

Now, not all input variables are generic, some are specific to individual modules.

For these, you can hand them directly to the module.

let's see whatterragrunt.hclit's really:

terraformar {
source="Link to Terraform module on Github"
}contains {
path = look_in_parent_folders()
}entries = {
module_specific_variable = "sorvente"
}

It's a very simple file.

You need a link to the module you want to apply, it only works with Github links, so there are no references to Terraform module registration here.

Ignoreto containFor now it's about finding these shared variables files, we'll come back to that

beginnerHere you can transfer all entries to modules.

Yes, oneProhibitedThe variable has the same name as a variable defined in one of our shared variables.yamlthen it will be collected automatically.

If the variable is not from a shared definition file, you can manually enter it here.

There are a few more things to mention about input variables while we're here.

Let's configure Terragrunt to use the first definition of any variable found so that we can substitute the generic values ​​provided by.yamlFiles with more specific ones further down the tree.

You will probably also encounter situations where you named the generic variable this wayenvironment namebut some modules expect it to be calledSurroundingsÖnumber_env.

This is a nice feature of Terragrunt because it quickly reveals where your modules aren't consistent and you can work to align things over time.

In the short term, you can map generic variable names to more specific module names.

local people {
env_vars = yamldecode(
archivo("${find_in_parent_folders("environment.yaml")}"),
)
}Beginner {
env_name = local.env_vars['nome do ambiente']
}

dependencies

After all, the input of one module will likely be the output of another.

Let's say our for exampleEx-Clustermodule outputsworker-sg-idthe security group ID used by the K8 team.

so ourData baseThe module receives an input parameter ofsg_to_allow_access_fromthe ID of a security group for which you are creating an ingress rule.

You can use the output of one module as input to another like this...

dependency "k8s-sg-worker" {
config_path = "../../eks-cluster"
}entries = {
sg_to_allow_access_from=dependency.k8s-worker-sg.outputs.worker-sg-id
}

This pretty much covers everything you need to style your Terragrunt files and not copy and paste your variables everywhere.

Terragrunt configuration file

But we still can't walkbodenHowever, some configuration is required to connect everything.

We've already seen that Terraform's state file includes all of our environment's resources.

This caused terraform commands to take longer to execute as our environment grew and we risked hitting all resources if we corrupted the state in any way.

In this Terragrunt configuration, we can create one state file per "leaf node" of the directory tree, essentially where there is aterragrunt.hclfile that defines a module to be applied, we create a new state file.

This makes Terraform operations super fast and reduces the consequences of corrupting a state file.

We need to create aterragrunt.hclfile indev/us-east-1/terragrunt.hcland instead of setting a Terraform module to apply it sets all our Terragrunt settings like everyone elseterragrunt.hclImport files withto containstatement we saw earlier

contains {
path = look_in_parent_folders()
}

search_in_parent_foldersis a Terragrunt built-in function that returns the firstterragrunt.hclFile found in parent folders.

So let's start with ourdev/us-east-1/terragrunt.hclArchive and define our state settings

remote_state {
Infrastructure = "s3"
configuration = {
Bucket="S3-CUBE-NAME"
clave = "${path_relative_to_include()}/terraform.tfstate"
region="AWS-REGION"
encrypt = true
dynamodb_table = "TABLA DYNAMO DB"
}
}

This uses S3 and DynamoDB to store state/get exclusive locks on the state file, as we've seen before without Terragrunt.

ButClave = ...line means that there will be a directory structure inside the S3 bucket that mimics the structure of Terragrunt folders with state files.

Also, remember how in pure Terraform we had to do some fiddly things to initialize the S3 bucket table and DynamoDB?

Terragrunt automatically creates them if they don't exist and solves this whole problem.

However, this has a downside: every Terraform module you apply must be defined

terraformar {
server "s3" {}
}

In the module so that Terragrunt can fill in the details when it's running.

There is a Github issue thread from 2017 that discusses this:https://github.com/gruntwork-io/terragrunt/issues/230

Now we need to tell Terragrunt where to find all the shared variables files we've defined.

entries = merge(
yamldecode(
file("${find_in_parent_folders("environment.yaml", find_in_parent_folders("environment.yaml"))}"),
),
yamldecode(
file("${find_in_parent_folders("region.yaml", find_in_parent_folders("environment.yaml"))}"),
),
yamldecode(
file("${find_in_parent_folders("app.yaml", find_in_parent_folders("environment.yaml"))}"),
),
)

like old timessearch_in_parent_folderscauses Terragrunt to look from where the modules were defined in the tree to find the first occurrence of the file.

The second parameter tofileis a default value that you can use if you cannot find the file. Here we resort to itenvironment.yamlthat we're going to make sure it's always there, and it means that if we have a situation where a module isn't nested deep enough for allapp.yaml,region.yamljenvironment.yamlFiles in their parent directories will not explode.

tieis Terraform's default function —https://www.terraform.io/docs/language/functions/merge.html

This means that variables defined at the bottom of the tree override the ones at the top.

One last configuration option and you're done.

The AWS provider we use needs to be configured. We can use Terragrunt's ability to generate files to place an identical configuration in the working directory before running Terraform instead of copying and pasting hundreds of times.

generate "aws_provider" {
ruta = "aws_provider.tf"
if_exist = "overwrite_terragrunt"
Content = << EOF
provider "aws" {
region="us-east-1"
}
weekend
}

This will generate a file calledaws_provider.tfin the working directory when running a command that passes our required options.

Now we can run Terragrunt commands and build our infrastructure.

There are two options for running Terragrunt commands:

1. In multiple folders, allowing us to create multiple modules at the same time
2. In a single module for faster/specific applications.

run all commands

If you want to run commands in multiple modules at the same time, theall runcommands can do this.

For example, when you runTerragrunt master planinsidedeveloperrun directoryTerraforming Planin each subdirectory and display the plan.

The first time you run commands on a folder, it will check if the Terraform state files and lock tables exist and prompt you to create them if they don't.

A single module is applied

If you only want to apply changes to a single Terraform module, you can do soCDin that directory and runTerragrunt <COMMAND>and only applies to the current working directory.

Modul-Caching

An annoying quirk of Terragrunt is that once a module has been downloaded, it will not recover it if the source changes.

For example if you haveref = github.com/module?ref=mi-sucursaland submit new changesmy branchafter i ranApply TerragruntYou won't notice that the font has changed and you will receive the new changes.

You should also clear Terragrunt's cacheem contraste com . -type d -name ".terragrunt-cache" -prune -exec rm -rf {} \;

We now have a fully functional Terragrunt setup, to sum up what we gain:

• Small per-module state files that speed up executionterraformarcommands and reduce the impact of losing/corrupting a single state file
• A well-defined framework for designing our infrastructure based on directory structures that reflect how the infrastructure is logically grounded
• The ability to use shared variables files, which allow multi-level variables to be automatically defined and propagated to Terraform modules to facilitate multi-level configuration consistency, with the ability to substitute more general values, if necessary.
• The ability to make one module dependent on another by building the modules in the correct order and passing output variables between them
• The inability to create random ad hoc features outside of a versioned Terraform module, so we reduce the chances of features/changes not propagating to other environments.
• The ability to automatically retry Terraform commands when certain errors occur, reducing the impact of broken/eventually consistent APIs

It's not perfect, although some limitations/problems:

• Every Terraform module you use must define a blankInternal processblock, which means you may have to modify any modules you have, and you cannot use community-sourced modules
• You cannot use Terraform's module registry references, which will lose the ability to freely specify locked versions and possibly mean that you must provide Git authentication credentials for Terragrunt to fetch modules from Github.
• You and all other team members need to learn about Terragrunt.
• Now you have one more tool to go along with Terraform
• Unless you've been disciplined about naming your module's input variables consistently, you won't get many of the benefits of shared variable files without some refactoring.

There are many code blocks in this post, you can see them all together in an example repository here:https://github.com/AaronKalair/ejemplo-terragrunt-repo