Terraform is one of the most popular open source infrastructure-as-code (IaC) solution. It makes it super easy to declaratively define your Infrastructure with all the major public clouds. We have used it extensively to build Opta.

But no tool is perfect and there are a lot of annoyances with Terraform.

Terraform uses state files to record the current infrastructure state. But the reality of the infrastructure could deviate from the state terraform expects due to multiple reasons.

• Terraform apply failed: Terraform apply can fail mid way due to many reasons and ends up in a state where the resource is created but not recorded in the state. And any further applys fail with “resource already exists” errors. You would end up either importing that resource manually or deleting the resource which was created incompletely.
• Manual changes to resources directly: Though this is a feature rather than a bug, but if during an incident you end up quickly fixing some configuration through the any other UI than terraform. This needs to be done in cases where the person on call doesn’t have all the permissions to run terraform or running it is too slow or terraform state is broken for some other reason. Then it becomes really tricky to bring terraform state in sync.

Terraform makes it very easy to reference the outputs of other resources/modules. But it only works if all of these resources are part of a single state. This leads to many issues:

• Permissions creep: Since any terraform plan/apply would go through all the resources and check their status. Anyone who needs to run terraform plan/apply would need an extensive set of permissions. For example: if I manage a database which is in a VPC exported by another resource. I would need permissions for the VPC as well.
• Lock conflicts: If the state gets too big, a lot of people end up running terraform commands. And since, every terraform command acquires a single global lock to run successfully, it becomes very hard to coordinate updates.

Ideally some of these problems are resolved if terraform is run from some centralized CI platform but most engineers would still be running terraform plan outside that.

Terraform produces an overly verbose plan of changes, which can be very tricky to read and a small change (like a tags change) can result in a plan where the more important changes can get lost in all the noise. In a large team, more infrequent users end up ignoring the plan which can cause various issues. It would be nice to also have a more human digestible change alongside the detailed option. We ended up completely revamping the plan interface for end users in Opta, where we give a risk score to each change based on some manual rules.

Most of the terraform modules are written by external providers. And the quality of these modules vary a lot. Even with the very popular providers like AWS, newer modules can have many bugs. For example with the EKS module, often times all resources are not deleted after running terraform destroy. Also updating module version can also be tricky, it usually require manual work, and dealing with non backward compatible versions

HCL is terraform’s DSL to declare the infrastructure. But it has a steep learning curve. Also it has a bunch of known limitations.

Teams usually want to spin up multiple environments by changing a few terraform variables like account_id, or environment_name etc. But this becomes super tricky for any medium/large sized configuration in terraform. This is partially caused by hard dependencies which I discussed earlier since you can’t share variables across terraform states. Each module within a given state can have a few variables based on which environment is deployed. For example, you want to use different instance types in prod vs qa. Managing all of this in HCL in a single large state file leads to a spaghetti monster.

To conclude, terraform is an essential tool in today’s devops driven engineering organizations. It has a huge community behind it and has evolved a lot in the past. But it is no golden hammer and and has its own fair share of shortcomings.