Cake
  • Log In
  • Sign Up
    • We have a number of infrastructure as code (IaC) tools available to us (although it feels like none of them really do infrastructure as code, or configuration management really), and the more I use them, the more I wonder about whether we have the right level of shareable units.

      I'll use Chef as an example of what I mean here, but one of the things I think about is Chef cookbooks. There is so much frustration with community cookbooks, essentially these libraries of deploying and configuring a specific piece of software and they are both too complex and not feature rich enough for folks. We debate over "best practices".

      It feels like part of the problem is we don't have the right level to look at from a unit that is shared. Or maybe we don't have enough standards in place to solve the problem within operating systems and software.

      Is it possible to share infrastructure as code across organizations? or even within an organization? Do we just not have the right level of shareable units?

      With tools like ARM templates and Cloudformation some of our IaC options change. I still don't want to write json or yaml for abstracting out infrastructure and those become even harder to share as a practice to learn from across companies.

    • I'll hazard a shot.

      My take is that Chef is not IaC at all. You can use it as such, but it's at root a provisioning tool. As is Puppet. Even Ansible - though it has evolved quite a multitude of uses as an IaC tool, originally it was not and at heart is still a very powerful provisioning system. SaltStack is about the same.

      My IaC journey effectively started with Terraform (because CloudFormation at that time made your eyes bleed out in about 15 minutes), and I'm still using it and am quite a fan of the whole Hashicorp stack (AKA Hashistack). I am aware that there are next generation IaC tools in the making, such as Pulumi, Ballerina or Metaparticle, but that's cutting edge and I'm eyeing them carefully (also some of this new generation of tools is too much invested in Kubernetes for my taste).

      I understand your frustration about json/yaml formats (even though e.g. CloudFormation is much more palatable today), but I think learning and using some kind of DSL for managing infrastructure programmatically is inescapable. In terms of sharing code, I think it is possible and have done it one way or another. In what I've seen done in practice, the key is a well thought-out version control CI[/CD] pipeline for IaC stuff, good communications including documentation practices, and as much self-service tooling as possible (e.g. look at Atlantis for a relatively simple Terraform-related example).

    • Interesting. Thank you. I hadn't heard of these next generation IaC tools yet :), but everything else I'm super familiar with. So what I'm curious about is, what is the right level of abstraction that is possible to share? I think CI/CD pipeline is great for individual projects that can be replicated within a company, but not so much when it comes to leveraging community. We end up with duplicate work, or super bloated artifacts right now (at least when it comes to Chef cookbooks for example). How do we as a whole community come together and start building out shareable patterns? Is it even possible? Is the best that we can hope for to share the problems we've solved?

    • Shared patterns at community level in the field that is barely 10 years old (if that) and moving at pretty supersonic speeds is a tall order. I don't think there's a universal answer or solution. Cloud at today's levels of ubiquity was born a few blinks ago. Serverless - about one. Containers, while around for a long long time, exploded and mixed their volcanic ash in everything and it's not like practitioners have converged on the right way to deal with them yet. Both the problems and the available means are mutating every day under our feet.

      Personally, I always try to fall back on generic engineering and management principles. Think systemic, document entity relationships, map solutions onto problems and vice versa, iteratively, learn from that, train your people to communicate effectively, in-team, cross-org, among the colleagues. Watch larger meta trends - IaC, immutable everything, observability, perhaps serverless as approach, etc. Always try to see your problems, old and new, in light of the current map of the whole territory as you see it, today.


      "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"

    • Found in my backlog a relevant link to a blog post on RFC-like process for engineering orgs - https://blog.pragmaticengineer.com/scaling-engineering-teams-via-writing-things-down-rfcs/