My team hosts several completely different projects on a production k8s cluster with a given configuration. We use AWS (and are starting to use Google Cloud). We have a QA cluster as well, that "sort of" mimics the production configuration, and has a few test projects deployed on it as well. I say "sort of" because right now we just have a collection of shell scripts and custom commands that an engineer manually ran to create the clusters. This had a bad consequence when we made a change to the configuration on the QA and production cluster, and it prevented deployment to that cluster until we reverted the change.
I am looking to implement Infrastructure as Code (IAC) and a corresponding CI/CD pipeline to prevent this. The goal is that a PR will trigger a build that will spin up a cluster by following the IAC steps, deploy some projects on it, run some tests, then spin down the cluster and provide a test report.
I am investigating Kops, Ansible, Chef, and Puppet as candidates for this task. Right now the shell scripts I mentioned before are mostly kops commands. I have been spending the last few days reading about the other three technologies as well, and I am not quite sure how they might fit into what I need.
Are there any recommendations for my particular use case? Implement IAC and a CI/CD pipeline that can spin up a new k8s cluster with the proposed changes and then spin it down once tests finish?
