Two major updates recently
Firstly, as suspected, I finally got the notification saying that Google Domains' registrations would be acquired by SquareSpace so all my registrations would be transferred over them should I not do anything. Obviously, I didn't want that, so I transferred them over to AWS (Route 53) so now my domains are registered with AWS, but are DNS managed by Cloud DNS. I had some weirdness when trying to migrate all my domains in bulk, with respect to the auth code not being accepted on one of the domains when doing the bulk migration, but it was accepted when I did the migration on that one domain alone, so... go figure.
Next update is Terraform. In case you didn't know, Hashicorp has changed the Terraform license and essentially made it no longer open source. This behaviour is similar to what Red Hat did with its RHEL offering and the backlash is just as bad.
Immediately I knew someone would fork it, and already, there's the OpenTF Initiative and this is the key part:
Our request to HashiCorp: switch Terraform back to an open source license.
We ask HashiCorp to do the right thing by the community: instead of going forward with the BUSL license change, switch Terraform back to a truly open source license, and commit to keeping it that way forever going forward. That way, instead of fracturing the community, we end up with a single, impartial, reliable home for Terraform where the whole community can unite to keep building this amazing ecosystem.
Our fallback plan: fork Terraform into a foundation.
If HashiCorp is unwilling to switch Terraform back to an open source license, we propose to fork the legacy MPL-licensed Terraform and maintain the fork in the foundation. This is similar to how Linux and Kubernetes are managed by foundations (the Linux Foundation and the Cloud Native Computing Foundation, respectively), which are run by multiple companies, ensuring the tool stays truly open source and neutral, and not at the whim of any one company.
OpenTF Initiative (https://opentf.org/)
Essentially, make Terraform open source again, or a fork from the MPL version will be made and maintained separately from Hashicorp's version. This will essentially lead to two, potentially diverging versions of Terraform, one BUSL and one MPL licensed
I'm already looking at alternatives and the two currently that I'm looking at are Ansible and Pulumi
Ansible I've had experience in , but there's two main issues with it:
- It's underused compared to Terraform and the providers are woefully undersupported and undermaintained
- It's Red Hat
Pulumi I've heard lots of good things about, but its a new technology, and I don't know if it is can "Import" existing infrastructure.
Guess a "spike" is worth doing for it.
It's been quite a long time since I did any updates on this blog so a couple of updates are in order
House
I've now been living in the new place for just over a year. Generally everything is good, we've started putting in a lawn and currently letting it get its roots in before we try to cut it
Twitter
Oh, boy, what an absolute train wreck. I've tolerated Elon's presence at Twitter because most of the stuff he did wasn't far off what Jack was doing prior. But cutting off all third party clients, forcing everyone onto the new TweetDeck (which likely will be paywalled too) has literally driven users away. Twitter has been losing users and revenue constantly, and it is totally unsurprising.
I've disabled my two TweetDeck profiles (a professional one, and a casual one) from Ferdium. But I doubt I will be going back any time soon.
Red Hat
Another train wreck of a situation. Red Hat's decision to first kill CentOS's stability, then for RHEL sources to only be accessible behind a subscription has pissed of a lot of users, even if it's not strictly speaking against license.
The only one left in its sights will be Fedora so that leads me onto the next update
Manjaro & Archlinux
I've been tinkering with Manjaro more and more lately, with its rolling release schedule meaning I never need to upgrade from a major version to another major version.
It's downside I'm finding is that some packages, especially those in the AUR are essentially "compile from source" packages which does the build on your machine during the install. This can take a varying amount of time depending on the code. With my RSS reader of choice: QuiteRSS, this build takes a mind numbing 2.5 hours to do, even on a high spec machine.
That's where I found out about setting up your own Arch repo. I've been tinkering with that, setting it up on GCP and fronted by a CDN. This works pretty well, but I still need to find how I can do scheduled builds to keep that up to date, but it looks like I'll be switching to Manjaro at some point in the near future. Sound works fine, using lyncolnmd's work.
Another downside with Manjaro, however is that its btrfs filesystem, my home directory backup, and CloneZilla don't seem to want to work well together
Wordpress & Domains
One final update, I will likely be transferring out the blenderfox.com domain out of Wordpress, while I had this registered as part of the blog, I'm finding it much harder to maintain this domain using Wordpress's very limited DNS management tools. I will likely transfer it out to Google Domains, even though there's talk of them shuttering that service. Secondary service would be AWS.
Kubernetes uses Service Accounts to control who can access what within the cluster, but once a request leaves the cluster, it will use a default account. Normally this is the default Google Compute Engine account in GKE, and this has extremely high level access and could result in a lot of damage if your cluster is compromised.
In this article, I will be setting up a GKE cluster using a minimal access service account and enabling Workflow Identity.
(This post is now also available on Medium)
Workflow Identity will enable you to bind a Kubernetes service account to a service account in GCP. You can then control GCP permissions of that account from within GCP -- no RBAC/ABAC messing about needed (although you will still need to mess with RBAC/ABAC if you want to restrict that service account within Kubernetes, but that's a separate article.)
What you will need for this tutorial:
- A Google account
- A Google Cloud account
-
Terraform on your local machine
-
kubectl on your local machine (can be installed as part of the Google Cloud SDK)
-
Google Cloud SDK on your local machine
- A Google Cloud project setup
- A service account with "Owner" permissions in your GCP project (the default compute engine account will normally work)
- A credentials json file from that account -- this can be generated using
gcloud iam service-accounts keys create credentials.json --iam-account={iam-account-email}
We will start by setting up our Terraform provider
variable "project" {
default = "REPLACE_ME"
}
variable "region" {
default = "europe-west2"
}
variable "zone" {
default = "europe-west2-a"
}
provider "google" {
project = var.project
region = var.region
zone = var.zone
credentials = file("credentials.json")
}
We define three variables here that we can reuse later -- the project, region and zone. These variables you can adjust to match your own setup.
The provider block (provider "google" {..}) references those variables and also refers to the credentials.json file that will be used to create the resources in your account.
Next we create the service account that we will bind to the cluster. This service account should contain minimal permissions as it will be the default account used by requests leaving the cluster. Only give it what is essential. You will notice I do not bind it to any roles.
resource "google_service_account" "cluster-serviceaccount" {
account_id = "cluster-serviceaccount"
display_name = "Service Account For Terraform To Make GKE Cluster"
}
Now let's define our cluster and node pool. This block can vary wildly on your circumstances, but I'll use a Kubernetes 1.16 single-zone cluster, with a e2-medium node size and have autoscaling enabled
variable "cluster_version" {
default = "1.16"
}
resource "google_container_cluster" "cluster" {
name = "tutorial"
location = var.zone
min_master_version = var.cluster_version
project = var.project
lifecycle {
ignore_changes = [
# Ignore changes to min-master-version as that gets changed
# after deployment to minimum precise version Google has
min_master_version,
]
}
# We can't create a cluster with no node pool defined, but
# we want to only use separately managed node pools. So we
# create the smallest possible default node pool and
# immediately delete it.
remove_default_node_pool = true
initial_node_count = 1
workload_identity_config {
identity_namespace = "${var.project}.svc.id.goog"
}
}
resource "google_container_node_pool" "primary_preemptible_nodes" {
name = "tutorial-cluster-node-pool"
location = var.zone
project = var.project
cluster = google_container_cluster.cluster.name
node_count = 1
autoscaling {
min_node_count = 1
max_node_count = 5
}
version = var.cluster_version
node_config {
preemptible = true
machine_type = "e2-medium"
# Google recommends custom service accounts that have cloud-platform scope and
# permissions granted via IAM Roles.
service_account = google_service_account.cluster-serviceaccount.email
oauth_scopes = [
"https://www.googleapis.com/auth/cloud-platform"
]
metadata = {
disable-legacy-endpoints = "true"
}
}
lifecycle {
ignore_changes = [
# Ignore changes to node_count, initial_node_count and version
# otherwise node pool will be recreated if there is drift between what
# terraform expects and what it sees
initial_node_count,
node_count,
version
]
}
}
Let's go through a few things on the above block:
variable "cluster_version" {
default = "1.16"
}
Defines a variable we will use to describe the version of Kubernetes we want on the master and worker nodes.
resource "google_container_cluster" "cluster" {
...
min_master_version = var.cluster_version
...
lifecycle {
ignore_changes = [
min_master_version,
]
}
...
}
The ignore_changes block here tells terraform not to pay attention to changes in the min_master_version field. This is because even though we declare we wanted 1.16 as the version, GKE will put a Kubernetes variant of 1.16 onto the cluster. For example, the cluster might be created with version 1.16.9-gke.999 -- which is different to what Terraform expects, so if you were to run Terraform again, it would attempt to change the cluster version from 1.16.9-gke.999 to 1.16, cycling through the nodes again.
Next block to discuss:
resource "google_container_cluster" "cluster" {
...
remove_default_node_pool = true
initial_node_count = 1
...
}
A GKE cluster must be created with a node pool. However it is easier to manage node pool separately, so this block tells Terraform to delete the default node pool when the cluster is created.
Final part of this block:
resource "google_container_cluster" "cluster" {
...
workload_identity_config {
identity_namespace = "${var.project}.svc.id.goog"
}
}
This enables Workload Identity and the namespace must be of the format {project}.svc.id.goog
Now let's move onto the Node Pool definition:
resource "google_container_node_pool" "primary_preemptible_nodes" {
name = "tutorial-cluster-node-pool"
location = var.zone
project = var.project
cluster = google_container_cluster.cluster.name
node_count = 1
autoscaling {
min_node_count = 1
max_node_count = 5
}
version = var.cluster_version
node_config {
preemptible = true
machine_type = "e2-medium"
# Google recommends custom service accounts that have cloud-platform scope and
# permissions granted via IAM Roles.
service_account = google_service_account.cluster-serviceaccount.email
oauth_scopes = [
"https://www.googleapis.com/auth/cloud-platform"
]
metadata = {
disable-legacy-endpoints = "true"
}
}
lifecycle {
ignore_changes = [
# Ignore changes to node_count, initial_node_count and version
# otherwise node pool will be recreated if there is drift between what
# terraform expects and what it sees
initial_node_count,
node_count,
version
]
}
}
Let's go over a couple of blocks again:
resource "google_container_node_pool" "primary_preemptible_nodes" {
...
node_count = 1
autoscaling {
min_node_count = 1
max_node_count = 5
}
...
}
This sets up autoscaling with a starting node count of 1 and max node count of 5. Unlike with EKS, you don't need deploy the autoscaler into the cluster. Enabling this will natively allow Kubernetes to scale nodes up or down. The downside is you don't see as many messages compared to the deployed version, so it's sometimes harder to debug why a pod isn't triggering a scaleup.
resource "google_container_node_pool" "primary_preemptible_nodes" {
...
node_config {
preemptible = true
machine_type = "e2-medium"
# Google recommends custom service accounts that have cloud-platform scope and
# permissions granted via IAM Roles.
service_account = google_service_account.cluster-serviceaccount.email
oauth_scopes = [
"https://www.googleapis.com/auth/cloud-platform"
]
metadata = {
disable-legacy-endpoints = "true"
}
}
...
}
Here we define the node config, we've got this set as a pool of pre-emptible nodes, of type e2-medium. We tie the nodes to the service account defined earlier and give it only the cloud-platform scope.
The metadata block is needed as if you don't specify it, the value disable-legacy-endpoints = "true" is assumed to be applied, and will cause the node pool to be respun each time you run terraform, as it thinks it need to apply the updated config to the pool.
resource "google_container_node_pool" "primary_preemptible_nodes" {
...
lifecycle {
ignore_changes = [
# Ignore changes to node_count, initial_node_count and version
# otherwise node pool will be recreated if there is drift between what
# terraform expects and what it sees
initial_node_count,
node_count,
version
]
}
}
Similar to the version field on the master node, we tell Terraform to ignore some fields if they have changed.
version we ignore for the same reason as on the master node -- the version deployed will be slightly different to the one we declared.
initial_node_count we ignore because if the node pool has scaled up, not ignoring this will cause terraform to attempt to scale the nodes back down to the initial_node_count value, causing pods to be sent into Pending
node_count we ignore for pretty much the same reason -- it will likely never be the initial value on a production system due to scale up.
With the basic skeleton setup, we can run Terraform to setup the stack. Yes we haven't actually bound anything to serviceaccounts, but that will come later.
Let's Terraform the infrastructure:
terraform init
terraform plan -out tfplan
terraform apply tfplan
Creation of the cluster can take between 5-15 minutes
Next, we need to get credentials and link into the cluster
gcloud beta container clusters get-credentials tutorial --zone {cluster-zone} --project {project}
or
gcloud beta container clusters get-credentials tutorial --region {cluster-region} --project {project}
You should get some output like this:
Fetching cluster endpoint and auth data.
kubeconfig entry generated for tutorial.
Now you should be able to run kubectl get pods --all-namespaces to see what's in your cluster (should be nothing other than the default system pods)
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system event-exporter-gke-666b7ffbf7-lw79x 2/2 Running 0 13m
kube-system fluentd-gke-scaler-54796dcbf7-6xnsg 1/1 Running 0 13m
kube-system fluentd-gke-skmsq 2/2 Running 0 4m23s
kube-system gke-metadata-server-fsxj6 1/1 Running 0 9m29s
kube-system gke-metrics-agent-pfdbp 1/1 Running 0 9m29s
kube-system kube-dns-66d6b7c877-wk2nt 4/4 Running 0 13m
kube-system kube-dns-autoscaler-645f7d66cf-spz4c 1/1 Running 0 13m
kube-system kube-proxy-gke-tutorial-tutorial-cluster-node-po-b531f1ee-8kpj 1/1 Running 0 9m29s
kube-system l7-default-backend-678889f899-q6gsl 1/1 Running 0 13m
kube-system metrics-server-v0.3.6-64655c969-2lz6v 2/2 Running 3 13m
kube-system netd-7xttc 1/1 Running 0 9m29s
kube-system prometheus-to-sd-w9cwr 1/1 Running 0 9m29s
kube-system stackdriver-metadata-agent-cluster-level-566c4b7cf9-7wmhr 2/2 Running 0 4m23s
Now let's do our first test. We'll use gsutil to run a list of GS buckets on our project.
kubectl run --rm -it test --image gcr.io/cloud-builders/gsutil ls
This will run a docker image with gsutil in it and then remove the container when the command finishes.
The output should be something like this:
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
AccessDeniedException: 403 Caller does not have storage.buckets.list access to the Google Cloud project.
Session ended, resume using 'kubectl attach test-68bb69b777-5nzgt -c test -i -t' command when the pod is running
deployment.apps "test" deleted
As you can see, we get a 403. The default service account doesn't have permissions to access Google Storage.
Now let's setup the service account we will use for binding:
resource "google_service_account" "workload-identity-user-sa" {
account_id = "workload-identity-tutorial"
display_name = "Service Account For Workload Identity"
}
resource "google_project_iam_member" "storage-role" {
role = "roles/storage.admin"
# role = "roles/storage.objectAdmin"
member = "serviceAccount:${google_service_account.workload-identity-user-sa.email}"
}
resource "google_project_iam_member" "workload_identity-role" {
role = "roles/iam.workloadIdentityUser"
member = "serviceAccount:${var.project}.svc.id.goog[workload-identity-test/workload-identity-user]"
}
Again, let's go through the blocks:
resource "google_service_account" "workload-identity-user-sa" {
account_id = "workload-identity-tutorial"
display_name = "Service Account For Workload Identity"
}
This block defines the service account in GCP that will be binding to.
resource "google_project_iam_member" "storage-role" {
role = "roles/storage.admin"
# role = "roles/storage.objectAdmin"
member = "serviceAccount:${google_service_account.workload-identity-user-sa.email}"
}
This block assigns the Storage Admin role to the service account we just created -- essentially it is putting the service account in the Storage Admin group. Think of it more like adding the account to a group rather than assigning a permission or role to the account.
resource "google_project_iam_member" "workload_identity-role" {
role = "roles/iam.workloadIdentityUser"
member = "serviceAccount:${var.project}.svc.id.goog[workload-identity-test/workload-identity-user]"
}
This block adds the service account as a Workload Identity User. You'll notice that the member field is a bit confusing. The ${var.project}.svc.id.goog bit indicates that it is a Workflow Identity namespace and the bit in [...] is the name of the Kubernetes service account we want to allow to be bound to this. This membership and an annotation on the service account (described below) will allow the service account in Kubernetes to essentially impersonate the service account in GCP and you will see this in the example.
With the service account setup in Terraform, let's run the Terraform apply steps again
terraform plan -out tfplan
terraform apply tfplan
Assuming it didn't error, we now have one half of the binding -- the GCP service account. We now need to create the service account inside Kubernetes.
You'll recall that we had a piece of data in the [...]: workload-identity-test/workload-identity-user this is our service account that we need to create. Below is the yaml for creating the namespace and the service account. Save this into the file workload-identity-user.yaml:
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: null
name: workload-identity-test
spec: {}
status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
iam.gke.io/gcp-service-account: workload-identity-tutorial@{project}.iam.gserviceaccount.com
name: workload-identity-user
namespace: workload-identity-test
The important thing to note is the annotation on the service account:
annotations:
iam.gke.io/gcp-service-account: workload-identity-tutorial@{project}.iam.gserviceaccount.com
The annotation references the service account created by the Terraform block:
resource "google_service_account" "workload-identity-user-sa" {
account_id = "workload-identity-tutorial"
display_name = "Service Account For Workload Identity"
}
So the Kubernetes service account references the GCP service account and the GCP service references the Kubernetes service account.
Important Note: If you do not do the double referencing -- for example, if you forget to include the annotation on the service account or forget to put the referenced Kubernetes service account in the Workload Identity member block, then GKE will use the default service account specified on the node.
Now it's time to put it to the test. If everything is setup correct, run the previous test again:
kubectl run --rm -it test --image gcr.io/cloud-builders/gsutil ls
You should still get the a 403 but with a different error message.
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
AccessDeniedException: 403 Primary: /namespaces/{project}.svc.id.goog with additional claims does not have storage.buckets.list access to the Google Cloud project.
Session ended, resume using 'kubectl attach test-68bb69b777-8ltvc -c test -i -t' command when the pod is running
deployment.apps "test" deleted
Let's now create the service accounts. This file should have been created by the earlier step:
$ kubectl apply -f workload-identity-test.yaml
namespace/workload-identity-test created
serviceaccount/workload-identity-user created
So now let's run the test again but this time, we specify the service account and also the namespace as a service account is tied to the namespace it resides in — in this case, the namespace of our service account is workload-identity-test
kubectl run -n workload-identity-test --rm --serviceaccount=workload-identity-user -it test --image gcr.io/cloud-builders/gsutil ls
The output will show the buckets you have:
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
gs://backups/
gs://snapshots/
Session ended, resume using 'kubectl attach test-66754998f-sp79b -c test -i -t' command when the pod is running
deployment.apps "test" deleted
NOTE: If you're running a later version of Kubernetes or kubectl, you may get the following error:
Flag --serviceaccount has been deprecated, has no effect and will be removed in 1.24.
In that case, you need to instead use the --overrides switch:
kubectl run -it --rm -n workload-identity-test test --overrides='{ "apiVersion": "v1", "spec": { "serviceAccount": "workload-identity-test" } }' --image gcr.io/cloud-builders/gsutil ls
Let's now change the permissions on the GCP service account to prove it's the one being used change this block:
resource "google_project_iam_member" "storage-role" {
role = "roles/storage.admin"
# role = "roles/storage.objectAdmin"
member = "serviceAccount:${google_service_account.workload-identity-user-sa.email}"
}
And change the active role like so:
resource "google_project_iam_member" "storage-role" {
# role = "roles/storage.admin" ## <-- comment this out
role = "roles/storage.objectAdmin" ## <-- uncomment this
member = "serviceAccount:${google_service_account.workload-identity-user-sa.email}"
}
Run the terraform actions again:
terraform plan -out tfplan
terraform apply tfplan
Allow a few minutes for the change to propagate then run the test again:
kubectl run -n workload-identity-test --rm --serviceaccount=workload-identity-user -it test --image gcr.io/cloud-builders/gsutil ls
(See earlier if you get an error regarding the serviceaccount switch)
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
AccessDeniedException: 403 workload-identity-tutorial@{project}.iam.gserviceaccount.com does not have storage.buckets.list access to the Google Cloud project.
Session ended, resume using 'kubectl attach test-66754998f-k5dm5 -c test -i -t' command when the pod is running
deployment.apps "test" deleted
And there you have it, the service account in the cluster: workload-identity-test/workload-identity-user is bound to the service account workload-identity-tutorial@{project}.iam.gserviceaccount.com on GCP, carrying the permissions it also has.
If the service account on Kubernetes is compromised in some way, you just need to revoke the permissions on the GCP service account and the Kubernetes service account no longer has any permissions to do anything in GCP.
For simplicity, here's the Terraform used for this tutorial. Replace what you need -- you can move things around and separate into other Terraform files if you wish -- I kept it in one file for simplicity.
variable "project" {
default = "REPLACE_ME"
}
variable "region" {
default = "europe-west2"
}
variable "zone" {
default = "europe-west2-a"
}
provider "google" {
project = var.project
region = var.region
zone = var.zone
credentials = file("credentials.json")
}
resource "google_service_account" "cluster-serviceaccount" {
account_id = "cluster-serviceaccount"
display_name = "Service Account For Terraform To Make GKE Cluster"
}
variable "cluster_version" {
default = "1.16"
}
resource "google_container_cluster" "cluster" {
name = "tutorial"
location = var.zone
min_master_version = var.cluster_version
project = var.project
lifecycle {
ignore_changes = [
# Ignore changes to min-master-version as that gets changed
# after deployment to minimum precise version Google has
min_master_version,
]
}
# We can't create a cluster with no node pool defined, but we want to only use
# separately managed node pools. So we create the smallest possible default
# node pool and immediately delete it.
remove_default_node_pool = true
initial_node_count = 1
workload_identity_config {
identity_namespace = "${var.project}.svc.id.goog"
}
}
resource "google_container_node_pool" "primary_preemptible_nodes" {
name = "tutorial-cluster-node-pool"
location = var.zone
project = var.project
cluster = google_container_cluster.cluster.name
node_count = 1
autoscaling {
min_node_count = 1
max_node_count = 5
}
version = var.cluster_version
node_config {
preemptible = true
machine_type = "e2-medium"
# Google recommends custom service accounts that have cloud-platform scope
# and permissions granted via IAM Roles.
service_account = google_service_account.cluster-serviceaccount.email
oauth_scopes = [
"https://www.googleapis.com/auth/cloud-platform"
]
metadata = {
disable-legacy-endpoints = "true"
}
}
lifecycle {
ignore_changes = [
# Ignore changes to node_count, initial_node_count and version
# otherwise node pool will be recreated if there is drift between what
# terraform expects and what it sees
initial_node_count,
node_count,
version
]
}
}
resource "google_service_account" "workload-identity-user-sa" {
account_id = "workload-identity-tutorial"
display_name = "Service Account For Workload Identity"
}
resource "google_project_iam_member" "storage-role" {
role = "roles/storage.admin"
# role = "roles/storage.objectAdmin"
member = "serviceAccount:${google_service_account.workload-identity-user-sa.email}"
}
resource "google_project_iam_member" "workload_identity-role" {
role = "roles/iam.workloadIdentityUser"
member = "serviceAccount:${var.project}.svc.id.goog[workload-identity-test/workload-identity-user]"
}
