I was testing some commands and I ran
$ kubectl delete nodes --alland it deletes de-registers all the nodes including the masters. Now I can't connect to the cluster (Well, Obviously as the master is deleted).
Is there a way to prevent this as anyone could accidentally do this?
Extra Info: I am using KOps for deployment.
P.S. It does not delete the EC2 instances and the nodes come up on doing a EC2 instance reboot on all the instances.
By default, you using something like a superuser who can do anything he want with a cluster.
For limit access to a cluster for other users you can use RBAC authorization for. By RBAC rules you can manage access and limits per resource and action.
In few words, for do that you need to:
Create new cluster by Kops with --authorization RBAC or modify existing one by adding 'rbac' option to cluster's configuration to 'authorization' section:
authorization: rbac: {}
Now, we can follow that instruction from Bitnami for create a user. For example, let's creating a user which has access only to office namespace and only for a few actions. So, we need to create a namespace firs:
kubectl create namespace office
Create a key and certificates for new user:
openssl genrsa -out employee.key 2048openssl req -new -key employee.key -out employee.csr -subj "/CN=employee/O=bitnami"
Now, using your CA authority key (It available in the S3 bucket under PKI) we need to approve new certificate:
openssl x509 -req -in employee.csr -CA CA_LOCATION/ca.crt -CAkey CA_LOCATION/ca.key -CAcreateserial -out employee.crt -days 500
Creating credentials:
kubectl config set-credentials employee --client-certificate=/home/employee/.certs/employee.crt --client-key=/home/employee/.certs/employee.key
Setting a right context:
kubectl config set-context employee-context --cluster=YOUR_CLUSTER_NAME --namespace=office --user=employee
New we have a user without access to anything. Let's create a new role with limited access, here is example of Role which will have access only to deployments, replicasets and pods for create, delete and modify them and nothing more. Create file role-deployment-manager.yaml with Role configuration:
kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: namespace: office name: deployment-manager rules: - apiGroups: ["", "extensions", "apps"] resources: ["deployments", "replicasets", "pods"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
rolebinding-deployment-manager.yaml with Rolebinding, which will attach your Role to user:kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: deployment-manager-binding namespace: office subjects: - kind: User name: employee apiGroup: "" roleRef: kind: Role name: deployment-manager apiGroup: ""
kubectl create -f role-deployment-manager.yaml kubectl create -f rolebinding-deployment-manager.yaml
So, now you have a user with limited access and he cannot destroy your cluster.
Anton Kostenko describes a good way of preventing what you've described. Below I give details of how you can ensure the apiserver remains accessible even if someone does accidentally delete all the node objects:
Losing connectivity to the apiserver by deleting node objects will only happen if the components necessary for connecting to the apiserver (e.g. the apisever itself and etcd) are managed by a component (i.e. the kubelet) that depends on the apiserver being up (GKE for example can scale down to 0 worker nodes, leaving no node objects, but the apiserver will still be accessible).
As a specific example, my personal cluster has a single master node with all the control plane components described as static Pod manifests and placed in the directory referred to by the --pod-manifest-path flag on the kubelet on that master node. Deleting all the node objects as you did in the question caused all my workloads to go into a pending state but the apiserver was still accessible in this case because the control plane components are run regardless of whether the kubelet can access the apiserver.
Common ways to prevent what you've just described is to run the apiserver and etcd as static manifests managed by the kubelet as I just described or to run them independently of any kubelet, perhaps as systemd units.