/deploy/aws 下执行 terraform apply 时,
遇到这个错误 : Error: tiller was not found. polling deadline exceeded
本来以为是这个 原因 ,后来发现不是版本号问题。
而是 k8s 初始化有问题,似乎是 ec2 工作节点没启动:
no nodes available to schedule pods
➜ aws git:(develop) ✗ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-9b6bd4456-2qkjt 0/1 Pending 0 165m
kube-system coredns-9b6bd4456-pgk2n 0/1 Pending 0 165m
kube-system tiller-deploy-7f4678d5c5-g2kjs 0/1 Pending 0 21m
➜ aws git:(develop) ✗ kubectl describe pod tiller-deploy-7f4678d5c5-g2kjs -n kube-system|tail -n 4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 62s (x16 over 21m) default-scheduler no nodes available to schedule pods
➜ aws git:(develop) ✗ kubectl describe pod coredns-9b6bd4456-2qkjt -n kube-system|tail -n 4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4m15s (x111 over 165m) default-scheduler no nodes available to schedule pods
➜ aws git:(develop) ✗ kubectl describe pod coredns-9b6bd4456-pgk2n -n kube-system|tail -n 4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4m28s (x110 over 165m) default-scheduler no nodes available to schedule pods
➜ aws git:(develop) ✗ kubectl get nodes
No resources found.
请问应该怎么诊断?我需要还提供什么信息?
➜ aws git:(develop) ✗ kubectl config current-context
eks_my-tidb-cluster
➜ aws git:(develop) ✗ helm version
WARNING: "kubernetes-charts.storage.googleapis.com" is deprecated for "stable" and will be deleted Nov. 13, 2020.
WARNING: You should switch to "https://charts.helm.sh/stable"
Client: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}
Error: could not find a ready tiller pod
@dulao5 We have abandoned the terraform deployment in our official docs because the eks terraform module is not stable enough, could you please follow our doc here to deploy EKS with eksctl?
Just let us know if any issues with the procedure, thanks!
Now both GCP and AWS documentation doesn’t use terraform.
Do you plan to discard the existing terraform code? (eg. aws main.tfgcp main.tf )
Or are you planning to solve the terraform instability problem?
Could you advise on how to achieve “infrastructure as code” using aws eksctl without using terrafrom?
During the EKS upgrade, data in the local storage will be lost due to the node reconstruction. When the node reconstruction occurs, you need to migrate data in TiKV.
If I upgrade my EKS cluster, can I do so without stopping the service?
What does “migrate data in TiKV” mean, offline a tikv node in the same TiKV cluster and migrate it to a new node , is it?
Now both GCP and AWS documentation doesn’t use terraform.
Do you plan to discard the existing terraform code? (eg. aws main.tfgcp main.tf )
Or are you planning to solve the terraform instability problem?
We do not maintain the terraform scripts anymore, but we leave them in the repo in case some community members may want to use or improve them.
Could you advise on how to achieve “infrastructure as code” using aws eksctl without using terrafrom?
Actually, we would like to focus on the functionalities of TiDB Operator. The setup of Kubernetes is not our focus.
During the EKS upgrade, data in the local storage will be lost due to the node reconstruction. When the node reconstruction occurs, you need to migrate data in TiKV.
If I upgrade my EKS cluster, can I do so without stopping the service?What does “migrate data in TiKV” mean, offline a tikv node in the same TiKV cluster and migrate it to a new node , is it?
Local storage is not recommended, if you upgrade your EKS cluster nodes, you will lose all your data with instance stores, so please use EBS and the TiKV team is also working on the performance optimization with EBS.
Thank you very much.
I have understood the first two questions.
Regarding question 3, can I know further details about “on the performance optimization with EBS”?
By the way,
According to a part of the code, it seems that the eks module uses low-level API such as aws_autoscaling_group and aws_launch_configuration, etc (eg).
If you use aws_eks_cluster or aws_eks_node_group or other high-level API, it may be stable.
@Hacker_sAKN3wef
I was able to build my cluster, but the EXTERNAL-IP of tidb-base svc is <pending>.
How should I check it?
kubectl get svc basic-tidb -n tidb-cluster
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
basic-tidb LoadBalancer 172.20.97.137 <pending> 4000:32069/TCP,10080:32646/TCP 3h23m
@Hacker_sAKN3wef
Hi, I have a few questions.
I noticed that you recommend using EBS as TiKV storage on EKS.
In performance, What’s the difference between using EBS and local disk for TiKV.