fluend-elasticsearch日志遇到的问题
来源:互联网 发布:西安软件新城附近楼盘 编辑:程序博客网 时间:2024/05/21 10:50
1.如何删除不一致状态下的rc,deployment,service.
在某些情况下,经常发现kubectl进程挂起现象,然后在get时候发现删了一半,而另外的删除不了
[root@k8s-master ~]# kubectl get -f fluentd-elasticsearch/
NAME DESIRED CURRENT READY AGE
rc/elasticsearch-logging-v1 0 2 2 15h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/kibana-logging 0 1 1 1 15h
Error from server (NotFound): services "elasticsearch-logging" not found
Error from server (NotFound): daemonsets.extensions "fluentd-es-v1.22" not found
Error from server (NotFound): services "kibana-logging" not found
删除这些deployment,service或者rc命令如下:
kubectl delete deployment kibana-logging -n kube-system --cascade=false
kubectl delete deployment kibana-logging -n kube-system --ignore-not-found
delete rc elasticsearch-logging-v1 -n kube-system --force now --grace-period=0
2.删除不了后如何重置etcd
rm -rf /var/lib/etcd/*删除后重新reboot master结点.
reset etcd后需要重新设置网络
etcdctl mk /atomic.io/network/config '{ "Network": "192.168.0.0/16" }'
3.启动apiserver失败
每次启动都是报
start request repeated too quickly for kube-apiserver.service
但其实不是启动频率问题,需要查看,/var/log/messages,在我的情况中是因为开启ServiceAccount后找不到ca.crt等文件,导致启动出错
May 21 07:56:41 k8s-master kube-apiserver: Flag --port has been deprecated, see --insecure-port instead.
May 21 07:56:41 k8s-master kube-apiserver: F0521 07:56:41.692480 4299 universal_validation.go:104] Validate server run options failed: unable to load client CA file: open /var/run/kubernetes/ca.crt: no such file or directory
May 21 07:56:41 k8s-master systemd: kube-apiserver.service: main process exited, code=exited, status=255/n/a
May 21 07:56:41 k8s-master systemd: Failed to start Kubernetes API Server.
May 21 07:56:41 k8s-master systemd: Unit kube-apiserver.service entered failed state.
May 21 07:56:41 k8s-master systemd: kube-apiserver.service failed.
May 21 07:56:41 k8s-master systemd: kube-apiserver.service holdoff time over, scheduling restart.
May 21 07:56:41 k8s-master systemd: start request repeated too quickly for kube-apiserver.service
May 21 07:56:41 k8s-master systemd: Failed to start Kubernetes API Server.
在部署fluentd等日志组件的时候,很多问题都是因为需要开启ServiceAccount选项需要配置安全导致,所以说到底还是需要配置好ServiceAccount.
4.出现Permission denied情况
在配置fluentd时候出现cannot create /var/log/fluentd.log: Permission denied错误,这是因为没有关掉SElinux安全导致.
可以在/etc/selinux/config中将SELINUX=enforcing设置成disabled,然后reboot
5.基于ServiceAccount的配置
首先生成各种需要的keys,k8s-master需替换成master的主机名.
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=k8s-master" -days 10000 -out ca.crt
openssl genrsa -out server.key 2048
echo subjectAltName=IP:10.254.0.1 > extfile.cnf
#ip由下述命令决定
#kubectl get services --all-namespaces |grep 'default'|grep 'kubernetes'|grep '443'|awk '{print $3}'
openssl req -new -key server.key -subj "/CN=k8s-master" -out server.csr
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -extfile extfile.cnf -out server.crt -days 10000
如果修改/etc/kubernetes/apiserver的配置文件参数的话,通过systemctl start kube-apiserver启动失败,出错信息为:
Validate server run options failed: unable to load client CA file: open /root/keys/ca.crt: permission denied
但可以通过命令行启动API Server
/usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://k8s-master:2379 --address=0.0.0.0 --port=8080 --kubelet-port=10250 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount --insecure-bind-address=0.0.0.0 --client-ca-file=/root/keys/ca.crt --tls-cert-file=/root/keys/server.crt --tls-private-key-file=/root/keys/server.key --basic-auth-file=/root/keys/basic_auth.csv --secure-port=443
&>> /var/log/kubernetes/kube-apiserver.log &
命令行启动Controller-manager
/usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://k8s-master:8080 --root-ca-file=/root/keys/ca.crt --service-account-private-key-file=/root/keys/server.key
& >>/var/log/kubernetes/kube-controller-manage.log
6.ETCD启动不起来
etcd是kubernetes集群的zookeeper进程,几乎所有的service都依赖于etcd的启动,比如flanneld,apiserver,docker.....
在启动etcd是报错日志如下
May 24 13:39:09 k8s-master systemd: Stopped Flanneld overlay address etcd agent.May 24 13:39:28 k8s-master systemd: Starting Etcd Server...May 24 13:39:28 k8s-master etcd: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379,http://etcd:4001May 24 13:39:28 k8s-master etcd: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag May 24 13:39:28 k8s-master etcd: recognized environment variable ETCD_DATA_DIR, but unused: shadowed by corresponding flag May 24 13:39:28 k8s-master etcd: recognized environment variable ETCD_LISTEN_CLIENT_URLS, but unused: shadowed by corresponding flag May 24 13:39:28 k8s-master etcd: etcd Version: 3.1.3May 24 13:39:28 k8s-master etcd: Git SHA: 21fdcc6May 24 13:39:28 k8s-master etcd: Go Version: go1.7.4May 24 13:39:28 k8s-master etcd: Go OS/Arch: linux/amd64May 24 13:39:28 k8s-master etcd: setting maximum number of CPUs to 1, total number of available CPUs is 1May 24 13:39:28 k8s-master etcd: the server is already initialized as member before, starting as etcd member...May 24 13:39:28 k8s-master etcd: listening for peers on http://localhost:2380May 24 13:39:28 k8s-master etcd: listening for client requests on 0.0.0.0:2379May 24 13:39:28 k8s-master etcd: listening for client requests on 0.0.0.0:4001May 24 13:39:28 k8s-master etcd: recovered store from snapshot at index 140014May 24 13:39:28 k8s-master etcd: name = masterMay 24 13:39:28 k8s-master etcd: data dir = /var/lib/etcd/default.etcdMay 24 13:39:28 k8s-master etcd: member dir = /var/lib/etcd/default.etcd/memberMay 24 13:39:28 k8s-master etcd: heartbeat = 100msMay 24 13:39:28 k8s-master etcd: election = 1000msMay 24 13:39:28 k8s-master etcd: snapshot count = 10000May 24 13:39:28 k8s-master etcd: advertise client URLs = http://etcd:2379,http://etcd:4001May 24 13:39:28 k8s-master etcd: ignored file 0000000000000001-0000000000012700.wal.broken in walMay 24 13:39:29 k8s-master etcd: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 148905May 24 13:39:29 k8s-master etcd: 8e9e05c52164694d became follower at term 12May 24 13:39:29 k8s-master etcd: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 12, commit: 148905, applied: 140014, lastindex: 148905, lastterm: 12]May 24 13:39:29 k8s-master etcd: enabled capabilities for version 3.1May 24 13:39:29 k8s-master etcd: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from storeMay 24 13:39:29 k8s-master etcd: set the cluster version to 3.1 from storeMay 24 13:39:29 k8s-master etcd: starting server... [version: 3.1.3, cluster version: 3.1]May 24 13:39:29 k8s-master etcd: raft save state and entries error: open /var/lib/etcd/default.etcd/member/wal/0.tmp: is a directoryMay 24 13:39:29 k8s-master systemd: etcd.service: main process exited, code=exited, status=1/FAILUREMay 24 13:39:29 k8s-master systemd: Failed to start Etcd Server.May 24 13:39:29 k8s-master systemd: Unit etcd.service entered failed state.May 24 13:39:29 k8s-master systemd: etcd.service failed.May 24 13:39:29 k8s-master systemd: etcd.service holdoff time over, scheduling restart.
核心语句
raft save state and entries error: open /var/lib/etcd/default.etcd/member/wal/0.tmp: is a directory
进入相关目录,删除0.tmp,然后就可以启动啦!
7.CentOS下配置主机互信
- 在每台服务器需要建立主机互信的用户名执行以下命令生成公钥/密钥,默认回车即可
ssh-keygen -t rsa
可以看到生成个公钥的文件
- 互传公钥,第一次需要输入密码,之后就OK了
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.199.132 (-p 2222)
-p 端口 默认端口不加-p,如果更改过端口,就得加上-p
可以看到是在.ssh/下生成了个authorized_keys的文件,记录了能登陆这台服务器的其他服务器的公钥
- 测试看是否能登陆
ssh 192.168.199.132 (-p 2222)
8.CentOS主机名的修改
hostnamectl set-hostname k8s-master1
9.Virtualbox实现CentOS复制和粘贴功能
如果不安装或者不输出,可以将update修改成install再运行
yum install updateyum update kernelyum update kernel-develyum install kernel-headersyum install gccyum install gcc make
运行完后sh VBoxLinuxAdditions.run
- fluend-elasticsearch日志遇到的问题
- elasticsearch遇到的问题
- elasticsearch遇到的一点问题
- ElasticSearch使用过程中遇到的问题
- 远程连接elasticsearch遇到的问题
- 生产环境使用elasticsearch遇到的一些问题以及解决方法
- GitHub使用elasticsearch遇到的一些问题及解决方法
- 使用elasticsearch遇到的一些问题以及解决方法
- elasticsearch中遇到的一些问题以及解决方式
- 使用elasticsearch遇到的一些问题以及解决方法
- 使用elasticsearch遇到的一些问题以及解决方法
- [转载] elasticsearch遇到的一些问题以及解决方法
- 使用elasticsearch遇到的一些问题以及解决方法(不断更新)
- Windows下安装Elasticsearch 5.0.0遇到的问题
- Elasticsearch使用中遇到的一些问题及相应解决方法
- elasticsearch使用中遇到的问题(二)
- elasticsearch使用中遇到的问题(三)
- elasticsearch 安装过程中可遇到的问题
- JUC原子类-04之 AtomicReference原子类
- PAT (Advanced Level) Practise 1018 Public Bike Management (30)
- 背景建模技术:帧处理(FrameProcessor)模块
- Spring与Angular为主打的简洁轻便的物品管理系统
- 541. Reverse String II
- fluend-elasticsearch日志遇到的问题
- 编写类String的构造函数、析构函数和赋值函数(2)
- Deep Learning学习笔记整理系列之一
- 二叉树前中后遍历互求
- 通畅工程
- 欢迎使用CSDN-markdown编辑器
- C语言攻略
- grub-efi-amd64-signed安装失败
- java 多线程编程