生产环境elasticsearch5.0.1集群的部署配置详解
来源:互联网 发布:mac导入照片到ipad 编辑:程序博客网 时间:2024/05/01 08:11
线上环境elasticsearch5.0.1集群的配置部署
es集群的规划:
硬件:
7台8核、64G内存、2T ssd硬盘加1台8核16G的阿里云服务器
其中一台作为kibana+kafka连接查询的服务器
其他6台都作为node和master两种角色
操作系统:centos7.2 x86_64
为方便磁盘扩容建议将磁盘进行lvm逻辑卷配置,可以参考:
aliyun添加数据盘后的物理分区和lvm逻辑卷两种挂载方式
http://blog.csdn.net/reblue520/article/details/54174178
tar -zxvf elasticsearch-5.0.1.tar.gz
echo 'pass'|passwd --stdin yunva
chown -R yunva.yunva /data
修改默认端口
sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
sed -i 's/^GSSAPIAuthentication yes$/GSSAPIAuthentication no/' /etc/ssh/sshd_config
service sshd restart
echo "fs.file-max = 1000000" >> /etc/sysctl.conf
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
echo "vm.swappiness = 1" >> /etc/sysctl.conf
sysctl -p
sed -i 's/* soft nofile 65535/* soft nofile 655350/g' /etc/security/limits.conf
sed -i 's/* hard nofile 65535/* hard nofile 655350/g' /etc/security/limits.conf
将java_home加入环境变量
cat >> /etc/profile <<EOF
export JAVA_HOME=/usr/java/jdk1.8.0_111
export PATH=\$JAVA_HOME/bin:\$PATH
EOF
source /etc/profile
/data/elasticsearch-5.0.1/config/jvm.options
sed -i 's/-Xms2g/-Xms32g/' /data/elasticsearch-5.0.1/config/jvm.options
sed -i 's/-Xmx2g/-Xmx32g/' /data/elasticsearch-5.0.1/config/jvm.options
echo "-Xss256k" >>/data/elasticsearch-5.0.1/config/jvm.options
sed -i 's/-XX:+UseConcMarkSweepGC/-XX:+UseG1GC/' /data/elasticsearch-5.0.1/config/jvm.options
vim /etc/elasticsearch/elasticsearch.yml(rpm安装方式的配置文件位置)
vim /data/elasticsearch-5.0.1/config/elasticsearch.yml
es master节点的配置:
# 节点名
cluster.name: yunva-es
# 集群的名称,可以不写
discovery.zen.ping.unicast.hosts: ["node-1","yunva_etl_es2", "yunva_etl_es3","yunva_etl_es4","yunva_etl_es5","yunva_etl_es6","yunva_etl_es7"]
node.name: yunva_etl_es6
node.master: true
node.data: true
path.data: /data/es/data
path.logs: /data/es/logs
action.auto_create_index: false
indices.fielddata.cache.size: 12g
bootstrap.memory_lock: false
# 内网地址,可以加快速度
network.host: 192.168.1.10
http.port: 9200
# 增加新的参数head插件可以访问es
http.cors.enabled: true
http.cors.allow-origin: "*"
gateway.recover_after_time: 8m
gateway.expected_nodes: 3
针对kibana的es配置(非node和master节点)
# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: yunva-es
node.name: yunva_etl_es1
node.master: false
node.data: false
node.ingest: false
action.auto_create_index: false
path.data: /data/es/data
path.logs: /data/es/logs
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
http.cors.enabled: true
注意修改配置文件vim /etc/hosts 列出集群节点名称和对应ip地址的对应关系(有内网dns并且配置的就不需要再次配置了)
echo "10.28.50.131 node-1" >> /etc/hosts
echo "10.26.241.239 yunva_etl_es3" >> /etc/hosts
echo "10.25.135.215 yunva_etl_es2" >> /etc/hosts
echo "10.26.241.237 yunva_etl_es4" >> /etc/hosts
echo "10.27.78.228 yunva_etl_es5" >> /etc/hosts
echo "10.27.65.121 yunva_etl_es6" >> /etc/hosts
echo "10.27.35.94 yunva_etl_es7" >> /etc/hosts
mkdir /data/es/logs
chown -R yunva.yunva /data
[yunva]$ cd /data/elasticsearch-5.0.1/bin/
./elasticsearch &
$ curl http://ip:9200/
{
"name" : "yunva_etl_es5",
"cluster_name" : "yunva-es",
"cluster_uuid" : "2shAg8u3SjCRNJ4mEUBzBQ",
"version" : {
"number" : "5.0.1",
"build_hash" : "080bb47",
"build_date" : "2016-11-11T22:08:49.812Z",
"build_snapshot" : false,
"lucene_version" : "6.2.1"
},
"tagline" : "You Know, for Search"
}
# 查看集群状态
$ curl http://ip:9200/_cluster/health/?pretty
{
"cluster_name" : "yunva-es",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 4,
"active_primary_shards" : 66,
"active_shards" : 132,
"relocating_shards" : 2,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
然后将配置好的es程序拷贝到其他服务器中,注意修改以下内容(network.host为内网地址,速度更快,节省互相复制、分片的时候处理带宽):
1.elasticsearch.yml文件的配置修改
node.name: 节点名称
network.host: es节点的内网IP地址
2./etc/hosts文件中内网ip和node.name的对应关系
es.conf
es集群的规划:
硬件:
7台8核、64G内存、2T ssd硬盘加1台8核16G的阿里云服务器
其中一台作为kibana+kafka连接查询的服务器
其他6台都作为node和master两种角色
操作系统:centos7.2 x86_64
为方便磁盘扩容建议将磁盘进行lvm逻辑卷配置,可以参考:
aliyun添加数据盘后的物理分区和lvm逻辑卷两种挂载方式
http://blog.csdn.net/reblue520/article/details/54174178
1.安装jdk1.8和elasticsearch5.0.1
rpm -ivh jdk-8u111-linux-x64.rpmtar -zxvf elasticsearch-5.0.1.tar.gz
2.添加yunva这个运行elasticsearch的用户(es必须使用非root用户启动)
useradd yunva -d /home/yunvaecho 'pass'|passwd --stdin yunva
chown -R yunva.yunva /data
修改默认端口
sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
sed -i 's/^GSSAPIAuthentication yes$/GSSAPIAuthentication no/' /etc/ssh/sshd_config
service sshd restart
3.针对es做的一些系统的优化配置
swapoff -aecho "fs.file-max = 1000000" >> /etc/sysctl.conf
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
echo "vm.swappiness = 1" >> /etc/sysctl.conf
sysctl -p
sed -i 's/* soft nofile 65535/* soft nofile 655350/g' /etc/security/limits.conf
sed -i 's/* hard nofile 65535/* hard nofile 655350/g' /etc/security/limits.conf
将java_home加入环境变量
cat >> /etc/profile <<EOF
export JAVA_HOME=/usr/java/jdk1.8.0_111
export PATH=\$JAVA_HOME/bin:\$PATH
EOF
source /etc/profile
4.es内存调整配置文件(建议配置为物理内存的一半或者更多最好不要超过32G,超过了也可能不会增强性能):
/data/elasticsearch-5.0.1/config/jvm.options
sed -i 's/-Xms2g/-Xms32g/' /data/elasticsearch-5.0.1/config/jvm.options
sed -i 's/-Xmx2g/-Xmx32g/' /data/elasticsearch-5.0.1/config/jvm.options
echo "-Xss256k" >>/data/elasticsearch-5.0.1/config/jvm.options
sed -i 's/-XX:+UseConcMarkSweepGC/-XX:+UseG1GC/' /data/elasticsearch-5.0.1/config/jvm.options
5.集群的主要配置文件
修改elasticsearch的参数vim /etc/elasticsearch/elasticsearch.yml(rpm安装方式的配置文件位置)
vim /data/elasticsearch-5.0.1/config/elasticsearch.yml
es master节点的配置:
# 节点名
cluster.name: yunva-es
# 集群的名称,可以不写
discovery.zen.ping.unicast.hosts: ["node-1","yunva_etl_es2", "yunva_etl_es3","yunva_etl_es4","yunva_etl_es5","yunva_etl_es6","yunva_etl_es7"]
node.name: yunva_etl_es6
node.master: true
node.data: true
path.data: /data/es/data
path.logs: /data/es/logs
action.auto_create_index: false
indices.fielddata.cache.size: 12g
bootstrap.memory_lock: false
# 内网地址,可以加快速度
network.host: 192.168.1.10
http.port: 9200
# 增加新的参数head插件可以访问es
http.cors.enabled: true
http.cors.allow-origin: "*"
gateway.recover_after_time: 8m
gateway.expected_nodes: 3
cluster.routing.allocation.node_initial_primaries_recoveries: 8
# 以下配置可以减少当es节点短时间宕机或重启时shards重新分布带来的磁盘io读写浪费
discovery.zen.fd.ping_timeout: 180s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 120s
针对kibana的es配置(非node和master节点)
# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: yunva-es
node.name: yunva_etl_es1
node.master: false
node.data: false
node.ingest: false
action.auto_create_index: false
path.data: /data/es/data
path.logs: /data/es/logs
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"
# 以下配置可以减少当es节点短时间宕机或重启时shards重新分布带来的磁盘io读写浪费
discovery.zen.fd.ping_timeout: 180s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 120s
注意修改配置文件vim /etc/hosts 列出集群节点名称和对应ip地址的对应关系(有内网dns并且配置的就不需要再次配置了)
echo "10.28.50.131 node-1" >> /etc/hosts
echo "10.26.241.239 yunva_etl_es3" >> /etc/hosts
echo "10.25.135.215 yunva_etl_es2" >> /etc/hosts
echo "10.26.241.237 yunva_etl_es4" >> /etc/hosts
echo "10.27.78.228 yunva_etl_es5" >> /etc/hosts
echo "10.27.65.121 yunva_etl_es6" >> /etc/hosts
echo "10.27.35.94 yunva_etl_es7" >> /etc/hosts
6.创建日志和数据存放目录
mkdir -p /data/es/datamkdir /data/es/logs
chown -R yunva.yunva /data
7.启动es服务:
# su - yunva[yunva]$ cd /data/elasticsearch-5.0.1/bin/
./elasticsearch &
8.检查单台服务是否正常:
$ curl http://ip:9200/
{
"name" : "yunva_etl_es5",
"cluster_name" : "yunva-es",
"cluster_uuid" : "2shAg8u3SjCRNJ4mEUBzBQ",
"version" : {
"number" : "5.0.1",
"build_hash" : "080bb47",
"build_date" : "2016-11-11T22:08:49.812Z",
"build_snapshot" : false,
"lucene_version" : "6.2.1"
},
"tagline" : "You Know, for Search"
}
# 查看集群状态
$ curl http://ip:9200/_cluster/health/?pretty
{
"cluster_name" : "yunva-es",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 4,
"active_primary_shards" : 66,
"active_shards" : 132,
"relocating_shards" : 2,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
然后将配置好的es程序拷贝到其他服务器中,注意修改以下内容(network.host为内网地址,速度更快,节省互相复制、分片的时候处理带宽):
1.elasticsearch.yml文件的配置修改
node.name: 节点名称
network.host: es节点的内网IP地址
2./etc/hosts文件中内网ip和node.name的对应关系
后续添加对集群服务的监控,可以参考:
zabbix通过简单shell命令监控elasticsearch集群状态
http://blog.csdn.net/reblue520/article/details/54412388
kibana.yml
server.port: 9529server.host: "192.168.1.2"elasticsearch.url: "http://192.168.1.1:9200"elasticsearch.pingTimeout: 600000elasticsearch.requestTimeout: 18000000
nginx.conf
user nginx;worker_processes auto;error_log /var/log/nginx/error.log;pid /run/nginx.pid;include /usr/share/nginx/modules/*.conf;events { worker_connections 1024;}http { log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { listen 80 default_server; listen [::]:80 default_server; server_name _; root /usr/share/nginx/html; include /etc/nginx/default.d/*.conf; location / { } error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } }}
es.conf
server { listen 80; server_name 1.1.1.1; location / { auth_basic "secret"; auth_basic_user_file /data/nginx/db/passwd.db; proxy_pass http://192.168.1.2:9200; proxy_set_header Host $host:9200; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Via "nginx"; } access_log off;}
0 0
- 生产环境elasticsearch5.0.1集群的部署配置详解
- elasticsearch5.4.0 生产环境部署 及插件 ik + head +kibana
- 在生产环境部署CDH集群遇到的问题
- ElasticSearch5.2.2 基本概念和集群配置详解
- ElasticSearch5.1.1集群部署
- elasticsearch5.2集群部署
- 生产环境部署之--Fastdfs 生产环境部署配置
- MongoDB实战-生产环境中分片的部署与配置
- elasticsearch5.5.2 配置 集群
- 生产环境下部署hadoop集群(包括dns配置,nfs共享,awk资源分发)
- Elasticsearch5.1集群+kibana部署
- 生产环境下hadoop集群配置实验
- ElasticSearch5.x 集群部署 及启动出现的问题解决
- webpack开发和生产两个环境的配置详解
- 生产环境部署MongoDB集群——性能优化
- tomcat生产环境JDK部署及虚拟主机等常用配置详解
- tomcat生产环境JDK部署及虚拟主机等常用配置详解
- Docker Swarm搭建生产环境的集群
- keil编译不能生成“.HEX”的解决方法
- visual studio 调试时提示 已加载“C:\Windows\SysWOW64\ntdll.dll”。无法查找或打开 PDB 文件。
- 使用maven一步一步构建spring mvc项目
- 解析 HTML -- SGMLParser
- 热修复框架Tinker的从0到集成之路
- 生产环境elasticsearch5.0.1集群的部署配置详解
- BFS入门-A计划
- linux系统——目录,目录项,索引节点(i node)的关系
- 建立live555海思编码推流服务
- Maven构建springMVC
- 算法竞赛入门经典 习题2-6 调和级数(harmony)
- A*B Problem
- 有无序的实数列V[N],要求求里面大小相邻的实数的差的最大值,关键是要求线性空间和线性时间。
- 前端工程目录