Presto常用配置详细

Presto常用配置详细

下载presto-server安装包

http://repo2.maven.org/maven2/com/facebook/presto/presto-server/

presto更新很快，这里是 presto-server-0.179.tar.gz

1、解压

tar zxfv presto-server-0.179.tar.gz

2、presto-server路径下，创建etc目录

mkdir etc

3、config.properties 配置文件，master与slave是不同的

master配置

coordinator=truediscovery-server.enabled=truenode-scheduler.include-coordinator=truehttp-server.http.port=8888query.max-memory=500GBquery.max-memory-per-node=76GBdiscovery.uri=http://hadoopslave10:8888failure-detector.threshold=0.10exchange.http-client.request-timeout=30mhttp-server.threads.max=1500task.max-worker-threads=64exchange.min-error-duration=30mexchange.max-error-duration=150mquery.max-length=100000000

slave配置

coordinator=falsehttp-server.http.port=8889query.max-memory=500GBquery.max-memory-per-node=76GBdiscovery.uri=http://hadoopslave10:8888exchange.http-client.request-timeout=30mhttp-server.threads.max=1500task.max-worker-threads=64exchange.min-error-duration=30mexchange.max-error-duration=150mquery.max-length=100000000                  

区别在于 ，master设置

coordinator=true
discovery-server.enabled=true
node-scheduler.include-coordinator=true

配置资源队列：

修改Presto/etc/config.properties，添加以下配置
query.queue-config-file=/home/presto/presto-server/etc/queue_config.json

{  "queues": {    "user.${USER}": {      "maxConcurrent": 5,      "maxQueued": 20    },    "test": {      "maxConcurrent": 2,      "maxQueued": 3    },    "admin": {      "maxConcurrent": 100,      "maxQueued": 100    },    "global": {      "maxConcurrent": 100,      "maxQueued": 1000    }  },  "rules": [    {       "user": "bob",   #bob为管理员，最多可以同时进行100个查询，有100个在等待队列中      "queues": ["admin"]    },    {       "user": "guest",#guest用户使用test的策略，最多可以同时进行2个查询，有3个在等待队列中      "queues": ["test"]    },    {      "queues": [        "user.${USER}",        "global"      ]     }     ]   } 

2、jvm.config 配置文件

-server-Xmx128G-XX:+UseConcMarkSweepGC-XX:+ExplicitGCInvokesConcurrent-XX:+CMSClassUnloadingEnabled-XX:+AggressiveOpts-XX:+HeapDumpOnOutOfMemoryError-XX:OnOutOfMemoryError=kill -9 %p-XX:ReservedCodeCacheSize=150M

3、log.properties配置文件

com.facebook.presto=DEBUG

4、node.properties文件

node.environment=datamartnode.id=hadoopmaster1node.data-dir=/home/presto/presto-server-0.179/data                                                

注意node.id 每一个节点不相同的，通常与hostname一样

5、etc目录下，创建catalog目录

mkdir catalog

6、hive.properties配置文件

connector.name=hive-hadoop2hive.metastore.uri=thrift://hadoopmaster1:9083hive.config.resources=/etc/hadoop/conf/core-site.xml, /etc/hadoop/conf/hdfs-site.xmlhive.allow-drop-table=truehive.storage-format=PARQUEThive.metastore-cache-ttl=0shive.metastore-refresh-interval=1shive.max-partitions-per-writers=1000

7、集群重启脚本，每个节点配置免密码

# restart slaves ssh hadoopmaster1 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopmaster2 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave1 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave2 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave3 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave4 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave5 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave6 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave7 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave8 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave9 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave11 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave12 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave13 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave14 "/home/presto/presto-server-0.179/bin/launcher restart" &ssh hadoopslave15 "/home/presto/presto-server-0.179/bin/launcher restart" &#restart masterssh hadoopslave10 "/home/presto/presto-server-0.179/bin/launcher restart" &