Hypertable - 管理指南

Administrator Guide

This document provides information on how to administer a hypertable deployment.

本文描述如何管理Hypertable。

Table of Contents

Monitoring System

Hypertable安装监控指导

Backup and Restore

如何备份和恢复表

Upgrading

如何升级到Hypertable的新版本

 

Monitoring System

（http://hypertable.com/documentation/administrator_guide/monitoring_system/）

The Hypertable Monitoring UI is an invaluable tool that provides insight into the health and perational characteristics of your Hypertable deployment.  We highly recommend that you install it (see below) and become intimately familiar with it.  This document provides an overview of the monitoring UI and describes how to get it up and running.

Hypertable UI监控系统是一个很好的工具，它提供了监视Hypertable健康运行和操作的内部视图。我们极力推荐安装(见下图)并熟悉它，本章是监控UI及其启动和运行的描述。

Range Servers Overview

When you first pull up the Monitoring UI inyour browser, the first page you come to is the RangeServer overview page. Itdisplays a table containing statistics for each range server in your system. Each row contains statistics for a single range server, including systemconfiguration information, disk use percentage, range count, clock skew, and basichealth information. The column on the left gives the range server proxy nameand is linked to a page containing RRD graphs of application and general systemstatistics for the range server.  The following lists the RRD graphsprovided on that page.

• scans / updates per second
• cells read / written per second
• table bytes read / written per second
• syncs per second
• disk activity - read / write IOPS
• disk activity - read / write bytes per second
• network activity - RX / TX rate
• load average
• cpu user time percentage
• cpu system time percentage
• outstanding scanner count
• cell store count
• range count
• disk use percentage
• virtual memory size
• resident memory size
• pages swapped in / out
• heap size
• heap slack bytes (unallocated memory on free lists)
• application tracked memory usage
• block cache memory / fill / hit rate
• query cache memory / fill / hit rate

当你第一次在浏览器中打开Monitoring UI时，第一页是RangeServer概览，它显示了一个表，包含系统中每一个RangeServer统计数据，一行一个RangerServer，包括系统配置信息，磁盘使用率，range数，时钟偏移以及基本的健康信息。左侧的列给出了这个range服务器代理的名字，并且它链接到一个包含RRD图信息和range服务器系统统计信息的页。下表列出了该页RRD图所提供的项目。

• 每秒扫描/更新次数
• 每秒读/写单元次数
• 每秒读/写表的字节数
• 每秒同步次数
• 磁盘参数-读/写IOPS
• 磁盘参数-每秒读写字节数
• 网络参数-RX/TX率
• 平均负荷率
• 用户cpu时间(%)
• 系统cpu时间（%）
• 未完成的扫描数目
• 单元存储数目
• Range数目
• 磁盘使用(%)
• 虚拟内存大小
• 驻留的内存大小
• 交换的页面数
• 堆大小
• 堆未分配大小
• 跟踪的应用程序内存使用
• 块缓存fill/hit率
• 查询缓存fill/hit率

 

Tables Overview

If you click the Tables tab at the top of themain monitoring page it will bring you to the Tables overview page.  Clickthe image on the right to see an example (data taken from a real Hypertabledeployment).  This page displays a table of statistics for each table inyour system.  Each row contains statistics for a single table, includingrange count, cell store count, cell count, disk use, memory use, average keysize, average value size, and compression ratio.  The column on the leftgives the full table name (including namespace path) and is linked to a pagecontaining RRD graphs of per-table statistics.  The following lists theRRD graphs provided on that page.

• scans / updates per second
• cells read / written per second
• table bytes read / written per second
• disk bytes read per second
• outstanding scanner count
• range count
• disk used
• memory used / allocated
• block index memory
• bloom filter memory / accesses / maybes

如果你点击monitor UI主页上部的Tables，就会跳转到表概览页。（上图是一个真实Hypertable部署的例子）这个页显示了你系统中每个表的统计信息，每行一个表，包括range数目，单元存储数目，磁盘使用，内存使用，平均key大小，平均value大小以及压缩率等。左边的列是表的完整的名字（包括命名空间），它链接到一个包含表统计信息的RRD的页，下表列出了该页RRD图所提供的项目。

• 每秒扫描/更新次数
• 每秒读/写单元次数
• 每秒读/写表的字节数
• 每秒读磁盘字节数
• 未完成的扫描数目
• range数目
• 已使用的磁盘
• 内存使用/分配
• 块索引内存
• bloom filter memory / accesses / maybes

Installation

The Monitoring UIis written in Ruby and uses the Ruby RRDTool bindings.  It requires ruby1.8.7 or greater.  Verify that you are running the correct version ofruby:

If your system has an older version of ruby, you'll need to upgrade ruby and ruby gems.  In some circumstances, you'll need to build ruby (and ruby gems) from source, see CentOS 5 Installation (below) for an example of how to do this.  Once you've verified that you are running the correct version of ruby, install there quired gems as follows.

Monitoring UI是用Ruby写的，使用了Ruby RRDTool绑定。它需要ruby1.8.7或更高版本。请确认你的ruby版本的正确性。

sudo gem installsinatra rack thin json titleize

The next step isto install RRDTool.  The following describes how to install RRDTool onvarious platforms.

下一步是安装RRDTool。下面描述了各种平台上如何安装RRDTool。

Redhat (CentOS)

sudo yum installrrdtool

Debian (Ubuntu)

sudo apt-get installrrdtool

Mac OSX

For the mac, we recommend that you use MacPorts to install rrdtool.  MacPorts requires that you first install Xcode.  Once you have Mac Ports installed, you can install rrdtool as follows.

对Mac，我们建议你使用MacPorts来安装RRDTool。MacPorts需要你安装Xcode。如果你安装好MacPorts，就可以采用如下的命令来安装RRDTools。

sudo port installrrdtool

CentOS 5 Installation

The CentOS(Redhat) 5 series of operating systems provides an older version of ruby that is incompatible with the Hypertable UI.  Run the following script as root to upgrade Ruby and to get the dependencies set up properly.

CentOS 5系列操作系统提供了一个老的Ruby版本，它与Hypertable UI不兼容。运行以下脚本来升级Ruby，并正确得到依赖的库。

#!/usr/bin/env bash

 

# Upgrade to ruby1.8.7

yum -y remove ruby

yum -y install gccgcc-c++ zlib-devel openssl-devel readline-devel sqlite3-devel

wget http://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7.tar.gz

tar xzvfruby-1.8.7.tar.gz

cd ruby-1.8.7

./configure

make

make install

cd ~

rm -rf ruby-1.8.7*

hash -r

 

wget http://rubyforge.org/frs/download.php/69365/rubygems-1.3.6.tgz

tar xzvfrubygems-1.3.6.tgz

cd rubygems-1.3.6

ruby setup.rb

cd ~

rm -rf rubygems-1.3.6*

hash -r

 

gem installcapistrano sinatra rack thin json titleize

 

yum installl rrdtool

 

/sbin/ldconfig

Starting and Stopping the UI

If you're using Capistrano to administer your Hypertable cluster, no special steps are requiredto start and stop the monitoring UI.  It will automatically start and stop with the Capistrano commands to start and stop Hypertable.

如果你采用Capistrano管理Hypertable集群，则启动和停止monitoring UI不需要额外的步骤，用Capistrano启停Hypertable的命令就可以自动启停它。

cap start

cap stop

If you're running Hypertable in standalone mode, you need to start and stop the monitoring UI manually as follows.

如果你采用单机版的Hypertable，你需要手工启停monitoring UI，命令如下：

/opt/hypertable/current/bin/start-monitoring.sh

/opt/hypertable/current/bin/stop-monitoring.sh

Accessing the UI

The monitoring UIis accessed through TCP port 38090, either on the standalone machine or on the master machine if running on a cluster.  Make sure port 38090 is not blocked by a firewall.  If the master machine (called master in the example below) is directly accessible, you can pull up the UI by pointing your browser to the following URL.

通过TCP端口38090可以访问monitoring UI，不管是单机部署方式还是集群部署方式（master机器），需要确保38090端口没有被防火墙封闭。如果master机器能够被直接访问，则在浏览器中使用如下的URL就可以得到UI页面。

http://master:38090/

If you don't have direct access to the master machine, you can create an ssh tunnel that allows you to access the UI.  Assuming that you have ssh access to a machine called admin.yourcompany.com which has direct (unblocked) access to the master machine, you can create a bash alias that will setup an ssh tunnel to the monitoring UI as follows.

如果你不能直接访问master机器，你可以创建一个ssh通道，允许你访问monitoring UI。假设你建立了一个ssh，可以访问一个叫admin.yourcompany.com的机器，而该机器能直接访问master机器，则你可以用bash alias创建一个访问monitoring UI的ssh通道，如下所示：

aliashypertable-monitoring="ssh -f -L 38090:master:38090 admin.yourcompany.com-N; echo 'http://localhost:38090/'"

If you're runninga shell other than bash, consult the shell documentation to find out how to setup a similar alias. To create the tunnel, run the alias as follows.

如果你不是运行的bash shell，关于如何建立一个类似的alias，请参阅该shell文档。为建立一个通道，运行如下的alias：

$hypertable-monitoring

http://localhost:38090/

 

 

 

Backup andRestore

（http://hypertable.com/documentation/administrator_guide/backing_up_and_restoring_tables/）

Hypertable backup and restore are accomplished with the HQL commands DUMP TABLE and LOAD DATAINFILE. It is possible to create backups with the SELECT command, but it is not recommended because it will output table data in sorted order. When loading data with LOAD DATA INFILE that is in sorted order, only one RangeServer will be kept busy at a time. To combat this problem, the DUMP TABLE command was introduced which will output table data in random order. Loading backups taken with the DUMP TABLE command is much more efficient since all of the participating RangeServers will be kept busy.

要完成备份与恢复，Hypertable采用HQL命令DUMP TABLE 和 LOAD DATA INFILE。虽然可以用SELECT命令来备份，但不是推荐的方法，因为它输出的是排序后的数据，当用LOAD DATA INFILE来恢复这些数据时，同一时刻只有一个RangeServer工作。为克服这个问题，就引入了DUMP TABLE命令，它输出的数据是随机的，载入用DUMP TABLE命令备份的数据，因为所有的RangeServer都可能参与工作，所以十分高效。

backup.sh

#!/usr/bin/env bash

 

if [ $# -lt 2 ] ;then

   echo "usage: $0 <namespace><table> [  ... ]"

   exit 0

fi

 

namespace=$1

shift

while [ $# -ge 1 ];do

   echo "Backing up '${namespace}/$1'..."

   echo "USE '${namespace}';" >create-table-$1.hql

   echo "USE '${namespace}'; SHOW CREATETABLE '$1';" | /opt/hypertable/current/bin/ht shell --silent >>create-table-$1.hql

   echo ";" >>create-table-$1.hql

   echo "USE '${namespace}'; DUMP TABLE'$1' INTO FILE '$1.gz';" | /opt/hypertable/current/bin/ht shell

   shift

done

restore.sh

#!/usr/bin/env bash

 

if [ $# -lt 2 ] ;then

   echo "usage: $0   [  ...]"

   exit 0

fi

 

namespace=$1

shift

 

while [ $# -ge 1 ];do

   table=`basename $1 | cut -f1 -d'.'`

   echo "Restoring'${namespace}/${table}' ..."

   cat create-table-$1.hql |/opt/hypertable/current/bin/ht shell

   echo "USE '${namespace}'; LOAD DATAINFILE '$1.gz' INTO TABLE '$table';" | /opt/hypertable/current/bin/htshell

   shift

done

 

 

 

 

 

Upgrading

（http://hypertable.com/documentation/administrator_guide/upgrading/）

This documentdescribes how to upgrade a Hypertable installation.

本节描述如何升级Hypertable。

Table of Contents

Determine if Upgrade is Format Compatible

Format Compatible Upgrade

• Step 1 - Stop Hypertable
• Step 2 - Installthe New Package
• Step 3 – Upgrade
• Step 4 - StartHypertable

Format Incompatible upgrade

Determine if Upgrade isFormat Compatible

确定升级是否格式兼容

To determinewhether your existing Hypertable installation is format-compatible with aversion that you are trying to upgrade to, we've provided a script, ugrade-ok.sh,to help you make that determination.  The script has the following usage:

为确定你现在的Hypertable的格式与你要升级的是否兼容，我们提供了一个脚本ugrade-ok.sh，该脚本可以帮助你确定。该脚本用法如下：

$/opt/hypertable/current/bin/upgrade-ok.sh

 

usage: upgrade-ok.sh<from> <to>

 

description:

  Determines whether or not the upgrade fromHypertable

  version <from> to version <to> isvalid.  <from> and <to>

  are assumed to be Hypertable installationdirectories

  whose last path component is either a versionnumber or

  the symbolic link "current" whichpoints to a Hypertable

  installation directory whose last pathcomponent is a

  version number.

 

return:

  Zero if upgrade is OK, non-zero otherwise

You can test the current installation against a particular version (e.g. 0.9.6.0) with a commandsuch as the following:

你可以用如下命令来检测一下你现在的版本是否与某个特定版本（例如0.9.6.0）兼容。

$/opt/hypertable/current/bin/upgrade-ok.sh /opt/hypertable/current 0.9.6.0

$ echo $?

0

You can test a specific version against another one with a command such as the following:

你也可以用如下的命令来检测某一版本是否与另一版本兼容。

$/opt/hypertable/current/bin/upgrade-ok.sh 0.9.3.0 0.9.4.0

Incompatible upgradea: 0.9.3.0 -> 0.9.4.0

$ echo $?

1

Format Compatible Upgrade

格式兼容下的升级

If you're upgrading to a new release of Hypertable that is format compatible with your old one, upgrading is quick and painless, just follow the four steps listed below.

如果你的旧版本与你要升级到的新版本兼容，升级就是一个快速简单的过程，遵照如下步骤：

Step 1 - StopHypertable

步骤1 – 停止Hypertable

For cluster installations, use the following command:

对集群部署，用如下命令:

$ cap stop

For standalone installations, use the following command:

对单机部署，用如下命令:

$/opt/hypertable/current/bin/stop-servers.sh

Step 2 - Installthe New Package

步骤2 – 安装新包

For cluster installations, follow the instructions under Hadoop Installation - InstallHypertable Binaries (works for MapR installation as well). For standalone installations, follow the instructions under Standalone Installation - InstallHypertable Binaries.  To install packages on ThriftBroker machines in a cluster environment in which you've listed the machines under the thriftbroker role in the Capfile, then the Hadoop Installation listed abovewill also upgrade those machines (works for MapR installations as well). If you've installed ThriftBroker-only packages manually, follow the instructions underThriftBroker Only - InstallHypertable Binaries.

对集群部署，遵循Hadoop情况下的安装指导 – 安装Hypertable二进制包（MapR下也如此）。对单机部署，遵循单机情况下的安装指导 – 安装Hypertable二进制包。在集群情况下安装包到ThriftBroker机器时，Capfile文件中已经有thriftbroker角色的机器列表，所以Hadoop包会升级那些机器（MapR下也如此）。如果你采用的是手工安装的ThriftBroker-only包，遵循ThriftBroker Only的安装指导 -安装Hypertable二进制包。

Step 3 – Upgrade

步骤3 – 升级

ClusterInstallation

集群安装

For cluster installations, run the followingcommand:

对集群安装，运行如下命令：

$ cap upgrade

This command performs the following actions:

1. Verifies that the upgrade is format compatible
2. Copies the contents of the conf/, hyperspace/, fs/, run/, and log/ directories from the old installation (referenced from the current symlink) into the new installation. If any of these directories are sym linked, then an identical symlink will be created in the new installation and the contents are not copied for that directory.
3. Sets the current symlink on all machines to point to the new installation

这个命令产生如下动作：

1. 验证升级是格式兼容的
2. 拷贝旧安装下conf/, hyperspace/, fs/, run/, 和 log/目录中的内容到新安装下。如果这些目录中有符号链接，则在新安装下建一个同样的链接，而不拷贝那个目录下的内容。
3. 在所有机器上设置current链接指向新安装目录。

StandaloneInstallation

单机安装

For standalone installations, run thefollowing commands.  You'll need to change the VERSION variable assignmentto the version you are upgrading to, in this example it is 0.9.5.5.

对于单机安装，运行如下命令。你需要修改VERSION变量，使其是新的版本，本例中，它是0.9.5.5.

VERSION=0.9.5.5

/opt/hypertable/$VERSION/bin/upgrade.sh/opt/hypertable/current $VERSION

rm -f/opt/hypertable/current

ln -s/opt/hypertable/$VERSION /opt/hypertable/current

Step 4 - StartHypertable

步骤4 – 启动Hypertable

For cluster installations, run the following command:

对集群安装，运行如下命令：

$ cap start

For standalone installations, run the following command:

对单机安装，，运行如下命令：

/opt/hypertable/current/bin/start-all-servers.shlocal

Format Incompatible upgrade

格式不兼容下升级

If the new release you are installing is incompatible with the old release, you will needto backup and restore all of your tables. See the section on how to backup and restore tables. Here's the basic upgrade sequence:

1. Backup your tables using the backup.sh script shown underBackup and Restore
2. Remove existing installation (e.g. rm -rf /opt/hypertable/$OLDVERSION)
3. Install the new release (see Installation)
4. Restore your tables using the restore.sh script shown underBackup and Restore

如果你要安装的新版本与老版本不兼容，你需要备份然后再恢复所有表。参见如何进行备份与恢复那节的文档。以下是升级的基本步骤：

1. 用backup.sh脚本备份你的表（见“备份与恢复”一节）
2. 删除存在的安装（例如rm -rf /opt/hypertable/$OLDVERSION）
3. 安装新版本（见“安装”一节）
4. 用restore.sh脚本恢复你的表（见“备份与恢复”一节）