Hadoop 磁盘配额测试

来源：互联网发布：java foreach 二维数组编辑：程序博客网时间：2024/06/04 19:58

最近需要使用到HDFS的磁盘配额(Quotas)功能，故稍微学习了一下，并做了一些简单的测试。

HDFS的Quotas分为两种:

Name Quotas ：用于限制HDFS指定目录下的文件或文件夹的数量；
Space Quotas：用于限制HDFS指定目录的最大使用空间，这里要特别注意：如果你的HDFS复制因子是2，那么你上传一个1G的文件，则可用 Space Quotas会减少2G.

下面是一些在我的测试集群测试结果：

hadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -mkdir /quato1hadoop@mt-15:/opt/hadoop/sbin$ hadoop dfsadmin -setQuota 2 /quota1hadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -put start-dfs.sh /quota1hadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -put stop-dfs.sh /quota1    put: The NameSpace quota (directories and files) of directory /quota1 is exceeded: quota=2 file count=3 the name quota value set for each directory, the available name quota remaining, the space quota value set, and the available space quota remaininghadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -count -q /quota1      All-Name-Quo       Remain-Name-Quo        All-Space-Quo   Remain-Space-Quo     File-Num    Directory-Num     Size/Name           2               0                      none            inf                  1              1           3705 /quota1hadoop@mt-15:/opt/hadoop/sbin$ hadoop dfsadmin -setQuota 4 /quota1DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.hadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -count -q /quota1           4               2            none             inf            1            1               3705 /quota1hadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -put stop-dfs.sh /quota1hadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -count -q /quota1           4               1            none             inf            1            2               6911 /quota1hadoop@mt-15:/opt/hadoop/sbin$ hadoop dfsadmin -setSpaceQuota 10M /quota1DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.hadoop@mt-15:/opt/hadoop/sbin$ hadoop fs -count -q /quota1           4               1        10485760        10471938            1            2               6911 /quota1hadoop@mt-15:/opt/hadoop$ du -sh mesos-hadoop-mr1-0.1.1-SNAPSHOT.jar6.1M    mesos-hadoop-mr1-0.1.1-SNAPSHOT.jarhadoop@mt-15:/opt/hadoop$ hadoop fs -put mesos-hadoop-mr1-0.1.1-SNAPSHOT.jar /quota1/15/08/19 10:46:31 WARN hdfs.DFSClient: DataStreamer Exceptionorg.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /quota1 is exceeded: quota = 10485760 B = 10 MB but diskspace consumed = 268449278 B = 256.01 MB......put: The DiskSpace quota of /quota1 is exceeded: quota = 10485760 B = 10 MB but diskspace consumed = 268449278 B = 256.01 MB

0 0