使用Bosh在OpenStack上部署CloudFoundry碰到的问题

CloudFoundry部署交流QQ群：176302388

1、部署Micro Bosh碰到的问题

1.1、Micro Bosh虚拟机上执行micro bosh部署命令时出错：

bosh micro deploy /var/vcap/stemcells/micro-bosh-stemcell-openstack-kvm-0.8.1.tgz

报错信息：

Could not find Cloud Provider Plugin: openstack

这其实不是真正的报错信息，可以查看抛出该异常的代码文件查看，文件位置：/usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_cpi-0.5.1/lib/cloud/provider.rb

将该代码块的异常处理去掉，改为如下，暴露出真正的异常：

module Bosh::Clouds  class Provider    def self.create(plugin, options)      require "cloud/#{plugin}"      Bosh::Clouds.const_get(plugin.capitalize).new(options)    end  endend

异常信息：

Failed to load plugingems/bosh_deployer-1.4.1/lib/bosh/cli/commands/micro.rb: Unable to activatefog-1.10.1, because net-scp-1.0.4 conflicts with net-scp (~> 1.1)

可以看出，这是由于gem包冲突造成的错误，查看gem包依赖，可以看出，Bosh的Gem包bosh_cli和bosh_deployer都对fog包和net-scp包有依赖，但是版本要求不一样，所以导致了该错误。

解决办法：

卸载掉bosh_cli、bosh_deployer、fog、net-scp的所有包，然后使用bundle方式进行安装

gem uninstall bosh_cligem uninstall bosh_deployergem uninstall foggem uninstall net-scp

在根目录下新建Gemfile文件，增加需要安装的Gems

source "https://RubyGems.org"gem "bosh_cli"gem "bosh_deployer"

保存后执行

#安装Gemsbundle install#验证Bosh Microbundle exec bosh micro

问题解决，重新执行部署Micro Bosh的命令

经验总结：有的时候报错的真正原因会被源码封装，以至于从日志中得到的错误信息不准确，在Google中找不到任何解决办法，通过深入阅读并修改源码后，暴露出真正的异常原因，找到并解决问题

1.2、endpoint版本问题

bosh micro deploy /var/vcap/stemcells/micro-bosh-stemcell-openstack-kvm-0.8.1.tgz

报错信息：

response => #<Excon::Response:0x00000000992b90 @body="{\"versions\": [{\"status\": \"EXPERIMENTAL\", \"id\": \"v2\", \"links\": [{\"href\": \"http://ip:9292/v2/\", \"rel\": \"self\"}]}, {\"status\": \"CURRENT\", \"id\": \"v1.1\", \"links\": [{\"href\": \"http://ip:9292/v1/\", \"rel\": \"self\"}]}, {\"status\": \"SUPPORTED\", \"id\": \"v1.0\", \"links\": [{\"href\": \"http://ip:9292/v1/\", \"rel\": \"self\"}]}]}", @headers={"Content-Type"=>"application/json", "Content-Length"=>"340", "Date"=>"Wed, 22 Aug 2013 16:38:30 GMT"}, @status=300>

这是应为glance endpoint的fog组件不支持v2.0，只支持v1.0和v1.1

解决办法：需要重新建v1.0的endpoint，执行以下命令：

keystone endpoint-create \    --region RegionOne \    --service_id c933887a2e3341b18bdae2c92e6f1ba7 \    --publicurl "http://10.68.19.61:9292/v1.0" \    --adminurl "http://10.68.19.61:9292/v1.0" \    --internalurl "http://10.68.19.61:9292/v1.0"

注意修改service_id和各url，并且需要删除原来的v2版本的endpoint，否则会报重复定义。

经验总结：文档一类的东西，并不是说官方的就表示肯定是正确的，要对任何技术文档都抱有怀疑态度，遇到问题首先考虑自己的操作是否是否，确认无误后要去了解官方文档的正确性，在安装openstack、Bosh、CloudFoundry的过程中，很多问题官方文档并未指出，但确实存在。

2、部署Bosh碰到的问题

2.1、部署Bosh时quota超额问题

执行bosh deploy部署Bosh时，会出现以下报错信息：

Creating bound missing VMs  small/2: Expected([200, 202]) <=> Actual(413 Request Entity Too Large)                              request => {:connect_timeout=>60, :headers=>{"Content-Type"=>"application/json", "X-Auth-Token"=>"38fe51b931184a30a287e71bc37cc05d", "Host"=>"10.23.54.150:8774", "Content-Length"=>422}, :instrumentor_name=>"excon", :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/excon-0.16.2/data/cacert.pem", :ssl_verify_peer=>true, :write_timeout=>60, :host=>"10.23.54.150", :path=>"/v2/69816bacecd749f9ba1d68b3c8bae1f1/servers.json", :port=>"8774", :query=>"ignore_awful_caching1362453746", :scheme=>"http", :body=>"{\"server\":{\"flavorRef\":\"25\",\"imageRef\":\"e205b9ec-0e19-4500-87fe-ede3af13b227\",\"name\":\"vm-b875d6d8-81ce-483b-bfa8-d6d525aaf280\",\"metadata\":{},\"user_data\":\"eyJyZWdpc3RyeSI6eyJlbmRwb2ludCI6Imh0dHA6Ly8xMC4yMy41MS4zNToy\\nNTc3NyJ9LCJzZXJ2ZXIiOnsibmFtZSI6InZtLWI4NzVkNmQ4LTgxY2UtNDgz\\nYi1iZmE4LWQ2ZDUyNWFhZjI4MCJ9LCJkbnMiOnsibmFtZXNlcnZlciI6WyIx\\nMC4yMy41NC4xMDgiXX19\\n\",\"key_name\":\"jae2\",\"security_groups\":[{\"name\":\"default\"}]}}", :expects=>[200, 202], :method=>"POST"}  response => #<Excon::Response:0x00000004edec30 @body="{\"overLimit\": {\"message\": \"Quota exceeded for instances: Requested 1, but already used 10 of 10 instances\", \"code\": 413}}", @headers={"Retry-After"=>"0", "Content-Length"=>"121", "Content-Type"=>"application/json; charset=UTF-8", "X-Compute-Request-Id"=>"req-c5427ed2-62af-47b9-98a6-6f114893d8fc", "Date"=>"Tue, 05 Mar 2013 03:22:27 GMT"}, @status=413> (00:00:02)

这是OpenStack相关工程quota配额不足而导致出现的错误

解决办法：修改openstack中该工程的quota配额，可以将CPU数、内存、硬盘、IP等都调整的的较大，并将原来的deployments删除，命令如下：

bosh deployments  bosh delete deployment 名称

回滚之前部署时的操作，否则会出现错误，提示某个vm找不到，回滚后如果还存在问题，看看是否有缓存的yml文件，还有nova库中的instances等几个实例相关的表是否还存在没有删除的内容。

2.2、部署Bosh时的RateLimit报错问题

Error 100: Expected([200, 202]) <=> Actual(413 Request Entity Too Large)  request => {:connect_timeout=>60, :headers=>{"Content-Type"=>"application/json", "X-Auth-Token"=>"23bc718661d54252aba2d9c348c264e3", "Host"=>"10.68.19.61:8774", "Content-Length"=>44}, :instrumentor_name=>"excon", :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/excon-0.16.2/data/cacert.pem", :ssl_verify_peer=>true, :write_timeout=>60, :host=>"10.68.19.61", :path=>"/v2/8cf196acd0494fb0bc8d04e47ff77893/servers/046ac2d3-09b5-4abe-ab61-64b33d1348e1/action.json", :port=>"8774", :query=>"ignore_awful_caching1366685706", :scheme=>"http", :body=>"{\"addFloatingIp\":{\"address\":\"10.68.19.132\"}}", :expects=>[200, 202], :method=>"POST"}  response => #<Excon::Response:0x000000044862d8 @body="{\"overLimit\": {\"message\": \"This request was rate-limited.\", \"code\": 413, \"retryAfter\": \"2\", \"details\": \"Only 10 POST request(s) can be made to * every minute.\"}}", @headers={"Retry-After"=>"2", "Content-Length"=>"161", "Content-Type"=>"application/json; charset=UTF-8", "Date"=>"Tue, 23 Apr 2013 02:55:06 GMT"}, @status=413>

查看Openstack文档，可以看到一下Limit信息：

Table 4.15. Default API Rate LimitsHTTP methodAPI URIAPI regular expressionLimitPOSTany URI (*).*10 per minutePOST/servers^/servers50 per dayPUTany URI (*).*10 per minuteGET*changes-since*.*changes-since.*3 per minuteDELETEany URI (*).*100 per minute

解决办法（修改所有计算节点上的nova配置：/etc/nova/api-paste.ini）：

1、去掉Compute API Rate Limiting配置

[composite:openstack_compute_api_v2]use = call:nova.api.auth:pipeline_factorynoauth = faultwrap sizelimit noauth ratelimit osapi_compute_app_v2keystone = faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2keystone_nolimit = faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v2[composite:openstack_volume_api_v1]use = call:nova.api.auth:pipeline_factorynoauth = faultwrap sizelimit noauth ratelimit osapi_volume_app_v1keystone = faultwrap sizelimit authtoken keystonecontext ratelimit osapi_volume_app_v1keystone_nolimit = faultwrap sizelimit authtoken keystonecontext osapi_volume_app_v1

去掉其中的ratelimit的filter即可。

2、修改Compute API Rate Limiting Value，在[filter:ratelimit]下增加limits配置并修改其中的值，如下代码：

[filter:ratelimit]paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factorylimits =(POST, "*", .*, 10, MINUTE);(POST, "*/servers", ^/servers, 50, DAY);(PUT, "*", .*, 10, MINUTE);(GET, "*changes-since*", .*changes-since.*, 3, MINUTE);(DELETE, "*", .*, 100, MINUTE)

修改完之后重启nova-api服务，参考Openstack官方文档：http://docs.openstack.org/havana/config-reference/content/configuring-compute-API.html

2.3、部署Bosh时碰到的director通信错误

执行bosh deploy发布bosh任务的时候，执行到update director的时候，会因为与虚拟机的通信错误造成失败，错误显示如下：

Updating job director  director/0 (canary) (00:01:58)Done             1/1 00:01:58Error 400007: `director/0' is not running after updateTask 3 error

命令行执行bosh vms显示如下：

Deployment `bosh-openstack'Director task 5Task 5 done+----------------------+---------+---------------+--------------------------+| Job/index            | State   | Resource Pool | IPs                      |+----------------------+---------+---------------+--------------------------+| blobstore/0          | running | small         | 50.50.0.23               || director/0           | failing | small         | 50.50.0.20, 10.68.19.132 || health_monitor/0     | running | small         | 50.50.0.21               || nats/0               | running | small         | 50.50.0.19               || openstack_registry/0 | running | small         | 50.50.0.22, 10.68.19.133 || postgres/0           | running | small         | 50.50.0.17               || powerdns/0           | running | small         | 50.50.0.14, 10.68.19.131 || redis/0              | running | small         | 50.50.0.18               |+----------------------+---------+---------------+--------------------------+VMs total: 8

可以看出director任务的状态是failing，连接到director的虚拟机上查看日志可以看到：

/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/connection/ruby.rb:26:in `initialize': getaddrinfo: Name or service not known (SocketError)        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/connection/ruby.rb:26:in `new'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/connection/ruby.rb:26:in `block in connect'        from /var/vcap/data/packages/ruby/2.1/lib/ruby/1.9.1/timeout.rb:57:in `timeout'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/connection/ruby.rb:124:in `with_timeout'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/connection/ruby.rb:25:in `connect'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/client.rb:204:in `establish_connection'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/client.rb:23:in `connect'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/client.rb:224:in `ensure_connected'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/client.rb:114:in `block in process'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/client.rb:191:in `logging'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/client.rb:113:in `process'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis/client.rb:38:in `call'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis.rb:150:in `block in get'        from /var/vcap/data/packages/ruby/2.1/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/redis-2.2.0/lib/redis.rb:149:in `get'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:425:in `job'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:357:in `unregister_worker'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:145:in `ensure in work'        from /var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:145:in `work'        from /var/vcap/packages/director/bosh/director/bin/worker:77:in `<main>'

解决办法（命令行执行）：

monit stop directormonit start director

执行后没有任何反馈，稍等片刻后回到Bosh Cli的机器上查看虚拟机状态如下：

Deployment `bosh-openstack'Director task 6Task 6 done+----------------------+---------+---------------+--------------------------+| Job/index            | State   | Resource Pool | IPs                      |+----------------------+---------+---------------+--------------------------+| blobstore/0          | running | small         | 50.50.0.23               || director/0           | running | small         | 50.50.0.20, 10.68.19.132 || health_monitor/0     | running | small         | 50.50.0.21               || nats/0               | running | small         | 50.50.0.19               || openstack_registry/0 | running | small         | 50.50.0.22, 10.68.19.133 || postgres/0           | running | small         | 50.50.0.17               || powerdns/0           | running | small         | 50.50.0.14, 10.68.19.131 || redis/0              | running | small         | 50.50.0.18               |+----------------------+---------+---------------+--------------------------+VMs total: 8

可以看出，director任务已经正常启动，此时，再重新部署下bosh即可正常，注意，不用删除原来的deployements，直接命令行执行bosh deploy即可。

root@bosh-cli:/var/vcap/deployments# bosh deployGetting deployment properties from director...Compiling deployment manifest...Please review all changes carefullyDeploying `bosh.yml' to `microbosh-openstack' (type 'yes' to continue): yesDirector task 9Preparing deployment  binding deployment (00:00:00)  binding releases (00:00:00)  binding existing deployment (00:00:00)  binding resource pools (00:00:00)  binding stemcells (00:00:00)  binding templates (00:00:00)  binding properties (00:00:00)  binding unallocated VMs (00:00:00)  binding instance networks (00:00:00)Done             9/9 00:00:00Preparing package compilation  finding packages to compile (00:00:00)Done             1/1 00:00:00Preparing DNS  binding DNS (00:00:00)Done             1/1 00:00:00Preparing configuration  binding configuration (00:00:01)Done             1/1 00:00:01Updating job blobstore  blobstore/0 (canary) (00:02:31)Done             1/1 00:02:31Updating job openstack_registry  openstack_registry/0 (canary) (00:01:44)Done             1/1 00:01:44Updating job health_monitor  health_monitor/0 (canary) (00:01:43)Done             1/1 00:01:43Task 9 doneStarted         2013-05-21 05:17:01 UTCFinished        2013-05-21 05:23:00 UTCDuration        00:05:59Deployed `bosh.yml' to `microbosh-openstack'

问题解决！

3、部署CloudFoundry遇到的问题

3.1、上传Cloudfoundry release包时碰到的问题

执行上传命令：

 bosh upload release /var/vcap/releases/cf-release/CF_Release_VF-131.1-dev.tgz

报错如下：

E, [2013-05-24T06:38:59.646076 #24082] [task:1] ERROR -- : Could not create object, 400//var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/blobstore_client-0.5.0/lib/blobstore_client/simple_blobstore_client.rb:30:in `create_file'/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/blobstore_client-0.5.0/lib/blobstore_client/base.rb:30:in `create'/var/vcap/packages/director/bosh/director/lib/director/blob_util.rb:9:in `block in create_blob'/var/vcap/packages/director/bosh/director/lib/director/blob_util.rb:9:in `open'/var/vcap/packages/director/bosh/director/lib/director/blob_util.rb:9:in `create_blob'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:373:in `create_package'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:291:in `block (2 levels) in create_packages'/var/vcap/packages/director/bosh/director/lib/director/event_log.rb:58:in `track'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:289:in `block in create_packages'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:286:in `each'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:286:in `create_packages'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:272:in `process_packages'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:131:in `process_release'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:48:in `block in perform'/var/vcap/packages/director/bosh/director/lib/director/lock_helper.rb:47:in `block in with_release_lock'/var/vcap/packages/director/bosh/director/lib/director/lock.rb:58:in `lock'/var/vcap/packages/director/bosh/director/lib/director/lock_helper.rb:47:in `with_release_lock'/var/vcap/packages/director/bosh/director/lib/director/jobs/update_release.rb:48:in `perform'/var/vcap/packages/director/bosh/director/lib/director/job_runner.rb:98:in `perform_job'/var/vcap/packages/director/bosh/director/lib/director/job_runner.rb:29:in `block in run'/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/bosh_common-0.5.4/lib/common/thread_formatter.rb:46:in `with_thread_name'/var/vcap/packages/director/bosh/director/lib/director/job_runner.rb:29:in `run'/var/vcap/packages/director/bosh/director/lib/director/jobs/base_job.rb:8:in `perform'/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/job.rb:127:in `perform'/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:163:in `perform'/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:130:in `block in work'/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:116:in `loop'/var/vcap/packages/director/bosh/director/vendor/bundle/ruby/1.9.1/gems/resque-1.15.0/lib/resque/worker.rb:116:in `work'/var/vcap/packages/director/bosh/director/bin/worker:77:in `<main>'

不能在bosh管理机中的blobstore虚拟机上创建文件，分析原因，可能是磁盘空间问题，SSH到blodstore的虚拟机上，查看目录/var/vcap/store的挂载点大小：

root@1154e252-382e-4cf7-bb2d-09adbc97a954:~# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/vda1             1.3G  956M  273M  78% /none                  241M  168K  241M   1% /devnone                  247M     0  247M   0% /dev/shmnone                  247M   52K  247M   1% /var/runnone                  247M     0  247M   0% /var/locknone                  247M     0  247M   0% /lib/init/rw/dev/vdb2              20G  246M   18G   2% /var/vcap/data/dev/loop0            124M  5.6M  118M   5% /tmp

挂载点无/var/vcap/store的格外挂载，根目录挂载大小为/dev/vda1，大小只有1.3G，上图中是删除了之前上传内容后的大小，没删除之前，/dev/vda1已经100%使用

解决办法：

1、创建大小20G（可根据实际情况设定）的/dev/vda2磁盘，挂载到/var/vcap/store，挂载前将其中的文件拷贝出来，挂载上之后再拷贝进去，结果：

root@1154e252-382e-4cf7-bb2d-09adbc97a954:~# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/vda1             1.3G  956M  273M  78% /none                  241M  168K  241M   1% /devnone                  247M     0  247M   0% /dev/shmnone                  247M   52K  247M   1% /var/runnone                  247M     0  247M   0% /var/locknone                  247M     0  247M   0% /lib/init/rw/dev/vdb2              20G  246M   18G   2% /var/vcap/data/dev/loop0            124M  5.6M  118M   5% /tmp/dev/vda2              19G  1.8G   16G  11% /var/vcap/store

2、执行release删除命令

root@bosh-cli:/var/vcap/deployments# bosh releases+---------------+-----------+| Name          | Versions  |+---------------+-----------+| CF_Release_VF | 131.1-dev |+---------------+-----------+Releases total: 1root@bosh-cli:/var/vcap/deployments# bosh delete release CF_Release_VF

3、再次执行release上传命令。

3.2、上传cf-service release包时发生错误

错误信息如下：

root@bosh-cli:~/src/cloudfoundry/cf-services-release# bosh upload releaseUpload release `CF-VF-Service-Release-0.1-dev.yml' to `bosh' (type 'yes' to continue): yesCopying packages----------------ruby (0.1-dev)                SKIPlibyaml (0.1-dev)             SKIPmysql (0.1-dev)               FOUND LOCALruby_next (0.1-dev)           SKIPpostgresql_node (0.1-dev)     FOUND LOCALmysqlclient (0.1-dev)         SKIPsyslog_aggregator (0.1-dev)   FOUND LOCALmysql_gateway (0.1-dev)       FOUND LOCALpostgresql92 (0.1-dev)        FOUND LOCALmysql55 (0.1-dev)             FOUND LOCALpostgresql_gateway (0.1-dev)  FOUND LOCALpostgresql91 (0.1-dev)        FOUND LOCALmysql_node (0.1-dev)          FOUND LOCALpostgresql (0.1-dev)          FOUND LOCALsqlite (0.1-dev)              SKIPcommon (0.1-dev)              SKIP....Release info------------Name:    CF-VF-Service-ReleaseVersion: 0.1-devPackages  - ruby (0.1-dev)  - libyaml (0.1-dev)  - mysql (0.1-dev)  - ruby_next (0.1-dev)  - postgresql_node (0.1-dev)  - mysqlclient (0.1-dev)  - syslog_aggregator (0.1-dev)  - mysql_gateway (0.1-dev)  - postgresql92 (0.1-dev)  - mysql55 (0.1-dev)  - postgresql_gateway (0.1-dev)  - postgresql91 (0.1-dev)  - mysql_node (0.1-dev)  - postgresql (0.1-dev)  - sqlite (0.1-dev)  - common (0.1-dev)Jobs  - mysql_node_external (0.1-dev)  - postgresql_node (0.1-dev)  - mysql_gateway (0.1-dev)  - postgresql_gateway (0.1-dev)  - mysql_node (0.1-dev)  - rds_mysql_gateway (0.1-dev)....Director task 20Extracting release  extracting release (00:00:08)Done                    1/1 00:00:08Verifying manifest  verifying manifest (00:00:00)Done                    1/1 00:00:00Resolving package dependencies  resolving package dependencies (00:00:00)Done                    1/1 00:00:00Creating new packages  ruby/0.1-dev: Could not fetch object, 404/ (00:00:01)Error                   1/16 00:00:01Error 100: Could not fetch object, 404/Task 20 errorE, [2013-07-22T02:45:32.218762 #16460] [task:20] ERROR -- : Could not fetch object, 404//var/vcap/packages/director/gem_home/gems/blobstore_client-1.5.0.pre.3/lib/blobstore_client/dav_blobstore_client.rb:48:in `get_file'/var/vcap/packages/director/gem_home/gems/blobstore_client-1.5.0.pre.3/lib/blobstore_client/base.rb:50:in `get'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/blob_util.rb:16:in `block (2 levels) in copy_blob'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/blob_util.rb:15:in `open'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/blob_util.rb:15:in `block in copy_blob'/var/vcap/packages/ruby/lib/ruby/1.9.1/tmpdir.rb:83:in `mktmpdir'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/blob_util.rb:13:in `copy_blob'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:352:in `create_package'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:283:in `block (2 levels) in create_packages'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/event_log.rb:58:in `track'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:281:in `block in create_packages'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:278:in `each'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:278:in `create_packages'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:264:in `process_packages'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:134:in `process_release'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:47:in `block in perform'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/lock_helper.rb:47:in `block in with_release_lock'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/lock.rb:58:in `lock'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/lock_helper.rb:47:in `with_release_lock'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/update_release.rb:47:in `perform'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/job_runner.rb:98:in `perform_job'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/job_runner.rb:29:in `block in run'/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.3/lib/common/thread_formatter.rb:46:in `with_thread_name'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/job_runner.rb:29:in `run'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/jobs/base_job.rb:8:in `perform'/var/vcap/packages/director/gem_home/gems/resque-1.23.1/lib/resque/job.rb:125:in `perform'/var/vcap/packages/director/gem_home/gems/resque-1.23.1/lib/resque/worker.rb:186:in `perform'/var/vcap/packages/director/gem_home/gems/resque-1.23.1/lib/resque/worker.rb:149:in `block in work'/var/vcap/packages/director/gem_home/gems/resque-1.23.1/lib/resque/worker.rb:128:in `loop'/var/vcap/packages/director/gem_home/gems/resque-1.23.1/lib/resque/worker.rb:128:in `work'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/bin/worker:74:in `<top (required)>'/var/vcap/packages/director/bin/worker:23:in `load'/var/vcap/packages/director/bin/worker:23:in `<main>'

初步分析，Ruby的package之所以SKIP，是因为bosh检测到系统中之前以上传到同版本的package，所以跳过该包的上传，但是为什么后边加载这个package的时候找不到了呢？苦苦思索，还是无法确定问题发生原因；再分析package上传后是存储在虚拟机blobstore之中的，但是上传信息是存于别的位置，检查blobstore状态，running状态，正常，突然想到，之前因为网络故障，重启过blobstore虚拟机，检查发现var/vcap/store挂载的/dev/vda消失，发现问题，解决如下：

1、重新挂载/dev/vda2到/var/vcap/store下

2、配置blobstore虚拟机开机挂载/dev/vda2

再次执行upload命令，一切正常。

3.3 部署CloudFoundry services时发生的错误

部署Services时，因为需要创建大量的虚拟机、创建大量的Volumes，所以可能会因为OpenStack环境本身的Quota限制引起一些错误，下边这个错误是因为Cinder本身初始配额为10.当创建的volume超过10的时候，就会发生错误：

Bosh中的报错信息：

E, [2013-08-22T08:45:38.099667 #8647] [task:323] ERROR -- : OpenStack API Request Entity Too Large error. Check task debug log for details./var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.3/lib/cloud/openstack/helpers.rb:20:in `cloud_error'/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.3/lib/cloud/openstack/helpers.rb:39:in `rescue in with_openstack'/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.3/lib/cloud/openstack/helpers.rb:25:in `with_openstack'/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.3/lib/cloud/openstack/cloud.rb:361:in `block in create_disk'/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.3/lib/common/thread_formatter.rb:46:in `with_thread_name'/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.3/lib/cloud/openstack/cloud.rb:342:in `create_disk'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/instance_updater.rb:377:in `block in update_persistent_disk'/var/vcap/packages/director/gem_home/gems/sequel-3.43.0/lib/sequel/database/query.rb:338:in `_transaction'/var/vcap/packages/director/gem_home/gems/sequel-3.43.0/lib/sequel/database/query.rb:300:in `block in transaction'/var/vcap/packages/director/gem_home/gems/sequel-3.43.0/lib/sequel/database/connecting.rb:236:in `block in synchronize'/var/vcap/packages/director/gem_home/gems/sequel-3.43.0/lib/sequel/connection_pool/threaded.rb:104:in `hold'/var/vcap/packages/director/gem_home/gems/sequel-3.43.0/lib/sequel/database/connecting.rb:236:in `synchronize'/var/vcap/packages/director/gem_home/gems/sequel-3.43.0/lib/sequel/database/query.rb:293:in `transaction'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/instance_updater.rb:376:in `update_persistent_disk'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/instance_updater.rb:73:in `block in update'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/instance_updater.rb:39:in `step'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/instance_updater.rb:73:in `update'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/job_updater.rb:63:in `block (5 levels) in update'/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.3/lib/common/thread_formatter.rb:46:in `with_thread_name'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/job_updater.rb:60:in `block (4 levels) in update'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/event_log.rb:58:in `track'/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.3/lib/director/job_updater.rb:59:in `block (3 levels) in update'/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.3/lib/common/thread_pool.rb:83:in `call'/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.3/lib/common/thread_pool.rb:83:in `block (2 levels) in create_thread'/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.3/lib/common/thread_pool.rb:67:in `loop'/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.3/lib/common/thread_pool.rb:67:in `block in create_thread'

OpenStack中的报错信息：

2013-08-22 16:44:57 ERROR nova.api.openstack [req-75ffcfe3-34ea-4e8d-ab5f-685a890d4378 57be829ed997455f9600a4f46f7dbbef 8cf196acd0494fb0bc8d04e47ff77893] Caught error: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded (HTTP 413) (Request-ID: req-30e6e7d6-313d-46b1-9522-b3b20dd3e2ab)2013-08-22 16:44:57 24345 TRACE nova.api.openstack Traceback (most recent call last):2013-08-22 16:44:57 24345 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 78, in __call__2013-08-22 16:44:57 24345 TRACE nova.api.openstack     return req.get_response(self.application)......2013-08-22 16:44:57 24345 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 109, in request2013-08-22 16:44:57 24345 TRACE nova.api.openstack     raise exceptions.from_response(resp, body)2013-08-22 16:44:57 24345 TRACE nova.api.openstack OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded (HTTP 413) (Request-ID: req-30e6e7d6-313d-46b1-9522-b3b20dd3e2ab)2013-08-22 16:44:57 24345 TRACE nova.api.openstack

解决办法：修改Cinder的quota配额，并重启cinder服务：

cinder quota-update <tenant_id> --volumes 20

cd /etc/init.d/; for i in $( ls cinder-* ); do sudo service $i restart; done

问题解决