Sysdig - 内核层的container 监控工具

原文 https://sysdigcloud.com/let-light-sysdig-adds-container-visibility/

Container Disk I/O Visibility with Sysdig

At this point, you should know the drill: this command line lets you see the top containers in terms of file I/O:

$ sudo sysdig -c topcontainers_fileBytes           container.name  --------------------------------------------------------------------6.79KB          mysql4.11KB          haproxy2.13KB          wordpress42.13KB          wordpress12.13KB          wordpress32.13KB          wordpress21.64KB          client

This command line shows the top processes in terms of file I/O, and tells you which container they belong to:

$ sudo sysdig -pc -c topprocs_fileBytes           Process         Host_pid        Container_pid   container.name  --------------------------------------------------------------------------------63.21KB         mysqld          11126           83              mysql29.25KB         curl            47169           11400           client29.25KB         curl            47167           11398           client29.25KB         curl            47165           11396           client2.07KB          docker          1746            11610           host1.37KB          rs:main         959             963             host832B            sleep           47170           11401           client832B            sleep           47168           11399           client832B            sleep           47166           11397           client832B            sleep           47164           11395           client

This command line shows the top files in terms of file I/O, and tells you which container they belong to:

$ sudo sysdig -pc -c topfiles_bytesBytes           container.name  Filename        --------------------------------------------------------------------------------63.21KB         mysql           /tmp/#sql_1_0.MYI6.50KB          client          /lib/x86_64-linux-gnu/libc.so.63.25KB          client          /lib/x86_64-linux-gnu/libpthread.so.03.25KB          client          /lib/x86_64-linux-gnu/libgcrypt.so.113.25KB          client          /usr/lib/x86_64-linux-gnu/libwind.so.03.25KB          client          /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.23.25KB          client          /usr/lib/x86_64-linux-gnu/liblber-2.4.so.23.25KB          client          /lib/x86_64-linux-gnu/libssl.so.1.0.03.25KB          client          /usr/lib/x86_64-linux-gnu/libheimbase.so.13.25KB          client          /lib/x86_64-linux-gnu/libcrypt.so.1

Let me mention another couple of useful things concerning disk I/O that you can do with sysdig.

Number one: observing activity on a file with a given name, no matter which container the file belongs to. For example, this command line shows all the data that is written to /etc/passwd,no matter which container this is happening in:

$ sudo sysdig -pc -c echo_fds "fd.name=/etc/passwd"------ Read 956B from  [mysql] [7a19ca8a7555]  /etc/passwd (cat)root:x:0:0:root:/root:/bin/bash.daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin.bin:x:2:2:bin:/bin:/usr/sbin/nologin.sys:x:3:3:sys:/dev:/usr/sbin/nologin.sync:x:4:65534:sync:/bin:/bin/sync.games:x:5:60:games:/usr/games:/usr/sbin/nologin.man:x:6:12:man:/var/cache/man:...

And sysdig is nice enough to tell us the name of the container and the process inside it that are performing the I/O operation.

Number two: the famous sysdig spectrogram chisel now understands containers! Here’s how to produce the spectrogram of the file I/O activity of the mysql container:

$ sudo sysdig -c spectrogram fd.type=file and container.name=mysql

and here’s the result:

Remember, this works from the host OS or from another container. Pretty powerful.

Sysdig Container Bonuses: User Tracking, Logs, and Request Flow Between Containers

Almost done! Thanks for sticking with me this far ;) I want to close this post with a few additional sysdig tricks that, hopefully, will make you more productive when managing containers.

1. Observing User Activity Inside a Container

For an attacker, a container is just another target machine. As a consequence, detecting suspicious activity and rogue processes inside containers is as important as it is for physical and virtual machines. Fortunately, sysdig can help. In particular, the spy_users chisel captures every command that is executed in the system, inside or outside a container, and prints it on screen:

$ sudo sysdig# ./sysdig -pc -c spy_users43760 13:35:18 root@client) sleep 0.143760 13:35:18 root@client) curl 172.17.0.743760 13:35:18 root@client) sleep 0.143760 13:35:18 root@client) curl 172.17.0.743760 13:35:19 root@client) sleep 0.143760 13:35:19 root@client) curl 172.17.0.743760 13:35:19 root@client) sleep 0.1

Note how the user and container name are printed for every command line. You can of course narrow this down to the commands executed inside a specific container – it’s just a matter of using a filter:

$ sudo sysdig# ./sysdig -pc -c spy_users container.name=mysql

2. Tailing Logs from Multiple Containers

The Docker guidelines suggest that containerized apps write their logs to stdout, so thatdocker logs can be used to watch the logs generated by a container. Sometimes, however, you will be stuck with a non compliant app that writes its logs somewhere in the file system. Would you like to see those logs without having to attach to the container? Use this command line:

$ sudo sysdig -pc -cspy_logswordpress3 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:49:00 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"wordpress4 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:49:01 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"wordpress1 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:49:01 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"wordpress2 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:49:01 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"wordpress3 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:49:01 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"wordpress4 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:49:01 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"wordpress1 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:49:02 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"

The spy_logs chisel shows all the I/O buffers that are written to files ending in .log or _log, inside or outside containers. The first entry in each line is the container name. The log above, for example, clearly shows how the load balancer is operating well: requests are routed to a different container every time, in FIFO order. Of course, we can also observe the logs generated inside a specific container. We do that by appending a filter to the command line:

$ sudo sysdig -pc -cspy_logs container.name=wordpress1wordpress1 apache2 /var/log/apache2/access.log 172.17.0.7 - - [21/Feb/2015:21:55:18 +0000] \"GET / HTTP/1.1\" 302 346 \"-\" \"curl/7.35.0\"wordpress1 apache2 /var/log/apache2/access.log 172.17.0.7 - -

3. Observing The Flow of Application Requests Across Containers

One of the hardest things to do when troubleshooting container-based infrastructures is observing the data that containers are exchanging. This is critical when trying to identify things like application bottlenecks, and it is essentially impossible to do today. Here’s how to do it with sysdig:

$ sudo sysdig -pc -A -c echo_fds "fd.ip=172.17.0.3 and fd.ip=172.17.0.7"------ Write 103B to  [haproxy] [d468ee81543a]  172.17.0.7:37557->172.17.0.3:80 (haproxy)GET / HTTP/1.1User-Agent: curl/7.35.0Host: 172.17.0.7Accept: */*X-Forwarded-For: 172.17.0.8------ Read 103B from  [wordpress1] [12b8c6a04031]  172.17.0.7:37557->172.17.0.3:80 (apache2)GET / HTTP/1.1User-Agent: curl/7.35.0Host: 172.17.0.7Accept: */*X-Forwarded-For: 172.17.0.8------ Write 346B to  [wordpress1] [12b8c6a04031]  172.17.0.7:37557->172.17.0.3:80 (apache2)HTTP/1.1 302 FoundDate: Sat, 21 Feb 2015 22:19:18 GMTServer: Apache/2.4.10 (Debian) PHP/5.6.6X-Powered-By: PHP/5.6.6Expires: Wed, 11 Jan 1984 05:00:00 GMTCache-Control: no-cache, must-revalidate, max-age=0Pragma: no-cacheLocation: http://172.17.0.7/wp-admin/install.phpContent-Length: 0Content-Type: text/html; charset=UTF-8------ Read 346B from  [haproxy] [d468ee81543a]  172.17.0.7:37557->172.17.0.3:80 (haproxy)HTTP/1.1 302 FoundDate: Sat, 21 Feb 2015 22:19:18 GMTServer: Apache/2.4.10 (Debian) PHP/5.6.6X-Powered-By: PHP/5.6.6Expires: Wed, 11 Jan 1984 05:00:00 GMTCache-Control: no-cache, must-revalidate, max-age=0Pragma: no-cacheLocation: http://172.17.0.7/wp-admin/install.phpContent-Length: 0Content-Type: text/html; charset=UTF-8

What does this do? The filter in the command line isolates the traffic between two containers, haproxy (whose IP is 172.17.0.7) and wordpress1 (whose IP is 172.17.0.3). Notice how we can see both ends of the conversation in a single stream, with no noise coming from other connections. And let me point out one more time: this is done without having to attach to the containers and/or install anything in them.

Conclusion

The title of this blog post is let there be light. This is what I thought when I first started using the sysdig container functionality as we were developing it. Here at sysdig we arein love with containers, but after having worked in the visibility space for many years, we know that containers won’t succeed without state of the art monitoring and troubleshooting. We believe what we’re showing in this blog post is a step in that direction, and we sincerely hope that it can make container adoption less painful and even more fun.

Of course there’s tons of space for improvement, so I encourage you to install sysdig (it’s a matter of 30 seconds!), and give us feedback on Twitter or github. During the next weeks, we’ll publish more technical blog posts that will cover deployment (including environments like CoreOS), better visualization of this information and other advanced topics.

Also, I expect you are wondering if there’s a way to get this goodness for distributed container-based infrastructures. That is exactly the goal ofSysdig Cloud, our SaaS offering, which is in open beta and which is adding container functionality starting today.Give it a try, and you could get visualizations like this one in a matter of minutes: