读书笔记之Black Hats Manual Software Security Auditing, Cracking, Debugging

Black Hats Manual 
Software Security 
Auditing, Cracking, Debugging

经典的关于如何写出security程序的书

http://www.lst.de/~okir/blackhats/

以下是整理该书中我认为比较重要的东西。

1. Buffer Overflows and Other Memory Problems
   Solution: Non-executable Stack
   
   telling the CPU via page protection flags that it's not allowed to execute instructions from the stack
   但是，hacker们会把exec得程序放在其它内存区域，比如static buffer，（如DNS所使用的保存store DNS responses）
   

   或者利用Jump into libc

   

   where addr1 is the address of the system() function. Note that this works even if the program you attack does not use system() at all; simply by mapping the libc shared object file, the library function is within the attacked process' address space. 
   
   
   如何预防buffer overflow:
   
   • Always check memory copy operations to make sure you're within the bounds of the destination buffer. When writing a function that is supposed to copy data to a buffer supplied by a caller, always include the size of the buffer, and make your function check that size when copying data to that buffer.
   • Use functions that do bounds checking. That means, use strncpy rather than strcpy, and snprintf rather than sprintf.^2.2 Avoid strcat, or at least make sure there's enough room in your buffer.

     Note that strncpy does not copy the terminating NUL byte if the destination buffer is too small. The proper way of using it goes like this:

     char buffer[BUFSZ];
     
     strncpy(buffer, somestring, sizeof(buffer)-1);
     buffer[sizeof(buffer)-1] = '\0';
     
   • When referencing the size of some buffer, always use the sizeof operator. In the example above, we could also have used BUFSZ instead - but what if we change the definition of buffer later and make it MAXFOOBYTES large, and forget to change the strncpy call? Using sizeof takes care of this nicely.
   • If you have complicated code, e.g. constructing a path name dynamically, don't try to count characters; instead, compare your pointer to the buffer end address. This check will always be accurate, while it's easy to lose track of how much room is really left in a buffer once the code gets complicated.^2.3 For example:char buffer[BUFSZ], *sp, *end;
     
     sp = buffer;
     end = buffer + sizeof(buffer);
     while (complicated condition) {
     more complicated stuff
     /* Now try to append string to buffer. Note that
     * the >= comparison takes care of the NUL byte at the
     * end of the appended string: */
     if (sp + strlen(string) >= end)
     return -1; /* too bad, not enough room */
     strcpy(sp, string);
     sp += strlen(string);
     }
     
   • If you find yourself doing a certain memory or string operation over and over again, try to find a suitable abstraction (preferably one that's simple!) and implement it once, or use one that's freely available. Typical examples of such abstractions may be strings,^2.4 lists, and read/write buffers (useful for networking code).

     Admittedly, this is something every book on software design tells you on page three. But in a security context, it nevertheless bears repeating because if you have the code once, you will be able to go over it with a fine tooth comb, looking for problems such as buffer overflows. If you have the same code scattered all over your project, in several copies, the motivation to do so (and the concentration with which you perform this) may be less, resulting in less secure code.

   • Look for debugging code, and double check it. A surprising number of very silly security problems (and buffer overflows involving sprintf) have been found in functions intended for debugging purposes. Apparently, we programmers tend to forget such code when looking for security holes.
   • There's also a school of coding that, instead of doing proper bounds checking, just makes all buffers ``large enough''. For instance, if the maximum amount of data accepted from the user is 1024 bytes, just make sure all buffers are 1024 bytes plus a sufficiently large epsilon. There are several programs that use this kind of approach; for some, it works surprisingly well (ssh is one of them), for others, this fails miserably. I do not recommend using this approach.
   
   
   
   

   使用StackGuard的技术
   
   

   待补充？？？
   
   

   使用libsafe
   

   Libsafe(created by AT&T) currently works on Linux only, but the code should be portable to other platforms as long as it uses the GNU compiler gcc, because it relies heavily on 
   some of its features. Libsafe must be compiled as a shared library
   原理：frame point + frame point&return adress(两者紧挨着)
   

   所以，If we combine these two, we have a rough outline of libsafe's mechanisms. Consider the libsafe implementation of memcpy, which is invoked with a destination buffer dst, a source buffer src, and the amount of data to copy, len. First, libsafe walks the chain of frame pointers until it finds one that is greater than the dst address: this is the stack frame of the function that declared the dst buffer. All it needs to do now is check whether dst+len exceeds the frame pointer address: if that's the case, then the memcpy operation is definitely a buffer overflow, and the program is terminated. 
   
   

   in order to provide effective protection with libsafe, you have to make sure that your application, as well as all the libraries you use, are compiled with frame pointer support. 
   
   

   其它的一些memory overflow发生的地方：详细可以参见，The art of hacking 一书。
   

   Heap memory corruption
   Single byte overflow
   Format bugs
   
   
   
   
2. Setuid applications
    http://www.aquaphoenix.com/ref/gnu_c_library/libc_439.html
   http://nob.cs.ucdavis.edu/bishop/secprog/1987-sproglogin.pdf
   
   找到SUID files： find / -type f -perm -04000 -ls
   
   Linux and HP-UX users can use Bastille (http://www.bastille-linux.org), a fantastic
   hardening tool from Jay Beale. Bastille will harden their system against many of the
   aforementioned local attacks, especially to help remove the SUID from various files.
   
   
   Close Open files
   
   方法一：
   

   Usually, it's best to close all open file descriptors except the standard ones (standard input, output and error) attached to descriptors 0, 1 and 2, respectively. The following code tries to do that in a portable way:

   int fd;
   
   #ifdef OPEN_MAX
   fd = OPEN_MAX;
   #elif defined(NOFILE)
   fd = NOFILE;
   #else
   fd = getdtablesize();
   #endif
   while (fd-- > 2)
   close(fd);
   方法二：
   

   The close-on-exec flag tells the kernel to close the file when executing another program. The purpose of this flag is exactly to prevent the kind of information leak described above. Assume you've opened a file using fopen, then you can set the flag like this:

   void set_cloexec(FILE *fp)
   {
   fcntl(fileno(fp), F_SETFD, FD_CLOEXEC);
   }
   
   
   Dropping Privileges with popen
   
   popen相当于以下的实现：
   we use this to read from a sub-process. First, the pipe is created, then we fork. After forking, the parent process closes the writing side of the pipe, and attaches the read file descriptor to a FILE object.

   The child process, on the other hand, closes the reading end of the pipe, and attaches the writing side to its standard output and standard error (file descriptors 1 and 2) using the dup2 system call. It then proceeds to drop privileges as described above, and execute the external program.

   int fds[2], pid;
   
   if (pipe(fds) < 0) {
   perror("pipe() failed");
   return NULL;
   }
   if ((pid = fork()) < 0) {
   perror("fork failed");
   close(fds[0]); close(fds[1]);
   return NULL;
   }
   if (pid != 0) {
   /* parent process: return FILE pointer for reading
   * from sub-process */
   close(fds[1]);
   return fdopen(fds[0], "r");
   }
   
   close(fds[0]);
   dup2(fds[1], 1); /* attach pipe to stdout ... */
   dup2(fds[1], 2); /* ... and stderr */
   
   /* drop privilege and run command as above */
   
   
   Clean up your Environment
   

   It is therefore important that whenever you execute another program from a setuid application, you should clean up the environment. Be sure that you take a whitelist approach that passes only known harmless variables instead of blacklisting variables that are known to be dangerous - you might miss one that turns out to open a huge hole.

   The following code runs the slip-up script with a predefined environment, except for the LANG variable which is copied from the caller's environment. The crucial part here is the use of execve rather than execv or execl. The latter invoke the sub-process with the current environment (which might have been poisoned by the user), while execve takes an array of name=value pairs and passes that as the sub-commands environment.

   
   Do not trust argv[0]
   
   As a matter of fact, argv[0] and the name of the program being executed are not related in any way. There is a convention that argv[0] should be the name of the program - but nothing in the whole wide Unix world enforces that convention.
   
   
   Do not trust file descriptors 0,1,2
   

   You should therefore make it a habit to ensure that file descriptors 0, 1, and 2 are valid, right at the start of your main procedure and before you do anything else. One common approach is:

   while (1) {
   

   int fd;
   
   fd = open("/dev/null", O_RDWR);
   if (fd < 0)
   

   exit(EX_OSERR);
   

   if (fd > 2) {
   

   close(fd);
   

   break;
   

   }
   

   }
   
   
   Double check strings used as command line arguments
   
   parse its arguments is to use the option separator --. This separator tells getopt to stop scanning for options, and treat the remaining words on the command line as plain arguments, even if they start with a dash.
   
   
   Do not trust signal
   
   
   
   Linux引入了一个叫做File sytem uid的概念：
   
   The main reason for introducing the fsuid was the case of network daemons that want to temporarily ``become'' a certain user when opening a file, etc. If the daemon changes its effective uid, it also allows that user to send it arbitrary signals, allowing the user to kill this daemon. Normally, this is not a huge problem. Sendmail, for instance, sets its effective uid when opening a user's .forward file. If a user sends this sendmail process a KILL signal all he achieves is disrupt his own mail delivery; the main sendmail daemon continues to run. In fact, I know only one network service that has ever really used the setfsuid system call, the Linux user space NFS daemon (and to some extent, setfsuid was created because of it).
   
   
   
3. Working with Temporary Files
   
   两种基本的错误：
   

   The program opens a file where the path name contains a ``hostile'' dirtectory. Hostile meaning that other users (read: potential attackers) can modify the contents of this directory. 
   
   The program blindly follows a (symbolic) link. It's worth mentioning that while most published exploits use symbolic links, hard links will work sometimes, too.
   

   一般都是利用race condition，比如一些set uid的程序的某些函数调用之间的空隙（比如lstat和fopen），因为不是原子的，可能导致其它的hacking进程有可能在两者之间执行，从而绕过一些简单的测试。
   
   

   In addition to this, the problem is very wide spread, and hard to eradicate. When the developers of the GNU C compiler recently added code that warns you if your program uses a known dangerous function such as mktemp, some 50 or 60 packages in OpenLinux triggered these warnings when recompiled! In the three or four weeks of non-stop bugfixing that ensued, morale of our security team reached an all time low, as you can probably imagine:-)注意：在使用各种C library的时候，请尽可能去了解他们的一些局限性或者说是跟其它的函数一起使用的时候，有哪些容易出现问题的地方。
   

   
   

   解决办法：
   Create a one-shot file(mkstemp, tmpfile)，同时注意提供的一些额外的可以增加安全的参数（O_NOFOLLOW等）
   

   关于sticky bit, after you've safely created your temporary file, you can re-open it many times if it resides in a directory like /tmp that has the sticky bit set. The sticky bit guarantees that even though the directory is writable by everyone, a file can only be removed by the user who created it. That means, once you've created the file, an attacker is unable to replace it with a symbolic link.

   
   
   Exchanging data via a temporary file
   Unix sockets and Named pipes
   

   容易被hacking通过设置权限等手段阻止正常使用或者是读取应该受保护的文件。There's an object in a hostile directory they want to access, but they need to make sure they cannot be fooled into following symlinks, hard links, or talking to a process they shouldn't be trusting.

   解决办法：
   

   The best cure I know is to check the file's owner using lstat. Consider the following code:

   snprintf(filename, sizeof(filename),
   "/tmp/krb5cc_%u", getuid());
   if (lstat(filename, &stb) >= 0) {
   /* Bail out if it's ... */
   if (!S_ISREG(stb.st_mode) /* ... not a regular file */
   || stb.st_uid != getuid() /* ... not owned by us */
   || stb.st_nlink != 1) /* or a hard link */
   fatal("Argh, spoofed temp file!");
   fd = open(filename, O_RDWR);
   } else {
   fd = open(filename, O_RDWR|O_CREAT|O_EXCL, 0600);
   }
   if (fd < 0)
   fatal("Cannot open %s: %s", filename, strerror(errno));
   

   At first glance, this looks like the typical race condition I was ranting about above. Of course, many things can happen between the lstat call and the open call. But in this case, the attacker will not be able to replace the file with something else because of the sticky bit on /tmp! If the information returned by lstat tells us the file is owned by us, we know we're safe because the attacker cannot remove whatever we just looked at.

   Note that the code snippet above works for everything except setuid applications; in a setuid application getuid will return the user ID of the user who invoked the program, i.e. the (potential) attacker.
   
   使用一个private的目录
   

   strcpy(dirname, "/tmp/fooXXXXXX");
   if (mktemp(dirname) == NULL
   || mkdir(dirname, 0700) < 0)
   fatal("Unable to create temporary directory: %m");
   putenv("TMPDIR", dirname);
   
   
   
   
   
   
    
4. Network Applications
       对于Fake DNS的简单check:the solution is to take the host name returned by the reverse lookup, do a forward lookup on it, and check whether the client's IP address shows up in the list of addresses associated with that name. If it doesn't, something's definitely fishy, and the name should not be trusted.^6.8 Figure 6.1 shows how to do a ``safer'' reverse lookup using this approach.
   
   Access control
   

   To make a long story short: the most watertight way of enforcing that certain requests can only be sent by local processes is to use a second socket (often called control socket because it's dedicated to special control tasks), and restrict who can connect to it. With UDP and TCP sockets, you can do this by binding the socket to address 127.0.0.1 and port xyz. Then, a remote attacker can send packets to port xyz on your machine as much as he might - they won't be delivered to your service because the control socket will only accept packets with a destination address of 127.0.0.1.

   An alternative solution is to use AF_LOCAL (aka AF_UNIX) sockets. This is a variety of sockets that work only locally, they are not accessible over the network. On several BSD variants and Linux, these sockets will even tell your application which user sent the request, allowing much more fine-grained access control!
   
   
   DNS name size
       The bottom line is: always make sure host names returned by gethostbyaddr and friends actually fit into your buffer, and reject the name if not. 
   

        Therefore, it is a good idea to always make sure that the address information returned in a struct hostent is actually what you expect it is by doing something like this:

   if (!(hp = gethostbyname(hostname)))
   fatal("unable to resolve hostname");
   if (hp->h_addrtype != AF_INET || hp->h_length != 4)
   fatal("bad address data from DNS!");
   memcpy(&sin.sin_addr, hp->h_addr_list[0], 4);
   
   
    

5. Presentation Layer Issues
   
   these bugs are caused by mixing up signed and unsigned integer handling, they are frequently referred to as signedness bugs. 
   
   
   Data size issuesNever assume that variable length data items received from the network fit into certain buffers just because that's what they should do. If you're using a programming language such as C, always NUL-terminate strings extracted from the packet.

   Signedness problemsWhen extracting counted items (strings, arrays, etc) from a packet, either explicitly make sure that the count does not become negative, or always use unsigned variables.Word size problemsAny checks that have a bearing on the application's security should be made after converting data to the canonical host format. For instance, user IDs should be assigned to a variable of type uid_t, and file names from HTTP requests should be unquoted exactly once.Packet boundary issuesIt is a good idea to have a set of function that deal with ``network buffers.'' You can verify the correctness of these functions a lot more easily than by going through all the packet handling code line by line. 
   
   
   
6. Application Layer issues
   Buffer overflow
   
   There are several ways in which you can protect yourself from this type of problem. One, which is really what I recommend, is to always make sure that whenever you copy a string received from the network to another buffer, you explicitly limit the amount of data copied, using functions such as strncpy and snprintf. 
   A different solution is to have a string handling library do all the string operations, and review the library for buffer overflows. one such library is Dan Bernstein's XXXlibrary.
   
   Yet another approach is to make sure that all buffers are large enough to hold whatever string you receive from the client. Note that an attacker can of course send arbitrarily long strings,
   but if your application does not accept network packets longer than say 2048 bytes, then you know no client supplied string argument can be longer than that.
   
   
   Dangerous Characters in Strings
   
   Embedding in structured files
   
   Embedding in HTML/XML  
   

   you should make sure that any input you receive from the client is sanitized before you embed it in XML documents(but after any HTTP unquoting). The most common approach is to quote the special characters <, > and & as &lt; &gt; and &amp; ... ,respectively.
   avoid the shell in general like the plague, and system and popen in particular! As mentioned above, the Right Thing is to use the fork and exec system calls. It's a bit of extra coding, but it is definitely more secure as it doesn't involve the shell anymore.
   
   

   Passing strings to the shell, pay more attention on single quotes, it will cause shell do some thing you wouldn't want. especially, when you get a url in which including evil strings wrapped by single quotes.
   
   

   int
   sane_url(const char *url)
   {
   int n;
   
   if (strncasecmp(url, "http:", 5)
   && strncasecmp(url, "https:", 6)
   && strncasecmp(url, "ftp:", 4))
   return 0;
   
   /* Be extra nasty: restrict the set of characters we accept.
   * Note that we can accept shell meta characters because we
   * use exec() rather than system() to run the command. */
   while (*url) {
   if (!isalpha(*url)
   && !isdigit(*url)
   && !strchr(".,?=;", *url))
   return 0;
   }
   return 1;
   }
   

   
   Using strings in path names, whenever you use a string supplied by an untrusted peer, make sure to check that it contains no .. path component. If you're worried that legal requests might include file names containing two adjacent dots, check for /.. and ../
   

   FILE *
   open_userfile(const char *name)
   {
   char pathname[1024];
   
   if (strstr(name, "../") || strstr(name, "/.."))
   return NULL;
   snprintf(pathname, sizeof(pathname), "%s/%s",
   BASEDIRECTORY, name);
   return fopen(pathname, "r");
   }
   
   

   Embedding in database queries
   
   Console output -- so-called control sequences
   
   Feeding data to external programs
   
   
   
   Passing Arguments through environment variables, Therefore, when you pass data through environment variables, either use fixed variable names, or if you need to support a flexible naming scheme, use a common prefix that doesn't collide with system variables. For example, the CGI mechanism prefixes all variables with HTTP_ for this reason. 
   
   
   
   Properly Dropping privilege
   
   the proper way to drop privilege in a server is this:pwd = getpwnam(username);
   if (pwd == NULL)
   fatal("user not found: %s", username);
   if (initgroups(username, pwd->pw_gid) < 0
   || setgid(pwd->pw_gid) < 0
   || setuid(pwd->pw_uid) < 0)
   fatal("unable to drop privilege: %m");
   

   If you don't care about supplementary groups, you can clear the list of groups altogether by using setgroups(0, NULL).

   If your server is also supposed to change its root directory, the call to chroot must happen after initgroups() and before setuid().

   
   
7. Denial Of Service
   Fork Bombs
   
   
   
   Memory hogs
   
   for globbing problem, really fixes is to put an upper limit of 100 or 200 on the number of names returned by the glob function.
   
   File system denial of service
   Saturating the network
   

   There are several defenses against these attacks. One is administrative; which is to configure the firewall to drop all packets sent to the broadcast address of some (sub-)network. Which is admittedly tedious, because the firewall admin needs to be involved every time an existing network is split up into two smaller subnets (because this creates a new broadcast address). However, this is the only defense against the smurf attack. Today, there are even several ``research'' projects on the net that regularly scan large portions of the Internet for these ``open multiplier'' networks, and notify the administrative contacts of those found vulnerable.

   An application should protect itself against the UDP echo attack by ignoring any UDP packets from ports below 512. Applications normally do not send packets from such a low port unless they explicitly bind to it. However, ports in this range are reserved for network servers. And coincidentally, all known services that are prone to such ping-pong effects live in that range. This is how inetd's builtin services such as echo, chargen and time defend themselves against the echo attack today. They simply drop any UDP packets from port numbers less than 512.

   Reverse DNS Lookups
   
   for multi-threaded services that create separate processes for ezch connection, another solution is to perform the address to name lookup in the sub-processes. Now if the client's DNS hangs, the only one getting hurt is the client. Processing for everyone else proceeds just as usual.
   
   Concurrency Problems
   
   Unfortunately, the RPC code in most Unix implementations doesn't attempt to address any of these concurrency problems. They can be worked around, but only at a considerable cost in terms of code complexity (you basically need to re-implement much of the network handling). So if you need a network server that cannot be incapacitated that easily, don't use RPC.
   
   
8. New Solutions
   Emily's coding corner
   
   always check return values
   Fail-open vs. fail-closed
   Don't do memory bugs
   
   
   Least privilege
   
   You should be careful with the nobody uid however. This is a very special uid that should not be used too much. Assume you've managed to modify all your network services so that they run under the nobody account. Now if an attacker manages to break one of your services, all it gives him is nobody privilege. However with all those network processes running under the same uid, this becomes a powerful tool because he can now attach to these processes using ptrace(2), modify their memory, etc. So if you have modified a network server to run with reduced privilege, think about creating a new system user specifically for this service rather than overloading nobody.
   Avoid setuid if a group does the trick
   Use setgid rather than setuid
   

   Does this make any difference? Yes, there's a small but important one. Assume you've made it setuid, and there's a bug in it that gives the attacker uid uucp. The program he attacked is owned by uucp as well, so he can replace it with a modified version that behaves just like the original one,^11.2 but ``steals'' the uids of all users who invoke the program. Who knows, maybe one day the super user will come along and run this utility?

   This can be avoided by making the program setgid uucp and giving it mode 555 permissions. An attacker cracking this program will not be able to modify the binary because all he gets out of cracking it is group uucp privilege.

   Don't fix it, rewrite it
   Helper programs
   
   One solution that works in a fair number of cases is to put that needs increased privilege and put that into a setuid helper program. Then, the entire application can be installed without setuid bits, and you only have to ensure that those few hundred lines of your helper application are secure.
   
   The fork approach
   Unix socket magic 以上两者都有参考代码，值得借鉴！！！
   Using chroot jails 
   
   this is often referred to as a jail.^11.8 However, a chroot environment does not provide perfect insulation; processes within the jail are still able to send signals to, and even ptrace other processes that run under the same uid. 
   

   The first issue is that after doing the chroot call, make sure to call chdir(/).
   The second issue is that, for the root user, it is trivial to break free from a chroot jail. So what you have to do is drop privilege, but only after doing the chroot call because that requires root privilege.
   
   Note that there's no 100% guarantee that non-root users are able to break out of a chroot jail. For instance, earlier implementations of the Linux /proc filesystem, when mounted below a chroot area, allowed any user to break free simply by doing a cd /proc/1/cwd. Also, if a process is able to obtain an open file descriptor to a directory outside the chroot area, it can escape by calling an fchdir with that descriptor. 
   

   Using tcpd style access control
   

   In case you're not familiar with it, tcpd is a wrapper for TCP servers, written by Wietse Venema, which performs access control checks based on the client's IP address, and other information. tcpd can be used by services started from inetd by making inetd start the wrapper first, which does the access checks, and then executes the real server program if the client is allowed to connect.
   

   The first is that the current implementation is somewhat slow.
   The second is that sometimes the sender's IP address cannot be relied upon. Similarly, you cannot always trust DNS information. This problem is not specific to Wietse's code, it's a general one.
   

   Using Capabilities
   CapabilityDescriptionCAP_CHOWNChange owner of any fileCAP_DAC_OVERRIDEOverride any file permissions (DAC = Discretionary Access Control)CAP_KILLSend signals to any processCAP_NET_BIND_SERVICEBind to any UDP/TCP port below 1024CAP_NET_RAWOpen raw and packet socketsCAP_IPC_LOCKLock shared memory segmentsCAP_SYS_RAWIOAllow ioperm and iopl (used to access e.g. hardware registers)CAP_SYS_CHROOTAllow use of chrootCAP_SYS_TIME

   Allow setting the system clock

   
   Using cryptography
   

   do not try to invent your own cypto
   use standard libraries
   known what your're doing
   
   
   DNS reverse lookup explained