Monitoring and Troubleshooting Using Event Logs

原文: http://www.windowsnetworking.com/articles_tutorials/Monitoring-Troubleshooting-Event-Logs.html

This article reviews best practices for working with Windows event logs including how to interpret event messages, how to configure event logs, how to search and filter events, how to view events on remote systems, and how to use EventCombMT.exe and other tools to monitor events on multiple systems.

The event logs on Windows systems are helpful for both troubleshooting when things go wrong and monitoring performance and behavior. An event log is a file that contains events, which are entries to the log that notify the user of some occurrence relating to the operating system or applications running on the system. An event includes information about the type of occurrence, the date and time when it occurred, the computer where it happened and the user who was logged on at the time, and other information such as event ID, the event category, and the source of the event. Events may also include further detailed information concerning the event and possibly a link to where more information can be found. Figure 1 below illustrates an example of an event from the DNS Server event log on a Windows Server 2003 domain controller:

Figure 1: Example of an event

 

Finding More Information About an Event

 

If an event contains a link and you click on it, a dialog box opens warning you that information about the event will be sent to Microsoft to see if they have more information available concerning the event:

Figure 2: Sending event information to Microsoft.

Clicking Yes opens the Help and Support Center and checks to see if there is any more information about the event that may be helpful. Figure 3 shows a typical response:

Figure 3: Additional help concerning the event.

How many times have you been frustrated by the lack of helpful information available this way concerning some obscure event? In the example above, the additional help provided is that “this error could be caused by either a high load on the domain controller or the failure of other domain controller services” and the suggested remedy is to “restart the DNS Server service” and check the event log for anything else that happened at the same time and could be a clue. In other words, its like the old mantra “when all else fails, try rebooting.” Where can you find more help?

Altair Technologies maintains a helpful site called EventID.net where users can search for additional information about obscure Windows events to help you interpret them. This site is community-based, meaning that users post their comments concerning events to create a community database that can then be searched by others. If you search EventID.net for information about the above event (source = DNS, event ID = 4004) the following is displayed:

Figure 4: Searching EventID.net for more information about event ID 4004 for DNS.

The really useful feature is under Details, where you can click the link “Comments and links for event id 4004 from source DNS” to see comments posted by other users:

Figure 5: Comments on event ID 4004 for DNS posted by users of EventID.net

The last comment is particularly useful as it indicates MS is aware of why this event occurs and suggests it can usually be safely ignored. Help and Support never told us that!

Configuring Event Logs

One of the first things you should do after you install a new Windows system is configure the event logs on that system. This is particularly important for servers where event logs can provide critical information to help you troubleshoot when things go wrong. Before we look at how to configure event logs, we need some background information on the different logs available, and Table 1 provides this below:

Event log

Log file

Function

Availability

Application log

AppEvent.evt

Records events as determined by each  software vendor

All Windows systems

Security log

SecEvent.evt

Records events based on how audit policy is configured

All Windows systems

System log

SysEvent.evt

Records events for Windows operating system components

All Windows systems

Directory Service log

NTDS.evt

Records events for Active Directory

Domain controllers only

DNS Server log

DnsEvent.evt

Records events for DNS servers and name resolution

DNS servers only

File Replication Service log

NtFrs.evt

Records events for domain controller replication

Domain controllers only

Table 1: Summary of Windows event logs

By default all event logs are:

• Stored in the %Windir%/system32/config folder
• Have a maximum size of 16 MB (Windows Server 2003) or 512 KB (Windows 2000/XP)
• Overwrite events more than 7 days old

Figure 6: Default configuration of DNS Server event log on a Windows Server 2003 DNS server.

Before you put your new Windows server into production, you should decide if these default settings are appropriate. Suggested best practices for configuring event logs on servers include the following:

• Increase the size of each event log to at least 50 MB. Since a typical event is about half a kilobyte in size, this means you’ll be able to store 100,000 events in each log. Note that the maximum supported size of each event log is about 300 MB. If your system drive has insufficient space for your event logs, you can move them to a separate volume by editing the subkey for each log under the HKLM/SYSTEM/CurrentControlSet/Services/Eventlog using Registry Editor, see Microsoft Knowledge Base article 315417 for more information.
• Change the overwrite behavior for the Security log to Do Not Overwrite Events if your enterprise is a high security environment. That way if the Security log fills up the system will shut down to ensure that no events in the Security log are lost. If you do this, make sure you also archive and then clear your Security log regularly to prevent such a shutdown from occurring unexpectedly.
• Change the overwrite behavior for the other event logs to Overwrite Events As Needed so that no overwriting occurs until the entire log becomes full. Again, be sure to regularly archive and clear your event logs to prevent the log from filling up and losing events because of overwrites.

If you have a number of computers and are running Active Directory on your network, you can also use Group Policy to configure event log settings. These settings are found under Computer Configuration/Windows Settings/Security Settings/Event Log in Group Policy Object Editor:

Figure 7: Group Policy settings for configuring event logs.

Searching and Filtering Events

While scrolling through the Event console lets you easily examine the most recent events that have been logged on your system, this quickly becomes impractical on busy systems where event logs are tens of megabytes in size. If you’re looking for instances of a particular kind of event however, you can use the Find and Filter options to speed things up.

Say you want to find all instances of Event ID 4004 in the DNS Server log as shown previously in Figure 1 above. To use the Find feature to accomplish this, right-click on the DNS Server log and select View --> Find, then fill in the Event ID and log name in the Find box:

Figure 8: Finding instances of Event ID 4004 in the DNS Server log.

Click the Find Next button and the first instances of this event is displayed in Event Viewer:

Figure 9: An instance of Event ID 4004 displayed in Event Viewer.

Then click Find Next to display the next instance of this event, and so on.

The frustrating thing about this approach is that the Find interface is not built directly into the Event Viewer window. So let’s try a different approach and use Filter instead. Right-click the DNS Server log again and select View --> Filter, then fill in the Event ID in the Filter tab of the DNS Server Properties sheet:

Figure 10: Filtering the DNS Server log for Event ID 4004.

Click OK and Event Viewer and the only events displayed in the DNS Server log are those having Event ID 4004:

Figure 11: All instances of Event ID 4004 are displayed.

From this information we could conclude that this was only a transient problem that happened a couple of weeks ago when we rebooted the DNS server.

Viewing Events on Remote Systems

Event Viewer also lets you connect to remote systems to view their event logs. The procedure is simple: right-click on the root (top) node in the console tree of Event Viewer and select Connect To Another Computer:

Figure 12: Connecting to a remote computer to view its event logs.

Then either type the name (NetBIOS or FQDN) of the remote computer or click Browse to find it in Active Directory. Click OK to connect:

Figure 13: Can’t connect to a remote computer to view event logs.

Oops, can’t connect! And the error message is cryptic. What went wrong? Typically this error message either indicates one of the following:

• You are not logged on with an account that has local Administrator access to the remote machine (a Domain Admins account should work).
• The Remote Registry service is not running or has been disabled on the remote machine.

Correct the situation and you should be able to connect to the remote machine and view its event logs.

Using EventCombMT.exe

In a previous article on WindowsSecurity.com we looked at the Account Lockout and Management Tools (ALTools.exe) download from Microsoft. One of these tools is EventCombMT.exe, which can be used to consolidate event logs from multiple computers into a single location for analysis. To use this tool double-click on EventCombMT.exe in the folder where you installed it, then specify the domain, servers, and kinds of events you want to find. For example, say you want to find all W32Time events on two servers (TEST230 and TEST235) in the testtwo.local domain:

Figure 14: Using EventCombMT to search for W32Time events on two servers.

Click Search and when the operation is finished a folder will open up displaying the results files generated:

Figure 15: Results files generated by our EventCombMT search.

Double-clicking on one of the two server files displays a comma-delimited list of W32Time events for that server:

Figure 16: Comma-delimited list of W32Time events on server TEST230.

You could then import these files into Excel to consolidate them for further analysis. EventCombMT also has a number of built-in queries you can use for common tasks like searching for locked-out accounts:

Figure 17: Searching for locked-out accounts using EventCombMT.

Other Event Monitoring Tools

EventCombMT.exe is useful but it isn’t very friendly to use. If you have a lot of computers whose event logs you want to monitor, you’re better off purchasing a commercial-quality tool for this purpose. We’ll end this article by mentioning two such tools:

Microsoft Operations Manager (MOM)

MOM is a Windows Server System product from Microsoft that lets you monitor events, health, and performance of computers on your network in real-time, consolidate such information in a central repository, and generate graphical web-based reports. MOM 2000 is showing its age however and will soon to be replaced by MOM 2005, which has a new Operator Console, greater security, enhanced rules, and improved reports. MOM 2005 also supports agentless monitoring, internationalization, and 64-bit agent support. For more information see this link on Microsoft’s web site.

GFI LANguard SELM

GFI LANguard Security Event Log Monitor (SELM) is a tool from GFI that lets you manage event logs on remote machines, consolidate event logs from multiple machines into a single repository, and view, report and filter events network-wide with easy and simplicity. You can also create your own custom alerts based on event ID, contents, and event condition so you can monitor specify issues across your network. SELM even lets you analyze event details, something MOM won’t let you do. GFI products are excellent--I speak from personal experience here--so this is one solution you should consider when looking for tools to monitor and troubleshoot event logs across your network.