Troubleshooting and Repairing Store Problems

Troubleshooting and Repairing Store Problems

This appendix has four main parts:

Problems with Full-Text Indexing

Problems with Permissions in a Mixed Exchange 5.5-Exchange 2003 Environment

Problems with Public Folder Replication

Other Problems

This section contains information about how to resolve problems that you may encounter with full-text indexing. It contains the following topics:

Safe Event Viewer Messages

Population Process Is Slow

Population Process Is Found in a Paused State

Deleted Message Is Still Visible in Search Results

Wrong Location Is Displayed After Moving the Index

Using Gather Log Entries to Identify Problems

Language Settings Problems

Queries Fail During Server Startup

Restoring Missing Performance Counters

Avoiding Disk Bottlenecks

High Paging

If you encounter problems with full-text indexing, Event Viewer and Performance Logs and Alerts are useful troubleshooting tools.

Safe Event Viewer Messages

Although Event Viewer is useful for troubleshooting full-text indexing problems, there are certain events (as described in the following sections) that do not necessarily indicate problems.

After you use Exchange System Manager to stop an index population, the Microsoft Search service may incorrectly log several copies of the following event message in the Event Viewer application event log:

This message does not indicate a problem, and you can ignore it.

When you shut down the Microsoft Search service in a clustered environment, you may see the following error:

This error is not actually a catastrophic failure. Wait for all services to shut down successfully, and then restart services, if necessary. To prevent this error from occurring, use Cluster Administrator to stop clustered resources, not the Services application in Control Panel. When you stop the service using Services in Control Panel, the cluster resource manager assumes that the resource failed, and it either attempts to restart the service or fails over the group.

When the Microsoft Search service is running, you may receive error messages similar to the following in the gather logs:

You can ignore these error messages. For more information about the gather logs, see 'Using Gather Log Entries to Identify Problems' later in this appendix.

If the population process is slow, Internet newsfeeds may be the cause. Internet newsfeeds may contain uuencoded binaries, which are indexed at a much slower rate than normal messages. When you run a population on a public folder that contains Internet newsfeeds, population time lengthens significantly.

Messages with large attachments may also cause performance problems if you have not optimized the maximum download size. The recommended setting is 4,000 megabytes (MB). Changing this setting requires editing the registry.

To change the maximum download size

Start Registry Editor.

In Registry Editor, set the following DWORD registry key to 4000 MB:

For more information about the setting the download size, see 'Optimizing Full-Text Indexing' in Appendix F, 'Using Full-Text Indexing.'

The Microsoft Search service pauses a population process if it cannot continue. To verify whether the Microsoft Search service, rather than an administrator, paused the population, check the event log. The Microsoft Search service logs an event when it must pause or stop the population. For example, the Microsoft Search service pauses a population if the disks are too full to add new information to the indexes or the log files. Usually, you can fix the problem (for example, by freeing space on a full drive), and resume the population. New documents added during the pause are not added to the index until the next population.

You can delete a message from a search results folder. The message is deleted, but the message remains visible in the search result window until you refresh the search.

If you use the Catutil tool to move the index, as described in Appendix F, 'Using Full-Text Indexing,' the index location displayed in Exchange System Manager is not updated. The index is moved successfully and functions correctly, but Exchange System Manager incorrectly displays the original location of the index. This is only a display error and does not affect the normal operation of full-text indexing. You cannot correct the display, but you can check the current location of the index at any time by checking the registry.

To check the current location of the index after using Catutil

In Registry Editor, view the following registry key:

Gather log files are generated during a population. These files contain log information for the Microsoft Search service. The files are located in the Program FilesExchsrvrExchangeServer_<servername>GatherLogs directory. The files have a .gthr extension.

If a particular document fails to be indexed for any reason, an entry is logged in the gather log file. Each entry lists the file name and the error number. To decode this error number, use the Gthrlog tool found in the Program FilesCommon FilesSystemMSSearchBin directory.

To use the Gthrlog tool to decode an error number from the gather log file

From the command prompt, type the following command, where <filename> is the name of the .gthr file:

Results from the tool are displayed at the command prompt.

The language settings of individual messages, attachments, the server, and the client computer affect indexing behaviors. This section provides guidelines for these behaviors and scenarios that illustrate the results of mixed language settings.

The guidelines that govern full-text indexing in mixed-language scenarios are complex. The following topics explain how various language settings affect indexing behaviors. Administrators can use this information to help determine the cause of user-reported search problems.

Language Setting of Individual Messages

The language setting of individual messages affects indexing behavior in the following ways:

If a message is a MAPI message, it has a Locale ID property, and full-text indexing uses this value to determine which word-breaker (identifies where individual words begin and end in a given text) to use. This property value comes from the Language setting in Microsoft Office on the client computer. If full-text indexing cannot find a word-breaker to match the Locale ID property, it uses the Neutral <0> property. For more information about how full-text indexing uses word-breakers, see Appendix F, 'Using Full-Text Indexing.'

If a message is created with Distributed Authoring and Versioning (DAV), it uses the 'Accept-Language' header to determine the correct locale.

If a message has no locale identified (which is often the case with messages from the Internet), the System Locale setting of the computer running Microsoft Exchange Server 2003 where full-text indexing is performed is used to determine the word-breaker.

The language setting of an attachment affects indexing behavior in the following way:

If an attachment is a Microsoft Office document, full-text indexing uses the language setting that was used to generate the document.

The language setting of the server affects indexing behavior in the following way:

If the message is non-MAPI (in other words, from the Internet), its Locale ID property is not set, and full-text indexing uses the System Locale setting of the server to determine which word-breaker to use.

The language setting of the client computer affects indexing behavior in the following way:

When a query is sent from Microsoft Office Outlook, the Locale ID of the client computer is also sent. If the Locale ID of the message does not match the Locale ID of the query, the search results are unpredictable.

Full-Text Indexing Behavior in Mixed-Language Scenarios

The following scenarios describe query behavior of content indexing with various language settings.

If you use U.S. language settings in Outlook, running on a client computer with U.S. language settings, to compose and submit a message to Exchange 2003, running on a server running Windows Server 2003 or Windows 2000 Server with U.S. language settings, the following process occurs:

Full-text indexing indexes the message using the U.S. language setting word-breaker.

Queries from the client computer with U.S. language settings are processed as expected.

In this example, the client computer is configured as follows:

The client computer has Hebrew language settings.

Office has U.S. language settings.

Outlook has Hebrew language settings.

If you compose a message on the client computer described in this example and submit the message to Exchange 2003 with the System Locale setting set to U.S., the following process occurs:

Full-text indexing uses the U.S. word-breaker to index the message. The Locale ID property of the message defaults to U.S. because of the Office settings.

Queries from the Hebrew client computer fail because the Hebrew document does not have the proper word-breaker applied.

In this example, the client computer is configured as follows:

The client computer has Japanese language settings.

Office has Japanese language settings.

Outlook has Japanese language settings.

If you compose a message on the client computer described in this example and submit the message to Exchange 2003 with the System Locale setting set to U.S., the following process occurs:

Full-text indexing uses the Japanese word-breaker to index the message.

Queries from the Japanese client computer succeed because the message was indexed and queried with the same Locale ID property.

During initialization, in the first few minutes after starting a computer running Exchange Server with full-text indexing, users might receive their mail but not receive results from queries. This failure to receive query results occurs because the Microsoft Search service is loading the index, and Exchange is loading the property store. Queries do not return results until these processes are complete.

Event messages similar to the following indicate that the counters used by the Performance Logs and Alerts service and the Performance application (also called System Monitor) are missing. If you receive one of these messages, restore the counters by restarting the Microsoft Search service. For more information about these monitoring applications, see the Windows Resource Kit.

Avoiding Disk Bottlenecks

To avoid disk read and write bottlenecks, use the following guidelines:

Disk queue length should be monitored.

The queue length is expected to average more than the number of spindles in the redundant array of independent disks (RAID) array.

The length should drop to zero occasionally.

The queue should be empty occasionally. Having something always in the queue indicates a problem.

The average time per disk write and per disk read should be close to the expected latency. The system should take roughly 10 milliseconds for a disk write or read. If the configuration has a hardware cache or a RAID controller, the time could be even less.

High memory-to-disk paging can indicate a memory bottleneck. Check your performance counters and monitor them for warning signs. In particular, check the Memory: Page writes/sec and Memory: Page reads/sec counters.

A user's inability to see public folders in Outlook is often the first sign of a permissions problem. This section describes ways that you can determine whether the problem is caused by permissions replication, and how you can track the source of the problem.

Determine What is Preventing a User from Seeing the Public Folder in Outlook

Determine which of the following situations is preventing a user from seeing the public folder in Outlook:

The public folder was not replicated to the server.

The public folder permissions were not converted successfully.

The best way to determine the cause of the problem is to view the folder tree in Exchange System Manager. If the public folder appears in the tree when Exchange System Manager is connected to a particular public folder store, but a user with a mailbox on the same server as the public folder store cannot see the public folder, the problem has to do with permissions, not replication. However, if the public folder does not exist in the tree, you may have a replication problem.

In mixed-mode environments where permissions in access control lists (ACLs) were not successfully converted to ptagNTSD data, users may not be able to access the folder, even though the permissions appear to be correct in Exchange System Manager. For more information about the conversion process and the properties involved, see 'Working with Permissions for Public Folders and Mailboxes' in Chapter 7, 'Managing Mailbox Stores and Public Folder Stores.'

When you use Exchange System Manager to view the permissions for a public folder in the default Public Folders tree (also called the MAPI tree), Exchange System Manager displays the permissions that are contained in the ptagACLData property (if one exists) rather than recalculating the permissions from the ptagNTSD property. In other words, Exchange System Manager displays permissions from the 'replicated in' property (which Exchange normally discards) rather than the permissions that are calculated from the ptagNTSD property, which actually control access to the folder. Use the following procedure to view the ptagNTSD permissions.

To view the ptagNTSD permissions on a folder in Exchange System Manager

In Exchange System Manager, in the console tree, right-click the public folder for which you want to view the properties, and then click Properties.

Click the Permissions tab in the public folder's properties.

To display the ptagNTSD ACL data, which controls access to the folder, hold down the CTRL key and click Client Permissions, and then click Advanced.

You can use diagnostic logging to record permissions events to the application event log in Event Viewer. By default, the public folder logging level is set to None, which logs only critical errors.

You can use the Diagnostics Logging tab in the Properties of a server running Exchange 2003 to increase the logging level on a public folder. This increased logging level allows you to obtain more detailed permissions information.

To view the attempts of individual users to access folders and show the permissions that are granted to users when they try to access folders, set the Logons and Access Control diagnostics to maximum.

To set the Logons and Access Control diagnostics to maximum

In Exchange System Manager, double-click Servers, right-click a server, and then click Properties.

Click the Diagnostics Logging tab.

Under Services, double-click MSExchangeIS, and then click Public Folder.

Under Categories, click Logons. Under Logging Level, click Maximum.

Under Categories, click Access Control. Under Logging Level, click Maximum.

For more information about diagnostics logging, see 'Use Diagnostic Logging and Event Viewer' in the Exchange Server 2003 Help.

Event ID 9548: Disabled user <user> does not have a master account SID

When users other than folder owners are not able to access a folder, look for events 9548 and 9551 in the application event log. (Event 9551 is discussed in the following section.)

If you view the client permissions for the folder using Exchange System Manager, initially they look correct. However, viewing the permissions using the Advanced dialog box (these are the raw permissions that are stored in the ptagNTSD property) reveals that only the owner has been converted successfully from the Microsoft Exchange Server 5.5 version of the permissions to the Exchange 2003 version.

There are two potential causes for this problem:

The Microsoft Active Directory directory service does not have a trust set up to the Microsoft Windows NT version 4.0 domain that holds the user's account.

The user has been disabled manually and does not have an external account.

You should be able to fix the problem using the following approaches:

Remove the disabled accounts from the ACL.

Give the disabled accounts associated external accounts.

Create a trust between the Windows NT 4.0 (or external Windows) domain and Active Directory. This trust gives the disabled accounts associated external accounts (and master account security identifiers (SIDs)).

Event ID 9551: An error occurred while upgrading the ACL on folder <folder> located on database <database>

When users other than folder owners are not able to access a folder, look for events 9548 and 9551 in the application event log (event 9548 is discussed in the previous section). When event 9551 occurs, it is logged each time a user attempts to access the folder.

There are three potential causes for upgrade problems:

No user connection agreement is in place to replicate the Exchange 5.5 mailboxes into Active Directory.

The user has been deleted from Active Directory.

There is replication latency.

When Exchange 2003 receives the replication message, Exchange will attempt to upgrade the data stored in ptagACLData to Windows NT SIDs. If the upgrade process fails, only owners are stored in ptagNTSD. No one else will be able to access the folder.

You should be able to fix the problem using the following approaches:

Remove the bad entry.

Replicate the missing user to Active Directory.

When users that belong to a specific distribution list or group cannot access a folder, look for events 9552 or 9556 in the application event log. Following are the event descriptions for Events 9552 and 9556:

In addition, Outlook users that attempt to set permissions involving users that do not have access may see the following error:

Administrators using Exchange System Manager who attempt to set permissions involving users that do not have access may see the following error:

The most likely cause for these errors is that the Exchange 5.5 distribution list to which the users belong was replicated into an Active Directory mixed-mode domain rather than into an Active Directory native-mode domain. As a result, the universal distribution group that corresponds to the distribution list was created in an Active Directory mixed-mode domain. The domain into which groups are replicated is configured in the user connection agreement that governs the migration of Exchange 5.5 distribution lists to Active Directory.

To be used in setting permissions, the universal distribution group must be converted to a universal security group. This conversion can only take place if the universal distribution group has been created in a native-mode domain. For more information about this conversion process, see 'Working with Permissions for Public Folders and Mailboxes' in Chapter 7, 'Managing Mailbox Stores and Public Folder Stores.'

To fix the conversion problem, do the following:

Create a native-mode domain in Active Directory.

Configure the user connection agreement to use the new domain for groups that it migrates from Exchange 5.5.

If you think there is a problem with folder replication (especially replication of the hierarchy), use Exchange System Manager to check whether folders have replicated. Do not rely on the view provided by Outlook to determine whether folders have replicated. The problem might relate to permissions, not replication.

To help identify replication issues, set diagnostic logging to Maximum for the MSExchangeIS: Public Folder categories Replication Incoming, Replication Outgoing, and Non-delivery Reports.

If replication messages are not being sent or received, check that normal e-mail routing between the servers works.

This problem could have one of the following causes, each of which has its own solutions:

Public folder stores do not have e-mail addresses.

Check that the Recipient Update Service has stamped the mail attributes onto the public folder store's directory objects correctly.

In mixed Exchange 5.5/Exchange 2003 organizations, check that Exchange 5.5 can access the directory entries for the Exchange 2003 public folder stores, and that Exchange 2003 can access the directory entries for the Exchange 5.5 public folder stores.

There is no route for mail to follow.

Check that normal mail traffic can flow between the servers.

If the replication message goes over an Exchange 5.5 Internet Mail Connector (IMC), check that the ResolveP2 registry key is set to . This registry key is located at:

Check also that the Exchange 5.5 Public Information Store object exists in the Active Directory Configuration container and has a valid X.400 proxy address (you can use ADSI Edit or the LDP utility to check attribute values).

Transport links are restricted to disallow system messages.

Check that there is a route for system messages between the servers. Winroute.exe indicates whether there are restrictions on the links.

The backfill can take a long time when a new server is installed and the initial status request gets lost or goes to a server that also has no knowledge of the hierarchy. To remedy this, make a change to the hierarchy on another server and check that it replicates through correctly. The server should backfill within 24 to 48 hours.

If a server does not appear to be backfilling, check whether new folders that have been added to other servers replicate as part of hierarchy replication to the backfilling public folder store. If they do replicate correctly to the backfilling public folder store, the server determines that it's not synchronized and writes an entry into the backfill array. Backfilling could take two or three days to complete.

This section contains information about how to resolve problems that do not fit into the other categories in this appendix. These issues include the following:

Unable to Access Permissions on a Public Folder (Invalid Windows Handle Error)

One or More Users Could Not Be Added to the Folder Access List

Mail Messages to Public Folder Were Not Delivered

Outlook Web Access Cannot View a Public Folder After the Tree Has Been Renamed

Message 'Operation Failed' When Attempting to Access a Tree Using Exchange System Manager

Exchange 5.5 Servers See Multiple Public Folder Stores on an Exchange 2003 Server

In a Mixed Exchange 5.5-Exchange 2003 Environment, Users Cannot Access a Public Folder Using Outlook Web Access

Attachment Exceeds Storage Limit on Public Folder

Unable to Access Permissions on a Public Folder (Invalid Windows Handle Error)

The most frequent cause of the Invalid Windows Handle Error in Microsoft Exchange Server 2003 is an administrator's use of the M: drive (the Exchange Installable File System) to modify permissions on a public folder. Servers running clean installations of Exchange 2003 do not have an M: drive, although it may still be accessible on upgraded servers that previously ran Exchange 2000.

This error can also arise if you use the wrong dialog box in Exchange System Manager to modify client permissions on a public folder, although this is unlikely to occur. For more information about the correct way to modify permissions on a public folder, see 'Special Considerations for Working with Client Permissions' in Chapter 7, 'Managing Mailbox Stores and Public Folder Stores.'

The underlying cause of this error is that, if you use the Windows user interface to modify client permissions for a public folder, the permissions are stored in such a way that Exchange is no longer able to convert the permissions to their MAPI form. If this happens, you will no longer be able to use the dialog boxes in Outlook or Exchange System Manager to edit the permissions.

To reset permissions on public folders

In Exchange System Manager, under the Public Folders node, create a new top-level folder.

Move the affected folder and subfolders (those with the wrong permissions settings) into this new folder.

Set the permissions on the new top-level folder so that an account with administrator permissions in Active Directory is the owner.

Right-click the new top-level folder, point to All Tasks, click Propagate Settings, and then select the Administrative Rights and Folder Rights check boxes.

After you click OK, the changes to the permissions are applied to all subfolders of the new top-level folder.

Move the affected folder and subfolders back to their original locations in the Public Folders tree.

Verify that, in Exchange System Manager, you can now modify the permissions.

One or More Users Could Not Be Added to the Folder Access List

Either Outlook users or administrators using Exchange System Manager could see this message when trying to grant users permissions to a folder in the Public Folders tree. When this error occurs, Default and Anonymous permissions on the affected folder do not work. Only users that were previously granted permissions to the folder are able to access it. However, if you try to use the Properties button to view the properties of one or more of those users in the folder's Client Permissions dialog box, you will get a MAPI error message. This user (or users) is the root of the permissions problem.

This permissions problem occurs when a user who does not have an Exchange mailbox creates or administers a folder in such a way that they user is granted explicit permissions to the folder (this can happen using Exchange System Manager or the Exchange Installable File System). The most likely cause is that someone used an account that has permissions to administer folders (for example, an account that belongs to the Enterprise Admins group), but no mailbox was ever created for that account.

To fix this problem, in the folder's Client Permissions dialog box, identify the user whose properties you cannot access. Remove the user from the folder's access control list, or go to the Active Directory Users and Computers snap-in and create a mailbox for that user.

If you have a mixed-mode Exchange organization, check that the public folder connection agreement has replicated the folder's directory objects correctly. Remember that you cannot e-mail general-purpose hierarchy folders from Exchange 2003 if the e-mail message travels by way of an Exchange 5.5 server.

In any Exchange organization, an e-mail to a folder first needs to go to a public folder store that supports the correct public folder tree to find the replica list for the destination folder. It may be that the public folder store that was chosen has not received the updated replica list of the destination folder yet.

Outlook Web Access Cannot View a Public Folder After the Tree Has Been Renamed

When you rename a public folder tree, you have to update all of the virtual directories that point to that tree. The changes will not finish propagating from Exchange to Internet Information Services (IIS) until after the public folder store has been remounted.

Therefore, if you rename a tree, you need to:

Update the virtual directories on the servers that hold public folder stores for this tree, so that they point to the new tree.

To propagate the change through Exchange and IIS, remount all of the public folder stores that support the tree.

To access the public folder trees, Exchange System Manager uses an OLEDB service that depends on the World Wide Web Publishing Service (W3SVC). If you have problems accessing a tree using Exchange System Manager, check the following:

Check that World Wide Web Publishing Service is running on the Exchange 2003 server.

Check that the Microsoft Internet Explorer settings do not have a non-existent proxy server configured.

This problem can occur if a new configuration connection agreement (Config CA) replaces an existing Config CA. This replacement can occur, for instance, if servers running Site Replication Service (SRS) are removed from the organization incorrectly.

The problem starts when the new Config CA replicates the Active Directory object for a default public folder store from an Exchange 2003 server in an Exchange 2003 administration group to the Exchange 5.5 Directory of an Exchange 5.5 server. The new Config CA does not 'see' that the Exchange 2003 default public folder store's object already exists in the Exchange 5.5 Directory because the object has the old Config CA's replication signature.

As a result of the replication cycle, a second default public folder store appears in the Exchange 5.5 Directory for the Exchange 2003 server. Because the server's container already has an object called Microsoft Public MDB, the new object is named Microsoft Public MDB - 1. However, this name is too long for a public folder store object in Exchange 5.5, and, as a result, the replication engines on Exchange 5.5 servers will fail to start throughout the organization.

The following errors will be logged:

The 'site folder teardown check' referred to in the error message is performed each time an Exchange 5.5 server starts, to determine whether any sites have been removed, in which case the list of site folders (SCHEDULE + FREE BUSY and so forth) needs to be cleaned up. You can do this clean-up by comparing details about all of the site folders with details about all of the public folder stores in the organization.

Because the string Microsoft Public MDB - 1 is too long, the replication thread fails with an Out Of Memory error (0x3f0) when it tries to get site details of the store with that name. This failure in turn causes the replication engine to fail to start. The only way to fix this problem is to remove both the incorrect directory object and the original correct directory object for the 2003 public folder store from the Exchange 5.5 Directory, and replicate the directory entry again.

Microsoft Outlook Web Access users cannot access folders that exist only on Exchange 5.5 servers. Check your public folder connection agreements to make sure that the folders are being replicated to at least one Exchange 2003 server.

Attachment Exceeds Storage Limit on Public Folder

After you install Exchange 2003 (or Exchange 2000 Server Service Pack 1 (SP1) or later), when you post a new item with an attachment that is larger than 1 megabyte (MB) to a public folder, you receive the following error message:

Attachments that are smaller than 1 MB are not affected. This issue occurs even if no limits are set on the public folder store.

This issue occurs because a system folder called OWAScratchPad is created when a user adds an attachment to a public folder post. This system folder has a limit of 1,024 kilobytes (KB).

To work around this issue, use Exchange System Manager to either increase or remove the limit on the OWAScratchPad folder.

To change or remove the size limit for attachments in public folders

In Exchange System Manager, right-click Public Folders, and then click View System Folders.

Expand Public Folders, then right-click the OWAScratchPad folder. Click Properties, and then click Limits.

Under Storage Limits, the Maximum item size (KB) is set to 1,024, or 1 MB, by default. To change the limit:

Under Storage Limits, change the limit in the Maximum item size (KB) box.

-or-

Click Use public store defaults. When this check box is selected, the limit settings are controlled by the Maximum items size (KB) setting and the Prohibit post at (KB) setting on the Limits tab of the public folder store Properties dialog box. However, if the store is configured by a system policy, the settings are located on the Limits tab of the policy's Properties dialog box.