Troubleshooting the Inbound Email Scheduler

Overview

The Check Inbound Mailboxes scheduler automatically retrieves unread emails from the system inbound email accounts configured in Admin > Inbound Email. Occasionally, the scheduler may fail or be delayed when attempting to import the emails. This article covers troubleshooting tips based on some common causes. 

Prerequisites

Some of the following troubleshooting steps require direct access to the database server or the inbound email server. To complete these steps, you must host Sugar on-site or enlist the help of a certified Sugar partner. At a minimum, you must log in to Sugar as an admin user to access the schedulers and inbound email settings in the Admin panel.

For more information on email configuration and schedulers, please refer to the Email and Schedulers documentation.

Symptoms

When the Check Inbound Mailboxes scheduler has encountered a problem, you may experience one or more of the following symptoms:

• Some or all of the email account's new messages fail to be imported to Sugar.
• There are excessive delays when sending or receiving messages.
• You see an "Invalid Inbound Email Credentials" error message.
• The Check Inbound Mailboxes scheduler job is failing.

Resolution

Inbound emails are imported through the Check Inbound Mailboxes scheduler job. When this job executes, it cycles through all of the active inboxes specified in Admin > Inbound Email. The import process pulls any unread messages from the inbox into Sugar. Once imported, the emails are marked as "read" in the inbox. The following sections explain some common reasons that the inbound email scheduler may not complete these steps as expected.

• Confirm the Connection to the Server
• Verify the IMAP Connection
• Verify the Inbound Port Numbers
• Resolve an "Invalid Inbound Email Credentials" Error
• Confirm the Unread Status of Missing Messages
• Check for Duplicate Messages
• Isolate a Problematic Email
• Adjust the Timeout Threshold

Confirm the Connection to the Server

If Sugar cannot connect to the mail server specified in Admin > Inbound Email, it may prevent you from saving the inbound email settings. Evaluate the following external factors that may be blocking the connection:

• Firewalls or other network infrastructures
• Incorrectly compiled PHP
• Failing SSL connection 

If your organization's network administrator has confirmed that these items are properly configured and you still cannot save the inbound email settings, try switching to a POP3 server option if it is available.

Verify the IMAP Connection

When polling email accounts, if Sugar cannot make an IMAP connection, it will be reflected in the Sugar Log. On-site customers can set the system's logging level to "Debug" via Admin > System Settings > Log Level. SugarCloud customers should refer to the Logging section of the SugarCloud Policy Guide for more information about log levels.

The Sugar Log will display output similar to the following line when the IMAP connection fails:

[FATAL] SCHEDULERS: could not get an IMAP connection resource for ID [ 139fb6c5-a66c-00fa-0645-507f55f9eb6a ]. Skipping mailbox [ My Mailbox Name ].

In most cases, this is nothing to worry about. If this message appears frequently, you should verify the connection to your mail account with Telnet or OpenSSL. For more information, please refer to the Knowledge Base article Testing Outbound Email Using Command Line. 

Verify the Inbound Port Numbers

You may experience issues if the port numbers specified in the Inbound Email settings are incorrect. Navigate to Admin > Inbound Email and open the affected inbound email account. Verify the port numbers according to the following tables, which list some common email and web port numbers.

Common Email Port Numbers

Common Web Port Numbers

Resolve an "Invalid Inbound Email Credentials" Error

If you see an error stating that the credentials for the affected inbound email account are not valid, navigate to Admin > Inbound Email, open the mail account, and re-enter the inbound email credentials to confirm that they are correct. Once saved, navigate to Admin > Repair and click "Rebuild Relationships". This will rebuild the relationship metadata and drop the cache file in to confirm that your changes are saved and processed.

Confirm the Unread Status of Missing Messages

Sugar will not process emails that are marked as "read" on the server. If some email messages are coming through to Sugar but others are not, confirm that the missing email messages are in an unread state.

Check for Duplicate Messages

If you have confirmed that a missing message is marked as unread but it is still not importing to Sugar, the message may have been identified as a duplicate. The next two sections explain how to find out if the message is a duplicate of another message in Sugar. 

Checking the Database

If you are not sure whether the email has been imported to Sugar, you can check the database for inbound emails sent to or from a specific email address by using the following query, replacing <email address> with the relevant value. 

SELECT emails.id, emails.name, emails_text.description, emails_text.to_addrs, emails_text.from_addr, emails.date_sent, emails.date_entered 
FROM emails_text 
INNER JOIN emails ON emails.id = emails_text.email_id 
INNER JOIN emails_email_addr_rel ON emails_email_addr_rel.email_id = emails.id 
INNER JOIN email_addresses ON email_addresses.id = emails_email_addr_rel.email_address_id 
WHERE type = 'inbound' AND email_addresses.email_address = '<email address>' 
ORDER BY date_sent DESC

To limit the results to emails imported in the last 7 days, adjust the WHERE statement as in the following query, again replacing <email address> with the relevant value:

SELECT emails.id, emails.name, emails_text.description, emails_text.to_addrs, emails_text.from_addr, emails.date_sent, emails.date_entered 
FROM emails_text 
INNER JOIN emails ON emails.id = emails_text.email_id 
INNER JOIN emails_email_addr_rel ON emails_email_addr_rel.email_id = emails.id 
INNER JOIN email_addresses ON email_addresses.id = emails_email_addr_rel.email_address_id 
WHERE date_sent > DATE_SUB(NOW(), INTERVAL 7 DAY) AND type = 'inbound' AND email_addresses.email_address = '<email address>' 
ORDER BY date_sent DESC

Checking the Logs

When a new email is importing, Sugar's debug log will contain some key notes about the import. On-site customers can set the system's logging level to "Debug" via Admin > System Settings > Log Level. SugarCloud customers should refer to the Logging section of the SugarCloud Policy Guide for more information about log levels.

The following sample content from a log file shows an email importing to Sugar:

[DEBUG] InboundEmail processing 1 email 24-------------------------------------------------------------------------------------- [DEBUG] *********** InboundEmail doing dupe check.
[DEBUG] ********* InboundEmail found [ inbox@sugar.crm ] as the destination address for email [ <CE8D775B.5AF08%inbox@sugar.crm> ]
...
[DEBUG] *********** NO duplicate found, continuing with processing.
...
[DEBUG] InboundEmail found multipart email - saving attachments if found.
[DEBUG] Localization: translating [<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); "><div><div><div style="font-family: Calibri, sans-serif;  ">Heres my example email</div></div></div></body></html>  ] from us-ascii into UTF-8
[DEBUG] Localization: translating [Here's my email body] from us-ascii into UTF-8
...
[DEBUG] ********************************* InboundEmail finished import of 1 email: Example Email

If the email already exists in Sugar, the log file will contain a section similar to the following:

[DEBUG] InboundEmail processing 1 email 24-------------------------------------------------------------------------------------- [DEBUG] *********** InboundEmail doing dupe check.
[DEBUG] ********* InboundEmail found [  inbox@sugar.crm  ] as the destination address for email [ <CE8D775B.5AF08%inbox@sugar.crm> ]
...
[DEBUG] InboundEmail found a duplicate email with ID (36ef71c4474ff9f07d64f3999d0df432)
[INFO] InboundEmail found a duplicate email: <CE8D775B.5AF08%inbox@sugar.crm>

The log tells you the ID of the duplicate email record in Sugar, so you can now find the original message in Sugar or delete it from the database as needed.

Isolate a Problematic Email

Sometimes the import process may be hanging on a specific email. When this happens, you will notice that the scheduler job is failing and that the latest emails have not been marked as read or imported into Sugar. To keep your instance moving and importing records, the best approach is to identify the problematic email and mark it as read. The sections below will outline the various ways to identify the email. Once you are up and running, you should then file a support case so that we can further investigate the issue.

Fetching the Problematic Email by Message Number

If you find that the system is hanging, you can fetch the problematic email by its message number. To do this, on-site customers can set the system's logging level to "Debug" via Admin > System Settings > Log Level. SugarCloud customers should refer to the Logging section of the SugarCloud Policy Guide for more information about log levels.

Once the scheduler runs and its attempt to pull the email fails, you will see a line or lines similar to the following in the log:

  [DEBUG] InboundEmail processing 1 email 24-----------------------------------------------------------------------------------------

The log will display this text for each message it analyzes. Find the last occurrence of this message in the log. If the scheduler is hanging on an email, it will hang on the last email it tried to import. In this case, "24" is the number of the message that has stalled the process. If you cannot identify the number of the problematic email message, please move on to the next section, Fetching the Problematic Email by Brute Force.

Next, gather the values that correspond to each component described in the following table. These values will be used to compose a connection string in later steps.

To compose the connection string for your instance, arrange the values that correspond to this table in the following format: 
{Server URL:Port/service=Protocol/Service}Mailbox

Using the example values in the table, the connection string will be {imap.gmail.com:993/service=imap/ssl}INBOX.

Alternatively, you can identify your system's connection string by modifying the ./inbound/modules/InboundEmail/InboundEmail.php file. Locate the following line:

  $connectString = $this->getConnectString($service, $this->mailbox);

Immediately below this line, add a new line with the following code:

  $GLOBALS['log']->debug("Using connection string: {$connectString}");

This will log the connection string to the log file so that you can retrieve it there, immediately following the text "Using connection string:". Copy the connection string and then identify the problematic email using the following script, replacing <your connection string>, <username>, and <password> with the actual values.

  <?php
//mailbox to connect to
$connectionString = '<your connection string>';
//the message number to fetch
$messageNumber = 24;
//open imap connection
$imap = imap_open($connectionString, '<username>', '<password>');
echo '<pre>';
echo '<p>-- Status information about the given mailbox --<p>';
print_r(imap_status($imap, $mbox, SA_ALL));
echo '<p>-- Information about the given message --<p>';
print_r(imap_headerinfo($imap, $messageNumber ));
echo '<p>-- The header of the specified message --<p>';
print_r(imap_fetchheader($imap, $messageNumber ));
echo '</pre>';

Once you have identified the problem email message, log in to your inbox and mark it as "read" so that the import process skips the email and continues importing the next email.

Fetching the Problematic Email by Brute Force

If you do not know the problem message's number, open the inbox and change the oldest unread email in the monitored folder to "read". Cycle the scheduler. You may need to repeat these two steps multiple times until the import process skips the current email and continues to the next email. This means that you have found the problem email and resolved the stalled import.

Adjust the Timeout Threshold

Depending on your symptoms, you may want to increase or decrease the job timeout configuration. By default, Sugar forces a scheduler job to fail If it attempts to run longer than the threshold. It then batches the remaining job for the next time the scheduler runs. Specified in seconds, the default jobs timeout threshold configuration is 3600 (e.g. 1 hour). For more information on job queues, please refer to the Scheduler Jobs documentation in the Developer Guide.

If you are a larger organization that is regularly importing a large number of emails but find that there is a delay when they are coming in, try increasing the timeout threshold by adding the following line to config_override.php: 

  $sugar_config['jobs']['timeout'] = 7200;

Setting the limit to 7200 doubles the default timeout to two hours. You can incrementally add more time to troubleshoot your issue, but it is not recommended to exceed a timeout configuration of 14400 (e.g. 4 hours) because increasing the timeout limit too much could negatively affect system performance.

Alternatively, importing your inbound email may cause unwanted stress on other parts of the system, which could result in other jobs not running as often as desired. You can attempt to expedite the import by decreasing the timeout threshold. To decrease the limit, add the following line to config_override.php: 

  $sugar_config['jobs']['timeout'] = 1800;

Setting the limit to 1800 decreases the default timeout by half to 30 minutes. This allows other jobs to continue without waiting for the slow job to complete. For the Inbound Email Scheduler, Sugar will process the imported emails in batches every time the job runs, reducing strain on the server and freeing up resources for other queued jobs to run.