Troubleshooting SugarCloud Storage

Overview

SugarCloud instances have limits for their allowed storage and customers must manage their storage usage to stay below the limit, or they can purchase more. Knowing how to manage your storage effectively can be challenging and this article will show what administrators can do to understand and address the issue.

Before beginning, always download a backup of your Sugar instance or start in a sandbox. If you need help, contact Sugar Support or your account manager if you wish to purchase additional storage. For further information on SugarCloud storage, please review the SugarCloud Managed Storage Limits page. For more information on sandboxes, please see the SugarCloud Sandboxes documentation. 

Assessing Your Storage

The first step to address storage concerns is to look at what your storage is and where the largest issues are. SugarCloud Insights is the perfect place to start, as it breaks your usage down by file system and database size.

The file system shows all the uploaded files in your instance. This includes documents, attachments in notes, and email attachments. If your file system is large, you may want to download a backup and work with Support to delete older files (e.g., delete anything older than three years). However, you can also manage your emails and their attachments yourself.

The database holds all the records and relationships in your Sugar instance. There are a lot of things that can be done on your own to help lower your storage. SugarCloud Insights will display the ten largest database tables used by your instance. From there, you can decide which records can be deleted to manage your storage.

Please note that the amount of storage consumed includes your production instance plus any sandbox instances hosted on Sugar's cloud service. SugarCloud Insights will show the total amount of storage used for each of your instances. If these are particularly large, you can contact Support to request that they remove uploaded files from the sandbox or empty large module tables. Additionally, when creating or refreshing your sandboxes, you can request that the sandboxes be made without uploads (i.e., without the file system attachments) to help manage storage. For more information, please see the SugarCloud Sandboxes documentation.

Checking Your Schedulers

When deleting a record in Sugar, the record is initially marked as deleted=1 in the database (i.e., soft deleted), which removes the record from the user interface but it still exists in the database, taking up storage. Once the Prune Database Scheduler on the First of the Month scheduler runs, the record is permanently deleted (i.e., hard deleted), freeing up the space. 

Schedulers automate the execution of custom and system processes, including database cleanup. You can view your schedulers in Admin > Schedulers. Some schedulers that run processes to help with storage are listed below, and you will want to ensure they are active. 

Note: Any times configured for schedulers will be based on the time zone of the admin user (user id = '1').

• Prune Database on 1st of Month: This scheduler is configured to run at 4:00 AM on the first day of every month but is inactive by default. This scheduler, when active, will permanently delete soft-deleted records and clear up database storage once a month. If you are trying to reduce storage, you may want to run this scheduler during off-peak hours, as it is resource intensive. Then, restore the scheduler to its original time after.
• Run Active Data Archives/Deletions: This scheduler is active by default and runs weekly on Saturdays at 10:00 PM. This scheduler will archive or delete any records set via the Data Archiver.
• Activity Stream Purger: This scheduler can help manage the number of activity stream records in the database if you have enabled Activity Streams. The scheduler is inactive by default and runs weekly on Sundays at 4:00 AM when enabled. This helps manage the size of the Activity Stream.
• Prune Tracker Tables: This scheduler is active and runs daily at 2:00 AM by default and deletes data in the tracker tables based on the tracker prune interval. This interval can be set from Admin > Trackers and defaults to 30 days. This helps manage the size of the Trackers tables.

Please note that if you are managing your data in a sandbox, most sandbox instances do not have cron enabled, which is required to run these schedulers. It is recommended to contact Support to request enabling cron in your sandbox. Alternatively, you can refresh your sandbox from production after the schedulers have run to help manage your storage. However, this requires completing these actions in your production environment first, which is not recommended.

Using the Data Archiver

The Data Archiver allows administrators to archive or delete Sugar records from their database. Jobs created in the Data Archiver can have one of two process types: Archive or Hard Delete. Only hard-deleting records will free storage space in your database. The Data Archiver allows you to mass-manage records using filters, which is significantly more convenient than attempting to mass-delete records from the list view 1,000 records at a time.

Pinpointing Common Problem Areas

Some areas come up often when managing SugarCloud storage. The following sections represent some commonly found large tables that you may see in SugarCloud Insights and provide specific actions that can be taken to address each of these common problem areas. 

Activity Streams

The activities and activities_users tables can fill up quickly; these tables are for the Activity Stream. Instances of Sugar created after 9.x come with this feature disabled, but older instances created prior may still have it enabled. If this feature is not being used or is causing storage issues, it can be turned off using System Settings > Enable Activity Streams. For more information on disabling the activity stream, please see the System documentation. After it has been disabled, you can use the Data Archiver to remove its entries in the database. 

Additionally, when active, the Activity Stream Purger Scheduler will remove records older than six months as of the current date from the activities table, and 25,000 records will be removed per scheduler job run. Please note that user posts, comments, and attachments are preserved. Still, the scheduler will remove other activity stream records generated from field updates, new record creation, and linking/unlinking records. You can contact Support to reconfigure how long data is stored in the Activity Stream (i.e., change from removing records older than six months to three months). 

Trackers

Trackers attempt to capture all actions done by all users in Sugar. The tracker options are disabled by default in Admin > Tracker, as it collects a large amount of data. If any options have been enabled to record tracker data, you can disable them via the Tracker settings page. Then, delete the tracker data using schedulers instead. To completely delete everything, navigate to Admin > Tracker and set the Number of days of Tracker data to store when Scheduler prunes the tables field to 1, as it cannot be 0. 

When the Prune Tracker Tables scheduler runs, the tables are cleared. 

Emails

When you archive your emails to Sugar, the storage can increase quickly. Emails add complexity as the email is stored in the database, while attachments are stored in the file system. When you delete emails in Sugar, they are set to deleted=1 (i.e., soft deleted). However, the attachments are immediately hard deleted and cannot be retrieved. Then, when the Prune database scheduler runs, the email is deleted from the database. You can also use the Data Archiver to hard delete these records and their attachments.

SugarBPM

Other tables you may commonly see as large in SugarCloud Insights are various SugarBPM tables, typically denoted with the prefix pmse_ in the database (e.g., pmse_inbox, pmse_bpm_flow).  However, in the Data Archiver, you will only see pmse_inbox listed in the modules. This table holds information about SugarBPM processes that are currently running or have been completed. Please note that when running the data archiver on the pmse_inbox table, all related data in related tables will also be deleted, including the pmse_bpm_flow table.

Using the Data Archiver, you can remove terminated or completed processes, which helps clean up space. 

Tracking Your Storage Over Time

Sugar does not track your storage usage over time. Therefore, keeping a record of the progress made to track trends, pinpoint problem areas that need addressing, and monitor changes is a good idea. You can track your storage over time using any spreadsheet editing program or even something as simple as a text document. Additionally, developers can access the SugarCloud Insights API and create more complex tracking. Please see the Downloading Backups From the Backups Module article for more information.