Introduction

In order to increase the productivity of sales, service and marketing teams, the provision of updated information in CRM systems from external systems is extremely common. As the external systems are very varied, the integration methodologies are equally distinct and with multiple challenges. During a Salesforce CRM implementation project, the worldIT team faced the challenge of keeping several objects synchronized with an external system that is essential for the company's operations, but with significant technical limitations from an integration point of view. This external system manages critical information such as customers, contacts, addresses, orders, products, and invoices. With the implementation of Salesforce, this data came into existence in both systems, making the need to maintain consistency and timeliness of information crucial.

The top priority was to ensure that Salesforce users always had access to the most up-to-date data, avoiding decisions based on outdated or incorrect information tag. To achieve this goal, it was necessary to develop a solution that would allow efficient and real-time synchronization of data between the two systems, even in the face of the technical limitations of the external system.

The project required a strategic approach to overcome the challenges of data integration and synchronization. The worldIT team dedicated themselves to creating a robust mechanism that would ensure data integrity, providing a fluid and reliable user experience in Salesforce. This initiative has not only improved operational efficiency but has also strengthened users' confidence in the accuracy of the information available.

Problem

During the project, one of the main difficulties encountered was the limitation of the external system in sending real-time updates to Salesforce. We were told that it was not feasible for the external system to invoke an update API every time a record changed. This meant that if a customer, contact, address, order, product, or invoice was modified in the external system, that change would not be automatically reflected in Salesforce. 

Solution

A solution was then devised in which Salesforce, through the query REST API, would collect the existing data in the ERP and synchronize it with its data.

The option to use recurring query processes to keep objects synchronized in Salesforce has been deleted (with the exception of Orders, which are always looking for new updates), as it did not guarantee that at any given time we would have the most current image synchronized.

So we chose to invoke a query API when the user enters the page of certain objects. Some standard (Customers, Contacts, Orders), others customized.

The way to implement this requirement was to use Visualforce Pages on each of the pages where we wanted the synchronization to be done.  

Strategy

The sync feature follows these steps: 

1. The user enters the detail page of a Salesforce object (example: Customer).

2. The visualforce page added to the page invokes an apex class that is shared by all pages that use this feature (SyncEntityFromPage)

3. In this apex class, the object we want to synchronize is identified (in this example, Account) and several parameters used in the synchronization process are filled in. For example, the object to sync, the id of the record, path to append to the url, a substring a list of Salesforce fields that we want to sync, and their name in the API that will be invoked, …

4. An apex class centralized for API invocation and result handling is invoked. This class that we call framework allows you to make the invocations used in this feature, as well as in the recurring synchronization processes, error invocation reprocessing processes, invocation of the logging API in the external system, ...

5. This framework centralizes all API invocations and works in a way that is totally agnostic to the process that invoked it. That is, the API to invoke, the fields to compare, ... they are always passed by the class that invoked it. There are no specific exceptions for process A or B.

6. In a summary, the framework follows the following steps:

• It collects all the parameters needed by the process that is invoking it.
• Consult a custom setting where the credentials and the token used in the authentication process are stored, as well as the url to be invoked.
• Makes the HTTP request to the correct endpoint (url collected in the custom setting concatenated with a specific path depending on the object. Ex “[url]/cliente/[id]”)

• In this synchronization feature, the GET method is sent only with header (where is the id of the object we want to query, as well as the authentication token), in other processes are sent POST with JSON body (the framework is shared by all types of integrations)
• If a 401 error is received indicating that the token is invalid, it is necessary to send a new request, this time a POST where in the body is sent the client id and secret of the system we want to consult. By authenticating, we receive a new token that will be updated in the custom setting and the process continues. This means that the user does not realize that there was an authentication error during the query.
• If a non-401 error is received, a message describing the error is returned to the Apex class that invoked the Framework.
• If the API returns success, the returned data is collected and added to a list.
• Having the list of fields and their values in the external system, we proceed to collect the values that are in Salesforce for the same fields.
• At the end of the queries to the external system and Salesforce, we have lists with the fields we want to compare, their value in the external system, their value in Salesforce, their type and their label.
• We can finally move on to the stage of the process where the values in both systems are compared, always paying attention to the type of field we are analyzing and making small adjustments so as not to have false positives (different formatting in dates, decimal places in numbers, Y=true/N=false in Booleans, …)
• If no difference was found, we return this indication to the Apex class that invoked the Framework.
• If differences have been found, the process draws an HTML table with a row for each difference found. Each row has the field label, the value in Salesforce, and the value in the external system. The html is designed with the CSS classes used by Salesforce tables, so that a table with the look and feel that the user knows appears on the screen.

7. The Apex class that invoked the Framework, now has the result of the API invocation. Be it an error, an indication that no differences were found, or html with a table of differences. This result is passed to the visualforce page (showmsg, outputHTML).

8. The result is shown on the page using messages from the ApexPages class and the html drawn in the framework.

9. Example where no differences were found between Salesforce data and the external system:

10. Example where differences were found:

11. When differences are found, the user is given the possibility to synchronize the data. This means updating the record in Salesforce with the data collected in the external system.

12. The transfer of data from Salesforce to the external system is done in an asynchronous process, so the differences found may exist because the external system is not yet up to date, so we give the user the option of not press the button and ignore the warning. On other objects where this theme does not exist (Products), the Sync button is not shown. In this case, the table with the differences is shown for 5 seconds and the synchronization is done automatically, without user intervention.

13. The visualforce page also has some javascript functions for controlling html elements.  

Result

Implementing this feature has brought numerous benefits, ensuring that Salesforce users have immediate access to the most up-to-date information as soon as they enter the object details page. This provides a clear and accurate view of the differences between the data in Salesforce and the external system, preventing the risk of overwriting data with stale values. 

In summary, the implemented functionality not only prevents the use of outdated data but also significantly improves the efficiency, reliability, and accuracy of internal processes, providing a competitive advantage to the company. 

Conclusion

It's not always easy to integrate data between two systems. Technologies are not the same and systems do not always have the same needs and priorities. What seems very simple for one system, can be quite complex for another. We often have to abandon the ideal solution and go outside the box to find new ways to solve problems. 

This was the way that the worldIT team found to meet the customer's needs and overcome the constraints in integrating with the external system. Thanks to this solution, our client's Salesforce users can trust that they are always working with up-to-date data, ensuring accuracy and efficiency in their operations. This project exemplifies our ability to address technical challenges and deliver robust and effective solutions. 

#Salesforce #Data Integration #worldIT