Values Store Service and Incremental Sync

Data sync has had incremental Sync mode for sometime all this requires is that you set Incremental mode and return a subset of Data from your Source that you want to compare and Sync with your target.

The trick is how to easily update the filter condition on your source to automatically move on each time you sync. With Ouvvi you can use the project properties and access the last sync time, however outside of Ouvvi it's a little trickier.

Introducing the Values Store Service this is a small Web Service hosted on the Web (you will need internet access for this) that can be used to Store, Retrieve and Update simple string values. This Service is a simple REST service and we have created a nice wrapper in Data Sync for you.

In order to implement the Incremental Sync with the Values Store service you would use Project Automation to get a value, update the source filter, synchronise and at the end update the value.

This is quite simple via the ValueStoreService object that exposes 2 methods GetValue and SetValue.

The API requires a guid ID value and a string Key value combined these must be unique.

The code below is an example of how you might use this, in Start() we get a Value from the Store if this is the first time i.e. there is no value in the store we return a default value of Today-1 Year.

We then use this value with a Helper Function GetFetchFilterXmlForModifiedSince which will return the CRM Fetch XML for us based on a DateTime value. We update the CRM fetch XML Filter property on the source with this and the Compare process starts.

After the Sync has completed and everything was OK, we then update the Store Value with the Time and Date when the process started so that the filter moves on. At this point if you compare again you will see zero results since there have been no changes on the source and the source returns no records.

class ProjectAutomationOverride : Simego.DataSync.Automation.ProjectAutomationShim
{    
    private string AppID = "{903A4B0A-586D-4666-95F4-EB858DBEF1F3}";
    private string AppKey = "AccountsSync1";
    private DateTime SinceValue;
    private DateTime StartValue = DateTime.UtcNow;
    
    public override void Start()
    {
        SinceValue = ValueStoreService.GetValue(AppID, AppKey, DateTime.Today.AddYears(-1));
        
        DataSourceA.FetchXmlFilterExpression = DataSourceA.GetFetchFilterXmlForModifiedSince(SinceValue);
    }    
    public override void End(ProjectAutomationResult result)
    {
        if(result.Success && result.HasChanges)
        {
            ValueStoreService.SetValue(AppID, AppKey, StartValue);    
        }
    }
}

If you want to host your own version of the Values Store Service contact us for the details and assistance in installing the service and re-directing the Data Sync ValueStoreService for your version.