The Batch Cleansing and Matching Processes
Faster processing is possible by concurrently processing records in batches.The batch cleansing and matching segregate the data loading and cleansing processes. These processes are background tasks available in the Project-Object Mapping screen for the Cleansing and Matching projects.
The batch processes require a job. When you create a project-object mapping, OneData a default batch job for both cleansing and matching. If, for some reason, either the default cleansing or matching batch job is not available, OneData creates one when you execute the batch job for the first time. You cannot modify or delete default batch cleansing and matching jobs because OneData uses them to track manually triggered jobs in the Job Center. You can also schedule batch jobs using the Add New Job option in the Batch <Cleansing or Matching> Execution screen.
Advanced logging options are also available.
How Does the Batch Cleansing Process Work?
The batch cleansing process performs the following operations:
1. It connects to the configured data quality cleansing server (for example, webMethods Locate or Trillium).
2. Cleanses records that are in the Unassigned state in the Consolidation (Staging) table.
3. When the cleansing process is completed successfully, moves the records to the Cleansed status.
How Does the Batch Matching Process Work?
The batch matching process performs the following operations:
1. It connects with the configured data quality matching server (for example, Trillium and webMethods Locate) to execute the data quality matching process which detects duplicate data.
2. Depending on the result from the data quality matching process, the batch matching process creates a new or updates an existing Gold record with the attributes from the record in the Consolidation Table based on survivorship rules.
3. Links the Gold record with the record from the Consolidation Table.
4. After processing records successfully, the batch process changes the status of the records from Cleansed to Create_Gold_Auto or Linked_To_Gold_Auto.
5. Creates a record in the Consolidation Gold table for the Create_Gol_Auto record status, or applies the Survivorship logic on the Consolidation Gold table.