/* Note : Make sure your table/dataset which you wish to process in batches has the column with running number such as an identity column or if not then dump the data in some local scoped temporary/hash table with column as RowID which can be generated using the ROW_NUMBER() function if you are using INTO command to dump the data. If you are using INSERT INTO... SELECT FROM then you can create RowID as an identity column. It is highly recommended to have clustered index on the RowID column */ /* Note : @Seed represents the starting row number @Increment represents the batch count */ -- It is advisable to implement the logging here to track when the process was started DECLARE @From_RowID INT , @To_RowID INT , @Seed INT = 1 , @Increment INT = 500 , @RowID INT -- Set @From_RowID & @To_RowID to process the first batch SET @From_RowID = @Seed SET @To_RowID = @Increment WHILE EXISTS (SELECT 1 FROM table_name WHERE RowID BETWEEN @From_RowID AND @To_RowID) BEGIN BEGIN TRY -- It is advisable to implement the logging here to track the progress so that specific decisions can be taken later. /* Data processing, INSERT, UPDATE or DELETE Note : Make sure to have the filter clause something like this "WHERE RowID BETWEEN @From_RowID AND @To_RowID" so that only specific set of records from the batch should get processed. */ -- Increment the @From_RowID & @To_RowID to process the next batch SET @From_RowID = @To_RowID + 1; SET @To_RowID = @To_RowID + @Increment; END TRY BEGIN CATCH -- It is advisable to implement the error logging here to track the progress and failures so that specific decisions can be taken later. -- In case any batch got failed due to any specific record in the batch then it's advisable to process that batch again for each individual record so that we can process the suuccessful records and also come to know which record is causing the failure. -- Set @RowID to the first Row ID of the batch SET @RowID = @From_RowID WHILE EXISTS (SELECT 1 FROM table_name WHERE RowID = @RowID) AND @RowID <= @To_RowID BEGIN BEGIN TRY -- It is advisable to implement the logging here to track the progress so that specific decisions can be taken later. /* Data processing, INSERT, UPDATE or DELETE Note : Make sure to have the filter clause something like this "WHERE RowID = @RowID" so that only specific row from the batch should get processed. */ SET @RowID = @RowID + 1; END TRY BEGIN CATCH -- It is advisable to implement the error logging here to track the progress and failures so that specific decisions can be taken later. -- Increment the @RowID to process the next record SET @RowID = @RowID + 1; END CATCH END SET @From_RowID = @To_RowID + 1; SET @To_RowID = @To_RowID + @Increment; END CATCH -- It is advisable to implement the logging here to track when the process was completed
One thought on “Pseudo Code for batch processing using WHILE LOOP”