I've been writing a data migration tool in C# which has to populate various tables the largest of which is 400,000 rows. I've been using LINQ to SQL so at first I tried creating entities one at a time and inserting them. This works ok but once you get to over a 1000 you notice things slowing down, and they continue slowing down until the whole thing just takes forever to do!!!
So the solution I discovered was to use XML. Rather than doing the inserts one at a time I add them to a dataset then when the dataset reaches a pre-determined size (I chose 2000 rows) I convert the dataset to XML and pass it to a stored procedure. SQL can do these 2000 inserts in less than a second so what once took 20-30 mins now takes less than a minute, not bad!!
The SQL syntax for processing the XML and inserting is:
CREATE PROCEDURE [dbo].[BulkInsertUsers]
@xmldoc nvarchar(max)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
declare @xml_hndl int
--prepare the XML Document by executing a system stored procedure
exec sp_xml_preparedocument @xml_hndl OUTPUT, @xmldoc
INSERT INTO [User]
(
u_id,
u_firstname,
u_lastname
)
SELECT
UserId, FirstName, LastName
FROM
OPENXML(@xml_hndl, '/NewDataSet/Users', 1)
WITH
(
UserId int 'UserId',
FirstName varchar(100) 'FirstName ',
LastName varchar(100) 'LastName '
)
END