SQL Bulkcopy run datatable in batches

c# sql sqlbulkcopy

Question

Currently I have a flatfile of around 10 million records.

I firstly read the file and map to columns into a datatable. Then from the Datatable using SQLBulkCopy move it into database. This is a slow process and times out a lot.

I wonder if this process could be done better.

1) I.e. rather than load my 10 million records into a datatable and then into database. Would it better to load "chunks" of 1 million into datatable and process them into database in a loop till all are processed ? Or if any one has any suggestions on how to improve this process would appreciate it !

foreach (var line in File.ReadLines(fileLocation + @"\" + filename))
        {
            var dataColumns = line.Split('\t');
            var dr = dt.NewRow();

            for (var i = 0; i < dataColumns.Length; i++)
            {
                dr[i] = dataColumns[i].Trim().Length == 0 ? null : dataColumns[i];
            }

            dt.Rows.Add(dr);
        }

        using (var destinationConnection = new SqlConnection(Settings.Default.ConnectionString))
        {
            var tableName = filename.Substring(0, filename.IndexOf('.'));
            destinationConnection.Open();

            try
            {
                using (var createAndDropTableCommand = new SqlCommand("sp_DropAndCreateTable", destinationConnection))
                {
                    createAndDropTableCommand.CommandType = CommandType.StoredProcedure;
                    createAndDropTableCommand.Parameters.Add("@TableToCreate", SqlDbType.VarChar).Value = tableName;
                    createAndDropTableCommand.ExecuteNonQuery();
                }

                using (var bulkCopy = new SqlBulkCopy(destinationConnection))
                {
                    bulkCopy.DestinationTableName = "dbo." + tableName;
                    bulkCopy.BatchSize = 10000;
                    bulkCopy.ColumnMappings.Clear();
                    foreach (DataColumn col in dt.Columns)
                    {
                        bulkCopy.ColumnMappings.Add(new SqlBulkCopyColumnMapping(col.ColumnName, col.ColumnName));
                    }

                    bulkCopy.WriteToServer(dt);
                }
            }
            finally
            {
                destinationConnection.Close();
            }
        }

Accepted Answer

You are definately on the right track.
SQL Bulk copy should be able to process over a million records in around 30 seconds. I have used DataSets in dbsourcetools.codeplex.com with SQLBulkCopy

            System.Data.DataSet oDataSet = new System.Data.DataSet();
            oDataSet.ReadXmlSchema(strSchemaXml);
            oDataSet.ReadXml(strTableXml);

            if (oDataSet.Tables.Count > 0)
            {
                using (System.Data.SqlClient.SqlConnection conn = new System.Data.SqlClient.SqlConnection(oTargetDatabase.ConnectionString))
                {
                    conn.Open();
                    System.Data.SqlClient.SqlTransaction oTran = conn.BeginTransaction();

                    System.Data.SqlClient.SqlBulkCopy oSqlBulkCopy
                        = new System.Data.SqlClient.SqlBulkCopy(conn, System.Data.SqlClient.SqlBulkCopyOptions.KeepIdentity, oTran);
                    oSqlBulkCopy.BulkCopyTimeout = 600; 
                    oSqlBulkCopy.BatchSize = 1000;
                    oSqlBulkCopy.DestinationTableName = strFullyQualifiedTableName; 
                    oSqlBulkCopy.NotifyAfter = 10000;
                    oSqlBulkCopy.SqlRowsCopied += new System.Data.SqlClient.SqlRowsCopiedEventHandler(oSqlBulkCopy_SqlRowsCopied);
                    foreach (System.Data.DataColumn ocol in oDataSet.Tables[0].Columns)
                    {
                        oSqlBulkCopy.ColumnMappings.Add(ocol.ColumnName, ocol.ColumnName);
                    }

                    oSqlBulkCopy.WriteToServer(oDataSet.Tables[0]);
                    oTran.Commit();
                    conn.Close();
                }
                System.Console.WriteLine("Wrote : " + oDataSet.Tables[0].Rows.Count.ToString() + " records ");
            }



Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why