Timeout Expired using SqlBulkCopy when there are 300Million rows in SQL Server Database

sql sqlbulkcopy timeout

Question

I wasn't having any problems in inserting ~15000 rows in SQL Server database when using SQLBulkCopy in certain conditions where there are 183M existing rows in the database.

But when there were 300M existing rows in the database, i experienced the exception:

Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Note that there are no constraints or anything.It is just a denormalized table.

using (var sqlBulkCopy = new SqlBulkCopy(sqlConn, SqlBulkCopyOptions.TableLock, null))
            {
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_HISTORY_PARMETER_ID,
                    SqlServerDatabaseStrings.SQL_FIELD_HISTORY_PARMETER_ID);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_SOURCE_TIMESTAMP,
                    SqlServerDatabaseStrings.SQL_FIELD_SOURCE_TIMESTAMP);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_VALUE_STATUS,
                    SqlServerDatabaseStrings.SQL_FIELD_VALUE_STATUS);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_ARCHIVE_STATUS,
                    SqlServerDatabaseStrings.SQL_FIELD_ARCHIVE_STATUS);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_INTEGER_VALUE,
                    SqlServerDatabaseStrings.SQL_FIELD_INTEGER_VALUE);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_DOUBLE_VALUE,
                    SqlServerDatabaseStrings.SQL_FIELD_DOUBLE_VALUE);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_STRING_VALUE,
                    SqlServerDatabaseStrings.SQL_FIELD_STRING_VALUE);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_ENUM_NAMEDSET_NAME,
                    SqlServerDatabaseStrings.SQL_FIELD_ENUM_NAMEDSET_NAME);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_ENUM_NUMERIC_VALUE,
                    SqlServerDatabaseStrings.SQL_FIELD_ENUM_NUMERIC_VALUE);
                sqlBulkCopy.ColumnMappings.Add(SqlServerDatabaseStrings.SQL_FIELD_ENUM_TEXTUAL_VALUE,
                    SqlServerDatabaseStrings.SQL_FIELD_ENUM_TEXTUAL_VALUE);

                sqlBulkCopy.DestinationTableName =
                    SqlServerDatabaseStrings.SQL_TABLE_HISTORYSAMPLEVALUES;
                sqlBulkCopy.WriteToServer(historySampleValuesDataRow);
                sqlBulkCopy.Close();

            }

Any ideas on what is happening? BTW, SQL Server Standard is what i'm using

Accepted Answer

You have 2 properties on your sqlBulkCopy object that can help you

  • BulkCopyTimeout: This is the value of the timeout, 30 seconds by default. Set this value to 0 to disable the timeout entirely.
  • BatchSize: Instead of inserting 300MM straight into the table, inserts the records at a batched size, maybe 1MM/time to prevent the timeout issue (and ease load a bit)

References:



Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why