Powershell: Implementing an IdataReader wrapper around streamreader

idatareader powershell sqlbulkcopy streamreader

Question

I am trying to load extremely large CSV files into SQL Server using Powershell. The code also has to apply on the fly regex replacements, allow for various delimiters, EOR, and EOF markers. For maintenance, I would really like all of this logic to exist in Powershell without importing assemblies.

To be efficient, I know I need to use the SQLBulkCopy method. But, all of the Powershell examples I see fill a DataTable and pass it which is not possible for me because of the file size.

I am pretty sure I need to wrap StreamReader in an Idatareader and then pass that to SQLBulkcopy. I found a couple great examples of this implemented in C#:
http://archive.msdn.microsoft.com/FlatFileDataReader
http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader

Is it possible to accomplish this functionality using native PowerShell without importing the C# assembly? I am specifically having a hard time converting the abstract class wrapper.

This is the code I have so far that does not pass an IdataReader and breaks on memory limits.

function Get-CSVDataReader()
{
param (
   [string]$path
)
    $parsedData = New-Object 'System.Collections.Generic.List[string]'
    #List<string[]> parsedData = new List<string[]>()

    $sr = new-object IO.StreamReader($path)

    while ($line = $sr.ReadLine())
    {
        #regex replace and other logic here
        $parsedData.Add($line.Split(','))
    }

    ,$parsedData #if this was an idatareader, the comma keeps it from exploding
}

$MyReader = Get-CSVDataReader('This should not fill immediately.  It needs a Read Method.')

Thanks a bunch for the help.

Popular Answer

If all you want to do is use a DataReader with SqlBulkCopy you could use the ACE drivers which comes with Office 2007/2010 and is also available as a separate download to open an OLEDB connection to to CSV file, open a reader and call WriteToServer

$ServerInstance = "$env:computername\sql1"
$Database = "tempdb"
$tableName = "psdrive"
$ConnectionString = "Server={0};Database={1};Integrated Security=True;" -f $ServerInstance,$Database
$filepath = "C:\Users\Public\bin\"

get-psdrive | export-csv ./psdrive.csv -NoTypeInformation -Force

$connString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=`"$filepath`";Extended Properties=`"text;HDR=yes;FMT=Delimited`";"

$qry = 'select * from [psdrive.csv]'

$conn = new-object System.Data.OleDb.OleDbConnection($connString)
$conn.open()
$cmd = new-object System.Data.OleDb.OleDbCommand($qry,$conn) 
$dr = $cmd.ExecuteReader()

$bulkCopy = new-object ("Data.SqlClient.SqlBulkCopy") $connectionString
$bulkCopy.DestinationTableName = $tableName
$bulkCopy.WriteToServer($dr)

$dr.Close()
$conn.Close()

#CREATE TABLE [dbo].[psdrive](
#   [Used] [varchar](1000) NULL,
#   [Free] [varchar](1000) NULL,
#   [CurrentLocation] [varchar](1000) NULL,
#   [Name] [varchar](1000) NULL,
#   [Provider] [varchar](1000) NULL,
#   [Root] [varchar](1000) NULL,
#   [Description] [varchar](1000) NULL,
#   [Credential] [varchar](1000) NULL,
#   [DisplayRoot] [varchar](1000) NULL
#)



Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Is this KB legal? Yes, learn why