Introduction
Steps for Data Manipulation and Insertion Using PowerShell
- Load the .NET Assembly: We will start by loading the necessary .NET assembly that provides us with the required classes for database operations.
- Create a DataTable Object: A DataTable object will be instantiated to mimic the structure of the SQL Server table where the data will be inserted.
- Define the Schema: We’ll define the columns of our DataTable, which should match the schema of the target SQL Server table.
- Generate Random Data: We’ll programmatically generate random data to populate our DataTable with rows that simulate the actual data.
- Establish a Database Connection: A connection to the SQL Server database will be established using the necessary credentials and connection string.
- Prepare for Bulk Insert: We will prepare a SqlBulkCopy object, which is optimized for bulk data operations and will handle the insertion of data from our DataTable to the SQL Server table.
- Execute the Bulk Insert: Finally, we will execute the bulk insert operation, transferring all the data from the DataTable to the SQL Server table in an efficient manner.
Step 1: Loading the ADO.NET Assembly
First, we need to load the necessary .NET assembly which allows us to work with databases:
Add-Type -AssemblyName "System.Data"Add-Type is a PowerShell cmdlet used to load .NET assemblies into your session, which is necessary for accessing classes that interact with SQL Server.
Step 2: Creating the DataTable
With the assembly loaded, we can now create a DataTable object:
$dataTable = New-Object System.Data.DataTableThe New-Object cmdlet is used here to create an instance of the DataTable class. DataTable is an in-memory representation of a single database table.
We then define the columns that our DataTable will have:
$dataTable.Columns.Add((New-Object System.Data.DataColumn 'ID',([int])))
$dataTable.Columns.Add((New-Object System.Data.DataColumn 'Value',([string])))Each DataColumn object represents a column in the table, where ‘ID’ is of type int and ‘Value’ is of type string.
Step 3: Populating the DataTable
Next, we populate the DataTable with random data:
$random = New-Object System.Random
for ($i = 1; $i -le 10; $i++) { $row = $dataTable.NewRow() $row['ID'] = $i $row['Value'] = "RandomValue_" + $random.Next(1, 100) $dataTable.Rows.Add($row)
}Here we’re using a for loop to create 10 rows of data. NewRow creates a new row that matches the schema of the DataTable. We fill the ‘ID’ column with a sequential number and the ‘Value’ column with a random number prefixed by a string.
Step 4: Establishing a SQL Server Connection
Now, we’ll establish a connection to the SQL Server:
$connectionString = "Server=myServerName;Database=myDatabaseName;Integrated Security=True;"
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection $connectionStringThe connection string contains the information needed to connect to the database. Integrated Security=True is used for Windows Authentication. Replace myServerName and myDatabaseName with your server and database name.
Step 5: Using SqlBulkCopy to Insert Data
Finally, we use SqlBulkCopy to insert the data from our DataTable into the SQL Server table:
$sqlBulkCopy = New-Object System.Data.SqlClient.SqlBulkCopy($sqlConnection)
$sqlBulkCopy.DestinationTableName = "myTableName"SqlBulkCopy is a class that allows for efficient bulk operations. We specify the destination table name with $sqlBulkCopy.DestinationTableName.
To execute the bulk copy:
try { $sqlConnection.Open() $sqlBulkCopy.WriteToServer($dataTable) Write-Host "Data has been successfully exported to SQL Server table."
} catch { Write-Host "Error: $_"
} finally { $sqlConnection.Close()
}Before we execute the script we need to make sure our destination table exists. Here is how you’d create our sample table using TSQL:
CREATE TABLE myTableName ( ID int NOT NULL PRIMARY KEY, Value nvarchar(255) NOT NULL
)The script is provided in its entirety below for your convenience:
# Load ADO.NET assembly
Add-Type -AssemblyName "System.Data"
# Create a DataTable
$dataTable = New-Object System.Data.DataTable
$dataTable.Columns.Add((New-Object System.Data.DataColumn 'ID',([int])))
$dataTable.Columns.Add((New-Object System.Data.DataColumn 'Value',([string])))
# Populate DataTable with random data
$random = New-Object System.Random
for ($i = 1; $i -le 10; $i++) { $row = $dataTable.NewRow() $row['ID'] = $i $row['Value'] = "RandomValue_" + $random.Next(1, 100) $dataTable.Rows.Add($row)
}
# SQL Server connection string
$connectionString = "Server=myServerName;Database=myDatabaseName;Integrated Security=True;"
# Establish SQL Server connection
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection $connectionString
# SQL Bulk Copy to insert data into the table
$sqlBulkCopy = New-Object System.Data.SqlClient.SqlBulkCopy($sqlConnection)
$sqlBulkCopy.DestinationTableName = "myTableName"
try { # Open SQL Server connection $sqlConnection.Open() # Write from the source to the destination $sqlBulkCopy.WriteToServer($dataTable) Write-Host "Data has been successfully exported to SQL Server table."
}
catch { Write-Host "Error: $_"
}
finally { # Close the SQL Server connection $sqlConnection.Close()
}Cmdlets and Classes Used
- Add-Type: Loads a .NET assembly into the PowerShell session.
- New-Object: Instantiates a .NET class. Used to create the DataTable, DataColumn, SqlConnection, and SqlBulkCopy objects.
- System.Data.DataTable: Represents one table of in-memory data.
- System.Data.DataColumn: Defines the schema of a column in a DataTable.
- System.Random: A .NET class that generates random numbers.
- System.Data.SqlClient.SqlConnection: Establishes a connection to SQL Server.
- System.Data.SqlClient.SqlBulkCopy: Allows for the bulk transfer of data into SQL Server.
By understanding these concepts and cmdlets, you can manipulate data structures in PowerShell and interact with SQL Server to perform a variety of data operations, making your scripting tasks more efficient and powerful.
Conclusion
And that’s it! You’ve now learned how to create a DataTable in PowerShell, populate it with data, and insert that data into a SQL Server table with SqlBulkCopy. This process is particularly useful for automating the insertion of large volumes of data into a database, making your data processing tasks more efficient. We hope it also helps extend your understanding of PowerShell as well.
Often you may want to use PowerShell to insert a new line into a file after a specific matching pattern.
This particular example looks for the pattern “Celtics” in the lines of the file specified at the path in the $my_file variable and then adds a new line with the text “Some New Text” after this specific line.
If the string “Celtics” is not found on any line in the file then the new line will not be added.
Suppose we have a text file in our current directory named teams.txt that contains information about various basketball teams.
We can use the Get-Content cmdlet to view the contents of this file:

Suppose that we would like to find the line that contains the string “Celtics” and then insert a new line of text after this line in the file.

Notice that a new line has been added to the file after the line containing the string “Celtics”, just as we specified.
Note that we used the Set-Content cmdlet to set the content of the existing file by outputting all of the existing content to the file along with the new line of text.
Note: You can find the complete documentation for the Set-Content cmdlet in PowerShell here.
PowerShell: How to Use Export-Csv with Append
PowerShell: How to Use Export-Csv with No Headers
PowerShell: How to Count Number of Lines in File
Here’s some PowerShell and SQL logic to serve as a foundation for importing CSV data. While you may need to refine the data and logic to better suit your specific requirements, I trust you can handle those minor adjustments to achieve your desired outcome effectively.
Powershell
$domain = "company";
$servers = 1..4 | ForEach-Object { "$domain`dc0$_"}
$servers | ForEach-Object {
$z = "";
$z = Get-WinEvent -ComputerName $_ -FilterHashtable @{ Logname = 'Security'; Id = 4740 } -ErrorAction SilentlyContinue $Array = $z | ForEach-Object { [PSCustomObject]@{ "Time" = $_.TimeCreated.ToString("yyyy-MM-dd hh:mm:ss tt") "Account Name" = "$($_.Message.Split(":").Trim()[4].Split()[0])\$($_.Message.Split(":").Trim()[-3].Split()[0])"; "Caller System" = $_.Message.Split(":").Trim()[-1] "Domain Controller" = $_.MachineName.Trim().Split(".")[0] } } If ($Array) {$Array | Export-Csv "\\CompanySQL01\csv$\AccountLockouts\$((Get-Date).ToString("yyyy-MM-dd_HHmmss"))_CompanyADLockouts.txt" -NoTypeInformation -Append;} };PowerShell
Note: This script is executed on the Microsoft SQL Server itself, facilitating its insertion into the table through Task Scheduler for automation. The drive letters referenced here are local to the SQL Server and not presented as UNC paths from its perspective.
Get-ChildItem "E:\Csv\AccountLockouts\*.txt" -File | ForEach-Object { Invoke-Sqlcmd -Query "EXEC ImportCompanyDomainAccountLockout @filepath = '$($_.FullName)'" -ServerInstance "CompanySQL01" -Database "Reporting"; Sleep -Seconds 3; Copy-Item $_.FullName -Destination "E:\Csv\AccountLockouts\_Archived\$($_.Name)" -Force; If ( Test-Path "E:\Csv\AccountLockouts\_Archived\$($_.Name)" ) { Remove-Item $_.FullName -Force; } };
Get-ChildItem "E:\Csv\AccountLockouts\_Archived\*" | ForEach-Object { $_ | ForEach-Object { Process { If( $_.LastWriteTime -lt (Get-Date).AddDays(-90)){ $_.FullName | Remove-Item -Force } } }; };SQL Stored Procedure (MS SQL)
Note: This stored procedure is defined within the SQL Server database where you intend to import the data into its respective table. PowerShell script #2 executes this procedure to manage the data import process accordingly.
USE [Reporting]
GO
/****** Object: StoredProcedure [dbo].[ImportCompanyDomainAccountLockout] Script Date: 2/28/2024 9:27:24 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[ImportCompanyDomainAccountLockout] @filepath nvarchar(500)
AS
SET NOCOUNT ON DECLARE @bulkinsert NVARCHAR(MAX) SET @bulkinsert = N'BULK INSERT [dbo].[company_lockedoutaccounts] FROM ''' + @filepath + N''' WITH (FORMAT=''CSV'',FIRSTROW = 2)' EXEC sp_executesql @bulkinsert
GOPowerShell (MySQL Import)
$DBname = $args[0]
$SQLPass = $args[1]
$SvrName = "localhost"
$SName = "sqlloginaccount"
$SPort = "3306"
$sourceFolder = "\\companyesync1\Exports$\$DBname"
$delim = "#"
$destpath = "S:/Backups/$DBname/Import/Rawdata/tmpImport"
New-Item -ItemType Directory -Force -Path $destpath
$finalPath = "S:/Backups/$DBname/Import/Rawdata"
New-Item -ItemType Directory -Force -Path $finalPath
Get-Childitem -Path "$sourceFolder/*.csv" -Name | ForEach-Object { $nameArray = $_.Split($delim) $FName = $nameArray[0] Get-Childitem -Path "$sourceFolder/$FName*.csv" -Name | Sort-Object | ForEach-Object { Move-Item "$sourceFolder/$_" "$destpath" -Force $TblName = $FName $ImportFile = "$destpath/$_" ## -- SQL Logic to loop [system.reflection.assembly]::LoadWithPartialName("MySql.Data") $mysqlConn = New-Object -TypeName MySql.Data.MySqlClient.MySqlConnection $mysqlConn.ConnectionString = "SERVER=$SvrName;PORT=$Sport;DATABASE=$DBname;UID=$SName;PWD=$SQLPass;Allow User Variables=True" $mysqlConn.Open() $MysqlQuery = New-Object -TypeName MySql.Data.MySqlClient.MySqlCommand $MysqlQuery.Connection = $mysqlConn $MysqlQuery.CommandText =" USE $DBname; CREATE TABLE IF NOT EXISTS $TblName ( `TimeInt` VARCHAR(10) NOT NULL, `TimeStr` DATETIME NOT NULL, `IsInitValue` INT NOT NULL, `Value` DOUBLE NOT NULL, `IQuality` INT NOT NULL, UNIQUE KEY `uk_Times` (`TimeInt`,`TimeStr`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; LOAD DATA LOCAL INFILE '$ImportFile' REPLACE INTO TABLE $TblName FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' (TimeInt,TimeStr,IsInitValue,Value,IQuality) " $MysqlQuery.ExecuteNonQuery() Move-Item "$ImportFile" "$finalPath" -Force } }



