Announced at DataGrillen 2019 today, the amazing dbatools PowerShell module has officially released version 1.0. This is a tremendous milestone for the best Open Source project built for data professionals.
By the 4th Invoke-DbaQuery, I found myself thinking “this repetitive typing kind of sucks.” Then I remembered Chrissy LeMaire’s segment in the first PSPowerHour where she talked about default values, and her accompanying dbatools blog post. Most of the blog posts and demos of this feature focus on using it from the command line, so I had overlooked the fact that I could use it from within a script as well, and even change the values when looping.
As it turns out – it works inside scripts and functions as well, and can make them a lot easier to read. And you’re not limited to the default parameters every time you call a given function; you can override the defaults by specifying the parameters when you call it.
$PSDefaultParameters is a hash, and the name/value pairs take the format of 'FunctionName:ParameterName' = ParameterValue.
I set the destination database as the default for Invoke-DbaQuery because there will be a number of additional queries I need to execute against that database. In Step 1, I’m using the parameter defaults for both Databaseand SqlInstance, so I don’t need to specify them when calling the function. In Step 2, I need to override only Database, so I’m just specifying that one parameter.
Word of warning: Because $PSDefaultParameters is a hash, you might want to “build up” defaults over the course of a function, script or session. In that case, get in the habit of appending to the hash so that you don’t overwrite the defaults you’ve previously set.
If you’re already splatting parameters, this is a feature you can take advantage of to take the next step in tidying up your scripts and making them easier to read & manage.
In my previous post, I mentioned that I wasn’t checking the status of my RSJobs because I had logging built into my function. I originally tried to log out to plain text files with Out-File, but quickly hit a wall. In a multi-threaded script, attempting to write to the same text file from multiple threads results in collisions and waits at best, and outright failure at worse. Perplexed, I turned to the PowerShellHelp channel in the SQL Community Slack and asked for suggestions for thread-safe logging to text files.
PowerShell has had a native method for spawning multiple “threads” ever since I can remember, in the form of the *-Job functions. They work OK, but there are a couple downsides:
Each job is its own PowerShell process, so it takes a non-trivial amount of time and memory to spin each up
There’s no built-in method for throttling the number of concurrent jobs
This combination will become an ugly mess if something spins out of control and you spawn dozens or hundreds of jobs. PowerShell jobs are better tailored to small-scale asynchronous background processing. You can wrap the functions to limit the number of concurrent jobs but again, there’s a lot of overhead involved in creating and tearing down jobs. Warren Frame created Invoke-Parallel, which uses runspaces (lighter-weight than jobs) and allows for throttling, but isn’t quite as full-featured as jobs are.
Boe Prox comes to the rescue with his PoshRSJob module. This module provides runspace-based functions that mirror PowerShell’s native job functions to create – Get, Receive, Remove, Start, Stop, and Wait – plus some additional functionality like the ability to import modules (without needing Import-Module inside the Scriptblock) and throttling the number of concurrent jobs running.
The above is a fairly trivial script but it serves the purpose of demonstrating multi-threading with this module.
In a loop, I’m creating two dozen RSJobs to fetch the PoshRSJob module from GitHub and write it out to a directory on my computer.
I’ve set a throttle limit of 3; while all 24 jobs will be created, only three will run at any given time and the remainder will be in a NotStarted state, patiently waiting their turn.
I’m passing value of the loop counter and my output directory into the job’s scriptblock so that I can write out 24 individually-named files.
So that the RSJob system can keep track of the jobs properly, I’m lumping them into a named Batch.
Because the module’s zip file is so small, I introduced a bit of a delay into each job. Why?
To give me time to capture screenshots while the jobs are in various states.
To prevent tripping GitHub’s anti-flooding protections.
So let’s fire this up!
I’ve started up the script and it’s created all the RSJobs, but only the first three are running because that’s where I set my throttle limit. Let’s check in on the progress with Get-RSJob – all the same concepts you’re used to with regular PowerShell jobs (receiving, removing, checking for errors, etc.) work with RSJobs too!
You still need to clean up after yourself! Don’t forget to Receive-RSJob and then Remove-RSJob. I have something special for that.
Let’s say you’ve got a lot of RSJobs. Or maybe long-running RSJobs. Wouldn’t a nice, user-friendly view of the progress be handy? You bet it would!
This is both dense and verbose, but I wrote it really quickly and it works (but not just for this post, I’ve used it in production), so I’m not tinkering too much with it right now. Once the jobs are created (and a few have started), this script runs a loop watching the output of Get-RSJob and updates a progress bar to reflect where we are. It also takes care of receiving and removing the completed jobs.
It’s true that I’m not checking for any errors in my job output. In the production process that I multithreaded, the Scriptblock inside each thread does its own error logging to a file.
Did you notice the paths I’m writing to and the terminal theme? Yeah, this works on PowerShell Core too. Everything you see here ran on macOS as I was writing this post.
For light to moderate use, PoshRSJob is a really handy way to multi-thread processes that aren’t natively multi-threaded in PowerShell. It doesn’t seem to scale well to hundreds of threads, though. There is an open issue that seems to be related on GitHub, but I’ll discuss my own experience with it on a near-future post.
Pat Phelan replied, suggesting that dbatools can do it, but after thinking on it for a bit and poking at a few functions, I realized that it’s not possible with a single function (yet). No worries. We can do it in three lines for now.
Grab the table from the source database and export the create script. I had to use -PassThru because otherwise, Export-DbaScript will create a file. Not the worst thing, but writing the file and the reading it immediately afterwards is a bit messy. Although now that I write that out, I suppose I could capture the filename that’s output and pass that to -QueryFile in the next step. Gotta love PowerShell – there’s always a few ways to do things.
Take the CREATE TABLE script that we just exported from the source database and run it against the destination.
Run Copy-DbaDbTableData to bring the data over from the source to the destination.
This is a really quick & dirty example where I’m copying between two databases on the same instance, but should give you the idea. You can copy across instances and copy multiple tables as well.
Cláudio Silva (blog | twitter) has written a post that expands on the above code to include scripting out other objects related to the table being copied. The code above overlooks keys, constraints and more. Check it out!
The first edition of the PSPowerHour is in the books and it looks like it was a big success. This one was dbatools-heavy but I chalk that up to the dbatools community having lots of free time because we’ve automated so many of our tasks 🙂
I signed in about half an hour ahead of the webcast and was the first one there. Shortly thereafter, I was joined by Michael Lombardi (t, then Jess Pomfret (b|t) and Chrissy LeMaire (b|t). After ironing out a few glitches, we got everyone in the right place and kicked off the broadcast. Everything ran very smoothly, especially considering the number of people involved – Michael and Warren F. (b|t) did a terrific job of orchestrating everything.
While watching and listening to Chrissy, Doug, Andrew & Jess give their demos, I ran through my own in my head a couple times, adding and rearranging a few things as I observed how they were doing theirs. The big dilemma for me was whether or not to run the camera or exclusively screen share (I ended up going with the screen share only). Having not rehearsed my demo enough in the weeks leading up to the event, I was still not sure where to dip into more detail or dial things back and seeing what others were doing helped quite a bit. Having familiar faces & voices ahead of me in the queue put my nerves to rest.
I wasn’t able to watch the sessions after mine in their entirety due to family commitments. Joshua’s Burnt Toast module looks like it’ll be fun to experiment with and add some nice functionality to scripts (I got to see about half of his demo), and I’m really looking forward to catching a replay of Daniel’s demo of PowerShell on the Raspberry Pi – I didn’t realize that it had been ported already!
I demoed Invoke-DbaSqlQuery and why one should use it over Invoke-SQLCmd – primarily for protection from SQL injection. Things didn’t go exactly the way I’d practiced; I ran short of time despite feeling like I rushed things and cutting back on some of what I had planned to say. The latter was in part because of the lead-ins from Chrissy, Andrew, and Jess. Because they did such a good job introducing dbatools, I was able to skip over it. But I was able to throw in a teaser for Matt Cushing’s (b|t) demo at the next PSPowerHour.
Running the demos inside a VM and screen-sharing just that VM made things easier for me as opposed to flipping between apps. My scripts will be available on GitHub along with the other presenters’ once the pull request is approved.
I achieved my goals:
I did it
I successfully demonstrated a SQL injection problem and explained why it’s so bad
I demonstrated how to make database queries from PowerShell both more reliable and safer
I learned about some new stuff that I desperately want to experiment with.
Next time around, I definitely need to rehearse more and get my timing down better but overall, I’m happy.
Strictly Necessary Cookies
Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.
If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.