Accessing the Internet from PowerShell: Download Files Part 2
So, last time we looked at download a file, and looked at what a CMDLet to do this should look like.
And here it is:
Most of the layout of this CMDLet should be familiar to you. There is the comment based help, then the CMDLetBinding. We then have a list of parameters, URL is the only one which is mandatory, and the others are all optional. URL is also set to accept the value from the pipeline, this is the first step to meeting the first requirement of our CMDLet. The filename and directory parameters are for specifying the intended destination files name and location. Finally clobber is a switch, meaning it is either on or off. Clobber will be the way the user of the CMDLet tells it to overwrite files.
Next you will notice a Begin {} block of code. In here we are going to be creating the WebClient object and setting all of its options. Begin{} blocks will only get executed once, no matter how many items are passed in on the pipeline. Begin{} will occur before any input data is processed. End{} which isn’t used, is only executed once all the processing is completed.
A Process{} block is next. Process{} will be executed for every item passed in on the pipeline to the CMDLet. In this block we spend time working out the destination filename and path, then we will test to see if it exists. If the file already exists, then we can either, raise a warning that we are overwriting the file, or throw an error. This will be determined by the clobber parameter.
Finally we are going to write the full url and destination path to verbose output and download the file. We will throw any errors encountered back to the calling code.
Let’s look at how to use the cmdlet:
So, now you have it. You can download files.
Accessing the Internet from PowerShell: Download Files Part 1
Today we are going to download files from the internet, and other few places as a continuation of the series on the WebClient class in the .Net framework. The WebClient class has a method called DownloadFile, which as the name says, allows us to download files from many sources including HTTP, HTTPS, FTP and \\server\share locations. The aim will be by the end of this to have a CMDLet allowing us to download files from these various locations.
So, how does the DownloadFile method work? Here is an example:
The only issue with DownloadFile is that we need to specify a filename as the destination. This is something we will need to work around/with.
So if we wanted design a CMDLet around this, what should we consider? Firstly, let’s just ignore headers and encoding types and all that junk and assume we will handle that. What else is there? Well for a CMDLet that downloads files, we probably want to be able to use it in the pipeline, that is, I might want to pipe an array or list of URLs to that CMDLet and have it download them all. Next I don't want to have to specify the destination file name and folder all the time, but that doesn't mean I might not want to specify one or the other down the track. Finally, I hate downloading a file twice, or accidently overwriting a destination file. I want the CMDLet to not overwrite files unless I tell it to, and if it is, warn me about it.
So here are the requirements:
- · We want to be able to pipe URLs to the CMDLet so they can be downloaded
- · Destination filename optional. CMDLet to determine filename if none specified
- · Destination folder/directory optional. We might want to send a file to a different directory than the current working directory.
- · By default, do not overwrite files. If we tell the CMDLet to overwrite files, it should warn us
Well supporting the pipeline when writing a CMDLet is easy, we simply use the begin{} process {} end {} syntax. If we do things in a clever way, we should be able to reduce the amount of work we are doing for a large list of downloads.
The second requirement, optionally specifying the filename to download the file to...that can be a little more difficult! The trick is to have two parameters, filename and directory. Depending on what is or isn’t specified, we can do 4 different things.
- 1. If both are specified -> join the two together
- 2. If no filename or destination directory is specified -> the destination is the current directory (converted from .) joined with the "leaf" part of the URL
- 3. If no filename is specified, but a directory is -> the destination is the specified directory joined with the "leaf" part of the URL
- 4. If filename is specified but a directory is not -> The destination is the current directory (converted from .) joined with the specified filename
The final requirement, not overwriting files by default, is also pretty simple, we can test to see if a file exist, if it doesn't then we are ok to download the file to the destination, otherwise we need a switch parameter, something like -clobber, to tell the CMDLet if its ok to overwrite the file. If it’s not ok, we will throw an error, if it is, we will use the write-warning to make sure we know that a file is being overwritten.
We will look at the cmdlet, next time.
Advanced Parameter Validation (revisiting get-filehash)
I am currently working on the rebuild of my template for all of my PowerShell scripts, trying to do things "better" this time around. I have learnt a few things over the last hour, one of which I don't understand (but everything works if I do it, so I will keep doing it), and another I just had to write about.
What is the great new thing? Well I wish i had known about it earlier, its the advanced parmater validation features.
Let's take a look at the first few lines of the get-filehash (original article, update) method I wrote about previously.
Now, I removed the comment based help syntax for brevity, but it is a little long, and there is the annoying enum defining the valid hash values to deal with as well, and then the section validating the file is a path etc. What if we could clean this all up?
Lets see:
Its a lot cleaner!
What we have is a ValidateScript, which is performing a test-path on the input. I am using the -pathtype 'leaf' to ensure we are hashing a file, not a folder. Then we have a ValidateSet, here we are saying, lets confirm the value for that parameter is in this array.
If either of these validations fail, we will get an error message along the lines of:
Get-FileHash : Cannot validate argument on parameter 'File'. The "Test-Path $_ -PathType 'leaf'" validation script for
the argument with value "D:\WIP\powershell\FileHash\filehash.jljdklf" did not return true. Determine why the
validation script failed and then try the command again.
At line:1 char:20
+ Get-FileHash -File D:\WIP\powershell\FileHash\filehash.jljdklf
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Get-FileHash], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationError,Get-FileHash
Get-FileHash : Cannot validate argument on parameter 'File'. The "Test-Path $_ -PathType 'leaf'" validation script forthe argument with value "D:\WIP\powershell\FileHash\filehash.jljdklf" did not return true. Determine why thevalidation script failed and then try the command again.At line:1 char:20+ Get-FileHash -File D:\WIP\powershell\FileHash\filehash.jljdklf+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidData: (:) [Get-FileHash], ParameterBindingValidationException + FullyQualifiedErrorId : ParameterArgumentValidationError,Get-FileHash
Not bad is it?
Another option to check out is ValidateRange. As you can guess, it allows you to validate the parameter against a range. An example would be [ValidateRange(1,10)], which will validate the paramater to ensure it is between 1 and 10!
Here is the new get-filehash in full...
Accessing the Internet from PowerShell: Get-WebPage Part2
Welcome back.
So last time we looked at the get-webpage CMDLet, but first, you may notice that it is different from this version of this CMDLet to that I showed off at Infrastructure Saturday; your right, it has been updated. The update is to make supplying user-agent headers more easily.
Let’s go back to the get-externalip function we defined a few months ago. Previously it was:
Now if we make use of the cmdlet we just finished:
Pretty neat?
I will leave it here, explore the cmdlet on your own.
Accessing the Internet from PowerShell: Get-WebPage Part1
So last time we looked at accessing the web from PowerShell, we looked a simple function which returned our external IP address, we then used that function to make a script to automatically alert us when our external IP address changed. This was a pretty simple function and script, but I want to spend some time discussing some of this further and build a full CMDLet.
In later posts I want to discuss how I believe you should design your PowerShell code, including CMDLets, functions, scripts and modules, but I want to touch on some of this today.
In the get-externalip function, we have simply created the .Net framework webclient object, set up some headers, and the called the downloadstring method. This is fine for this simple script, but it isn't great design in the long term. What we need is a CMDLet that downloads a URL from the internet (or maybe a local source) and then make use of that CMDLet. The CMDLet should enable us to have a powerful framework where we can easily leverage all of the webclient class functionality, but in a refined and controlled PowerShell manner.
We should also ensure that we provide all the help and documentation with our CMDLets, so anyone can use them. For this we will use the PowerShell Comment Based Help Syntax, something that is an over looked part of PowerShell. Seriously, Microsoft should be encouraging more use of this, every language has something like similar to this, it’s not something special in PowerShell, but if every PowerShell developer used the comment based help syntax, the world would be a much better place.
Let’s take a look at a CMDLet that will get the HTML representation of a page, this CMDLet will also allow us to set things like the proxy server, headers, credentials for the remote page and the user agent!
The CMDLet below is based upon my Infrastructure Saturday 2012 presentation on PowerShell.
Whoa, quite a bit here, around 100 lines of code! Most of which though is documentation. As you can see, we have the start of the function definition, then the comment based help syntax. This syntax is providing documentation for the get-help CMDLet, including descriptions, parameters, inputs, outputs and examples.
Next we have the CMDLetBinding, we are not specifying anything extra here, just telling PowerShell that this is a CMDLet.
After this we have parameters, there all look pretty simple, only one is mandatory, URL. I have also specified types for all of the parameters. Whilst the specification of types is option, I am not doing this wherever possible, it reduces errors and provides very useful input validation.
Now we have the body of the CMDLet. As expected, we are making a new instance of the WebClient class. Then we have some if statements. If these optional parameters have been specified, then we need to set the various parts of the WebClient object.
I hard set the encoding, it makes things simple an easy.
We then create an object to hold the result, and within a try {} catch{} block will call download string. If we do catch any errors, simply throw them to the calling code. We don’t particularly care about handling errors here, the calling code should; however its crucial to handle errors in a controlled manner.
Finally return the resulting page.
Next time we will look at some examples and some further discussion of the code.
Update to Hashing Post
Just decided to make the CMDLet in the article: PowerShell File Hashing , a little more clear by renaming it to Get-FileHash, it makes more sense in the long run in my head, and explains what the CmdLet does.
Git to Azure
Just a quick guide on publishing web content to Windows Azure using Git!
First things first, make sure you have TortoiseGit and its dependencies installed, and create a new Windows Azure web site in the management console.
For this example, we will be publishing out own mirror of the http://isitup.org site. The sites code is available from GitHub here: https://github.com/sjparkinson/isitup
Let's begin!
Part 1 - Configuring Windows Azure for Git
- Visit https://manage.windowsazure.com
- Select the website you wish to use Git to publish to
- Select "Setup Git publishing"
- If you have never specified deployment credentials, select "reset deployment credentials" and specify a username and password
- Note down the Git Clone URL
Part 2 - Pulling and Pushing with Git
- In your local working copy folder, right click and select "Git Clone"
- The URL will be "https://github.com/sjparkinson/isitup"
- The directory will be the base URL for the Git working copy/repository for me "c:\users\defaultusername\workinprogress"
- Hit ok
- wait for a sucess message, and hit close
- Right click on the isitup folder, select TortoiseGit, and then Push
- select "Arbitary URL" enter the URL noted earlier
- Hit OK
- Enter your password
- Wait for completion
- DONE!!!
One I prepared earlier http://isitup.azurewebsites.net
PowerShell file hashing
So, I was reading the Microsoft Security Response Centres post on verifying your Windows Update files, available here, and saw their comments about using PowerShell to verify the upates. They used some code from Mike Wilbur's Blog, here, which is a nice simple little bit of code to get your SHA256 hashes for files.
I decided that the world needed something a bit more, and started to see if we could make a cmdlet that used the pipeline, and also checked the faile we were passing to the cmdlet actually existed. Whilst I was pokeing around in the .Net framework, I also noticed that SHA1, SHA384, SHA512, MD5 and RIPEMD160 was also available to us.
So I spent some time, and have developed the following piece of code:
To use the code, copy the code above into a .ps1 file, and then import-module yourfile.ps1, the function is called get-hash.
As you can see, I am using the comment based help, and there are some nice examples included.