Accessing the Internet from PowerShell

I had many potential titles for this series of posts, as yes, this is the first of a series. The aim of the series is to slowly guide you through interacting with web pages from within PowerShell, extending to downloading and uploading files, posting data to forms and interacting with WebDAV. My aim is to start by talking about each individual task, accessing a page, posting/uploading, download files, etc; cover off some of the issues we might encounter or what things to be on the look out for when working within a corporate environment (authentication and proxy servers) and finally bring everything together into 1) a working PasteBin scrapper that will look for the occurrences of certain works and notify us and 2) a PowerShell module which will allow us to quickly perform all of the above mention web/internet related tasks. I will provide a tonne of examples along the way to assist in helping you understand all of the different concepts that are involved.

WebDav on Apache

Sounds simple enough, but this is just an intro post for some other cool PowerShell related posts.

One thing I needed was a WebDav server, and well, why not use my handy little Qube server.

If you want to set up the Apache WebDav module (in this case on NetBSD) simply:

1. Ensure you have the apache DAV module installed
2. We need a folder for the DavLock database, i did the following:
   
3. Once you have the folders, add the following lines to your http.conf. Note this will assume that the folder webdav is stored wherever the rest of your htdocs are, so you need to make the folder or create a symbolic link from there to where you want WebDav files to be stored.
   
4. Restart/Reload the Apache Service
5. DONE!

Notes:

I should probably explain the lines we are entering in the httpd.conf file, as they will cause you some confusion if I don't.

The first line is specifying where we want the DavLockDB, in this case, we want the files to be called DavLock, in the folder we created earlier.

We then have the usual Apache location tags, in this case we want the location to be /webdav. Change this to suit your needs. You could also use Directory tags.

The next three lines are to do with authentication and authorisation. I only ever perform authentication with HTTP Digest (the first line), and I always where possible create a new authentication realm which Apache calls the AuthName. Finally we need to have a list of usernames and passwords, AuthUserFile will point to where we have a file of usernames and passwords setup in the correct method for Apache.

The file section, is something that was covered in the Apache documentation, and to be frank, I don't fully understand as the Apache configuration layout is still perplexing to me. What I know is that this line is limiting who can access the webdav location, in this case, valid-user means anyone who authenticates successfully and isn't a guest will have access. I am sorry I couldn't be any more helpful.

So now we have all of this sorted, I will be able to show you all soon, some of the cool PowerShell stuff I have been investigating. As a teaser, I have been looking at the various was PowerShell can integrate with web applications, as well as new methods for PowerShell to copy and save data that it creates.

I have also been working on some updates to my standard template, minor tweaks in formatting really, as well as some fun stuff around making the copy script I use every day an even more powerful and flexible tool.

PrimalScript 2012

There is a new version of PrimalScript 2012 available. As a long time user of PrimalScript for all of my scripting work, I was quite excited to see the latest version. 

From first impression, everything looks very polished and the Ribbion is very easy to use. Navigation is very quick and easy.

I am running the 30 day trial, apparently my upgrade assurance expired which is a bit of a shame, if there are no bugs, will definitely pay the small fee to upgrade to the latest version.

All in all, worth taking a look at, and still the best PowerShell IDE to date.

Change to code content

I hated how code was being displayed on my site, so you may notice I have started to change it.

All code snippets will now be stored in pastebin.org, and then displayed using embedded links.

Ping Monitoring

I made a script to ping an address and alert me if it doesn't succeed. As an added bonus, it will also alert if the response time becomes to large. Once again, this is based on the usual template.

Enjoy.

Restart stopped services which are set to automatic

This script is handy for your Exchange 2010 deployments, it will find any automatic service which is stopped and start it (optional) and let you know that it happened. This is similar to a previous scripts I have posted.

As usual it uses my usual template.

Why you reboot?

I wanted to know when and by what/who/why servers rebooted. This is just a simple script, and you configure the it to run at start up in Windows task scheduler.

XML + PowerShell = Really neat config files

There are often times where you need to store data within a file to make your script work, examples include:

Complex configuration information
Historical information
Complex data storage
There are a number of ways you can address this when writing a normal .Net application, but the majority of these can easily be used within Powershell. These options include simple text files, SQL databases and what I want to talk about today, XML files!

Using XML files is very simple in PowerShell, most of the hard work will be performed by the Powershell engine and the .Net libraries it is built upon.

Example XML File

Reading data within an XML file
To read an XML file into a variable, simply

Now if we wanted to access all of the XML elements under the "Folders" element, all we would have to do is:

But if there are no "folder" elements below "folders", you will have a nice error message when you go to run that foreach loop! You can resolve this by testing if there are elements under the "folders" element by:

if the above returns false, then there are no elements sitting under "folders". Note that this command is CASE SENSITIVE!!! folders and Folders are not the same. BEWARE!

This is enough for today, and should allow you to understand some of the scripts I will be posting up soon.