GitHub Issue Templates and Plaster

GitHub recently announced new features and support for multiple issue templates. Before, GitHub supported a single template, by supporting multiple types of issues, we can gather the right information from the beginning.

When we create multiple issue templates, users will be prompted to specify the type of issue they want to create. We can target specific questions or information requests for specific types of issues.

GitHub provides two pre-created templates, bug report and feature request. These pre-created templates suit most situations. Bug report issues can focus on reproduction, expected behavior and other troubleshooting information. Feature request templates can focus on understanding the what, how and why a new feature is required. You can also create your own templates for other situations, you don’t have to use what GitHub provides.

I've updated my Plaster template for PowerShell modules to contain a bug report and feature request template. I modified GitHub’s template to ask for things like: • PowerShell transcripts. • Operating System and PowerShell versions. • A list of any other modules that might imported.

You can create templates via the GitHub interface, or simply include them in the ISSUE_TEMPLATE folder in your repository. You can also create the templates yourself. You need to include the name and a description for each template. Much like this:

---
name: Bug report
about: Create a report to help us improve

---

So, what do the new templates look like for a user? If you select Issues > New Issue, you can now select what kind of issue you'd like to raise. In the case in the image, we can choose bug report or feature request. Users can opt for “Open a regular issue”, in this case GitHub will use ISSUE_TEMPLATE.MD.

You can also define multiple pull request templates in the same manner. Simply put each template into the PULL_REQUEST_TEMPLATE folder. The experience is like creating an Issue. I'm creating different pull request templates for Planet PowerShell, so we can correctly track changes.

If you want to start using these templates as part of your Plaster templates, you can see my examples over at my Plaster template.

Automating Office 365 deployments in CloudFlare

A few weeks ago, I wrote about Posh-CloudFlare, a PowerShell module I created for managing CloudFlare hosted domains. Since then, I was working on extending an Office 365 deployment, and realized that what was needed was a script which could automate the configuration of new domains. With that in mind, I developed a new PowerShell script,  Posh-Office365CloudFlare.

Let's understand the process for the addition and configuration of a new domain for Office 365.

The process starts with the Office 365 Portal. We navigate to the Domains section, click the "Add Domain" button, and after ignoring the introduction, we proceed to step 1. This step starts with us entering our domain name, let's use our old favorite contoso.com. Now we will be asked to verify that we own this domain, either through the creation of a TXT record or an MX record. The typical method is to use is that of a TXT record, created at the root of our desired domain with a value something like "MS=mx********".

After we create the domain, and the wizard successfully sees the appropriate record, we will be allowed to proceed to the next step. Step 2 isn't one that I usually make use of. I typically don't want to modify my users email domains, nor do I want to add new users at this time. I skip this step and move straight on to step 3.

Step 3 starts with another quick introduction screen, and then we will be asked if we would like the DNS for this domain to be managed by Microsoft. Obviously, we are going to answer no and move on. Finally, we reach an important step, we are asked what we want to do with this domain. First, "Outlook for email, calendar, and contacts", or in other words, email; the second, "Lync for instant messaging and online meetings", which is kind of obvious.

If you select “Outlook for email, calendar, and contacts”, then we will be told to create the following records in contoso.com:

• MX - @.contoso.com - which points to contoso-com.mail.protection.outlook.com (priority 0)
• CNAME – autodiscover.contoso.com – which points to autodiscover.outlook.com
• CNAME – msoid.contoso.com – which points to clientconfig.microsoftonline-p.net
• TXT – @.contoso.com – which contains a SPF record

If you select “Lync for instant messaging and online meetings”, then we will need to create the following records for contoso.com:

• CNAME - sip.contoso.com - which points to sipdir.online.lync.com
• CNAME - lyncdiscover.contoso.com - which points to webdir.online.lync.com
• CNAME – msoid.contoso.com – which points to clientconfig.microsoftonline-p.net
• SRV - _sip._tls.contoso.com - with its appropriate port, weight, priority and target
• SRV - _sipfederationtls._tcp.contoso.com - with its appropriate port, weight, priority and target

Reviewing this list of records, we will notice that the only record that changes for each domain is the MX record. The record consists of the domain name we want to add, with dashes replacing the original dots in the domain name. As you can see in the above example, cotoso.com's MX record points to contoso-com.mail.protection.outlook.com, where as awesomecompany.net would point to awesomecompany-net.mail.protection.com. 

What about some records that could actually help our users? What if I said we could redirect sub domains of our own to the Outlook Web Access page? Wouldn't it be awesome if a user entered https://mail.contoso.com into their browser, and ended up with the Outlook Web Access? This can be achieved by creating a CNAME record that points to mail.office.com. Let's have our script create entries for mail and webmail perform this redirection.

Now back to the script.

This was a simple script, it doesn't have any complex logic, it will need the following information:

• CloudFlare API Token and email address; this is obvious as we need to talk to the CloudFlare Client API.
• The domain name.
• Do we want to create mail records? Lync records or both?

This is a very, very simple script, we just need to have a set of New-CFDNSRecord calls, with various controls depending on what we require.

For example, creating the MX record is as simple as:

This script only took an hour or so for testing and development time, however there was quite a bit of effort directed to changes in the Posh-CloudFlare and the New-CFDNSRecord CMDLet. If you look at the diff's between the last few versions, you will notice the following changes:

• The CMDLet now accepts input from the pipeline (in this case via property name).
• Restructure the CMDLet into Begin/Process/End (required for proper handling of pipeline input).
• Implementation of parameter sets.
• Cleanup of the validation of parameters.

I added parameter sets to New-CFDNSRecord with the aim to remove the somewhat faulty validation that I had previously. Whilst this sounded, and looked like it was simple, it actually took a few tried to ensure that the CMDLet would function appropriately. This was really interesting and deserves its own post in the future.

Parameter validation was updated in all of the CMDLets to improve email address validation. Previously, validation consisted of testing for an "@" character. Now I am using a regular expression.

Finally, I have spent some time cleaning up the code, not just within New-CFDNSRecord, but across all of the CMDLets. I have been trying, where possible to use ISE Steroids to ensure that everything I right is neat and presentable; it is a fantastic resource.

My final thought on all of this journey is, why couldn't Microsoft have implemented something like this? Microsoft has integrated the process with a bunch of other DNS providers, including the likes of GoDaddy, Network Solutions, 1 and 1 and even Yahoo Small Business. Why can't it also look at CloudFlare?

You can find the finished script over at GitHub, at Posh-Office365CloudFlare, the script is called Register-Office365.ps1. I have included comment based help with examples.

Kieran Jacobsen

Revisiting Syslog in PowerShell

I have performed a number of updates to the PowerShell SYSLOG module since this post. You can read the latest post. The module has been renamed to Posh-SYSLOG.

The GitHub location has been moved to https://github.com/poshsecurity/Posh-SYSLOG.

The module is now available on the PowerShell Gallery.

---

I often wonder, as I am sure most developers do, if people ever actually read and use the code that I post online. Was it helpful to them or was it useless? Did they use it for something interesting? One piece of code which I know people do use, is my PowerShell SYSLOG code.

A few weeks ago, a user opened my very first GitHub issue! This issue appeared at first to be simple, but as the user and I started to delve into the complexities of the various SYSLOG RFCs, I realized it was far from it.

Before we get into the issue, the code and the resolution, it is worth highlighting that there quite a few IETF RFCs that relate to SYSLOG messages. The two primary ones being:

• RFC 3164 - BSD SYSLOG. This actually wasn't an IETF standard.
• RFC 5424 - IETF SYSLOG. This is a IETF standard. This obsoletes RFC 3164.

There are also RFCs:

• RFC 3195 - Reliable Delivery for SYSLOG
• RFC 5425 - TLS Transport Mapping for 
• RFC 5426 - Transmission of SYSLOG Messages over UDP
• RFC 5427 - Textual Conventions for Syslog Management
• RFC 5848 - Signed Syslog Messages
• RFC 6012 - Datagram Transport Layer Security (DTLS) Transport Mapping for SYSLOG
• RFC 6587 - Transmission of SYSLOG Messages over TCP

I want to send a very big thanks out to the user, DFCH for reporting the issue, helping me understand the RFCs in question and also testing the resulting code.

The Issue

So what was the issue? As DFCH stated:

“The Cmdlet send-syslog.ps1 states in its description to send a syslog message as defined in RFC 5424. However the generated timestamp in the Cmdlet incorrectly formats a timestamp when none is specified by the caller, nor does it validate or convert the timestamp if specified by the caller.”

— https://github.com/kjacobsen/PowerShellSyslog/issues/1

I will admit that I hadn't ready RFC 5424 or RFC 3164 in a huge amount of detail. As soon as I did it was very obvious that the code was not producing an appropriate timestamp, it also become evident that the overall message I was sending did not meet the RFC specification.

From my analysis, it appeared that I had crossed parts of both RFC 5424 and RFC 3164, ending up with code that wasn't fully complaint to either, and in the long run, not entirely useful.

As DFCH reported, the code didn't not generate the appropriate timestamp, with issues in how it was formatted as well as the precision. Resolving these issues was quite simple, the timestamp could be formatted as recommended by DFCH, this not only resolved the format issue but also increased the precision. Validation of the caller specified timestamp was also easy to implement. I simply changed the parameter to take an object of type DateTime instead of a String. 

But this was just the start of the fixes, as I continued to read and understand the RFCs, I realized my messages were incorrectly formatted as well.

Message Formats

There are two valid SYSLOG message structures as defined in RFC 3164 and 5424. 

Firstly, RFC 3164 specifies the message structure to be the following:

<PRI>TIMESTAMP HOSTNAME TAG CONTENT

Where:

• PRI - Value based on severity and facility
• TIMESTAMP - What date and time with format MMM dd HH:mm:ss
• HOSTNAME - Who is sending the message
• TAG - Name of the process or program generating the message
• CONTENT - Obviously the message being sent

Next, RFC 5424 specifies the message structure as:

<PRI>VERSION TIMESTAMP HOSTNAME APPNAME PROCID MSGID STRUCTUREDDATA [CONTENT]

Where:

• PRI - Value based on severity and facility
• VERSION - Version of the SYSLOG message (typically 1)
• TIMESTAMP - What date and time with format: yyyy-MM-ddtHH:mm:ss.ffffffzzz
• HOSTNAME - Who is sending the message
• APPNAME - Name of the process or program generating the message
• PROCID - Process ID of the application or script
• MSGID - An Identifier to assist in troubleshooting
• STRUCTUREDDATA - RFC 5424 specifies a method of sending key/value pairs
• CONTENT - Obviously the content of the message

One thing to note with RFC 5424 is that the majority of the fields are optional, you still need to send something to ensure the correct layout however, so the nil value "-" is sent. I should also point out that the RFC states that the CONTENT section at the end is completely optional. If nothing is sent, you don't need to even send the nil value. 

My original code on the other hands, was sending messages with the structure of:

<PRI>TIMESTAMP HOSTNAME CONTENT

Where:

• PRI - Value based on severity and facility
• TIMESTAMP - What date and time with format: yyyy:MM:dd:-HH:mm:ss zzz
• HOSTNAME - Who is sending the message
• CONTENT - Obviously the message being sent

How did this happen? 

Well, there are a few reasons why this occurred, in no particular order.

1. I borrowed some of the logic and ideas from other .Net and PowerShell code samples
2. I didn't read either RFC
3. The SYSLOG servers I tested against were not stringent in their rendering of messages received.
4. I referred to Wikipedia when I was checking that my messages were correctly formatted.

In hindsight, the biggest mistakes were using Wikipedia as my guide, and not testing against a more RFC compliant server.

Using Wikipedia as a source for developing compliant code is probably a bad idea, on this occasion it was a great learning experience. Previously the SYSLOG Wikipedia article did not correctly describe the layout and formatting of the full message, crucially missing out the information about the TAG field. The article has been updated since then with corrections ensuring that it is more understandable. It should be noted however that overall, the Wikipedia article is still focused on RFC 3164 and not 5424.

Let’s look at how we clean-up the code.

Additional Parameters

To ensure that we have enough information to support RFC 5424, I needed to add some additional Parameters (which are not mandatory). These include ApplicationName, ProcessID, MessageID, StructuredData and a switch RFC 3164.

The switch RFC 3164 will simply tell the code to send a message in the RFC 3164 format, instead of sending it via RFC 5424 which is its default.

Hostname Generation

Previously, if the hostname parameter was not specified in my code, I simply used the hostname.exe. Whilst there isn't any problems with this, however RFC 5424 actually specifies in some detail how the hostname field should be determined:

1. The FQDN of the server
2. A static IP address
3. The hostname of the server (Windows will always have on of these)
4. Dynamic IP address
5. A NILVALUE (-)

I have updated my code to generate the hostname component of the SYSLOG message via the first 3 steps.

Application Name

The Application name can be a little difficult. Typically from within a function we can determine the name of the script which is calling the function via 2 properties of the $myInvocation variable: ScriptName and PSCommandPath. I have used ScriptName with success in the past, and hence decided to use it again. There is one thing to note, if I am sitting at a console and call send-syslogmessage, then ScriptName will be null, and if that is the case, we will simply use “PowerShell”.

Process ID 

The Process ID is new requirement to ensure RFC 5424. We get this simply from the $PID global variable.

Message ID and Structured Data

These two will always be user specified, if the user doesn't specify them, then send the default RFC 5424 nil value of "-".

Message Generation

Now that we have all of the information required for either RFC, we can now look at message generation. When it comes time, I simply have an ‘if’ statement that controls which format we want to use. The script will then format the timestamp and message accordingly.

Message generation looks like the following:

For RFC 3164, I fixed up the timestamp, and also added in the application name. For RFC 5424 there are some significant changes. I now correctly include the SYSLOG version (1), and then included the corrected timestamp, application name, process ID, message ID and structured data.

The Future

If there was a demand, I would be interested in extending the CMDLet to support the transmission of messages via TCP, as well as sending signed messages. Right now, I don't have a need for such things.

Conclusion

Now that all of those changes have been completed and tested, I have pushed the changes up to the PowerShellSyslog GitHub repository.

I want to thank DFCH again for raising the issue and helping me through the development of the fixes.

Kieran Jacobsen

Posh-CloudFlare managing CloudFlare using PowerShell

The aim of the Posh-CloudFlare module is to simply and automate the management of CloudFlare hosted DNS zones using PowerShell and the CloudFlare Client API. I have made the module available via the PoshSecurity GitHub, here Posh-CloudFlare.

I started looking at CloudFlares API several months ago, as part of another post which I am still working on. Back then I was simply looking at the creation and deletion or records.

Things changed when I found that I needed to spend quite a bit of time working with DNS. Provisioning new infrastructure within cloud environments is something I spend a significant amount of time doing, and am actively investigating the automation of it, and as such, become interested in other parts of the API.

This module now implements all of the Client API, with 22 CMDLets in total. To simplify things, I have documented what CMDLet maps to what API call below:

CMDLets	API Actions
get-CFDNSZoneStatistics	3.1 - "stats" - Retrieve domain statistics for a given time frame
get-CFDNSZone	3.2 - "zone_load_multi" - Retrieve the list of domains
get-CFDNSRecord	3.3 - "rec_load_all" - Retrieve DNS Records of a given domain
get-CFDNSZoneStatus	3.4 - "zone_check" - Checks for active zones and returns their corresponding zids
Get-CFIPThreatScore	3.6 - "ip_lkup" - Check threat score for a given IP
get-CFDNSZoneSettings	3.7 - "zone_settings" - List all current setting values
Set-CFDNSZoneSecurityLevel	4.1 - "sec_lvl" - Set the security level
Set-CFDNSZoneCacheLevel	4.2 - "cache_lvl" - Set the cache level
Set-CFDNSZoneDevMode	4.3 - "devmode" - Toggling Development Mode
Clear-CFDNSZoneCache	4.4 - "fpurge_ts" -- Clear CloudFlare's cache
Clear-CFDNSZoneFileCache	4.5 - "zone_file_purge" -- Purge a single file in CloudFlare's cache
Add-CFBlackListIP Add-CFWhiteListIP Remove-CFListIP	4.6 - "wl" / "ban" / "nul" -- Whitelist/Blacklist/Unlist IPs
Set-CFDNSZoneIPVersion	4.7 - "ipv46" -- Toggle IPv6 support
Set-CFDNSZoneRocketLoader	4.8 - "async" -- Set Rocket Loader
Set-CFDNSZoneMinification	4.9 - "minify" -- Set Minification
Set-CFDNSZoneMirage2	4.10 - "mirage2" -- Set Mirage2
New-CFDNSRecord	5.1 - "rec_new" -- Add a DNS record
Update-CFDNSRecord	5.2 - "rec_edit" -- Edit a DNS record
Remove-CFDNSRecord	5.3 - "rec_delete" -- Delete a DNS record

The Client API can be a little tricky at first, I have developed the CMDLets in a manner to simplify the learning curve. Typically any API call which modifies or removes a DNS record, would require a rec_id to be specified. This field can be found by querying all of the records in the zone. I have simplified things by performing the search and other API queries for you. You can still specify a rec_id if you like.

Switches and parameter validation sets have been used to simplify some of the other CMDLets, particularly those around minification, security and other zone wide settings.

Finally I have tried where possible to make good use of the Pipeline. There are still a number of areas that could be improved.

Getting Started

The first thing you will need to do, is obtain your API Token. This can be found on your Account page. You will need this, and the email address you use to sign into CloudFlare for the majority of the CMDLets. For CMDLets which modify DNS Zones or records, you will need to specify the zone as well.

To obtain the module, simply perform a git clone to your preferred module location as below:

I have included a demo script, Posh-CloudFlare-Demo.ps1 at the root level of the module, which you can run on the namespace of your choice. I recommend not using your corporate production domain. At the top of this script, simply update the API Token, Email and domain name fields as required.

You can then run the script, and see it manipulate the DNS zone. I am not responsible if this breaks production. This script shows you each CMDLet and it's output. I don't recommend simply running the script, I recommend stepping through each line so you gain more of an understanding.

Potential Uses

The automatic provisionment of cloud hosted environments is why this was developed as well as another project I will announce in the coming future. For now, I see myself working on at least one module to support the automation of Office 365 provisioning, including creating the TXT, MX and SRV required.

Warnings

Firstly, I haven’t finished up the PowerShell help – Naughty! I will work on this one as I go.

Secondly, there might be some bugs. Whilst I have tried to test the majority of the permutations of the code, I can’t be fully sure I haven’t missed something. If you find one, please feel free to contact me and I will make the required fixes, or even better, push your updates up to GitHub.

Kieran Jacobsen