‘wget’ – Or ‘Get-WebFile’

Writing Chef Recipes has taken allot of my time recently. And for better or worse, 90% of the recipes need to be what we are calling ‘Platform Agnostic’ – That is, they need to be able to run on both Red Hat Enterprise Linux (RHEL) & Windows Kernel version 6.1 or greater (So, Windows 7 onward). This has had its challenges and when one platform is easy; the other tends to not be. One example is getting files from an HTTP(s) URL. Linux has multiple options depending on what you are trying to achieve (wget, curl), Windows also some options, but I wouldn’t call them straight forward. This is what has brought me to writing this latest function – Get-WebFile – Or realistically – wget for windows.

A few things to note:

  • I am using the comment based help for this function (as I said I would moving forward)
  • The functionaility is no where near as full as wget – But we can build this out as needed, so I won’t be stressing (yet)
  • I am using the correct naming convention for a Powershell function (incidentally I remember this by the fact the commands are all Verb-Noun, or “Action (to an) Object”. Verbs are the ones that Powershell really complains about). Here are the “approved verbs'” for Powershell

This uses the .NET class System.Net.WebClient to download the file using the DownloadFile method (Who would’ve guessed) – The Microsoft documentation is found here – And this will come in handy in the future as we want to expand the functionally of this function. I can see that there is a DownloadFileAsync method in there. This could be good to fire off multiple downloads and have them concurrently download (More on that later I reckon, but that can be another post).

So, if you can read the function, it has:

  • 2 Parameters
    • $URL – The lcoation of the file to download
    • $Output – The path/filename to output the download to – Defaults to the original filename and outputs to the CWD (Current working directory)
  •  Wraps up around the System.Net.WebClient .NET class
  • Uses the Get-Time so we can output a total amount of time the down load took
    • This should be expanded to give an average speed as well ($a = file size, $b = Time taken – So the calculation would be $a/$b = (Average speed) – But that ins’t a requirement just at this stage – Another post I reckon? 😀

function Get-Webfile {
<#
.SYNOPSIS
     Downloads a file from a URL and outputs it to a location 
.DESCRIPTION
     This function uses the System.Net.WebClient to download a file from an HTTP(S) URL. 
.NOTES
     Name       : Get-Webfile
     Author     : Sean Vucich
     Version    : 0.1     
     DateCreated: 2017-03-31
     DateUpdated: 2017-03-31     
     Blog       : https://1024hobbytes.com/ 
.LINK
     https://1024hobbytes.com/ 
.EXAMPLE
     To download a file from the location "http://example.com/myfile.txt" to the current working dir named after the original file downloaded Get-Webfile -URL "http://example.com/myfile.txt"
#>
param(
    [parameter(Mandatory=$true)]$URL,
    [parameter(Mandatory=$false)]$output=$null
    )
    # Get the current time for a nice little out put to console at the end of the download
    $start_time=Get-Date
    # This is just incase you didn't supply an output - This will just assume you want to 
    # output to the current dir naming the file based on the orignal URL file name
    if ($output -eq $null)
        {
            $spliturl = $url.split('/')
            $output = $spliturl[-1]
        }
    (New-Object System.Net.WebClient).DownloadFile($url, $output)
    Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
}

In action!

Now, how does it all look in action? Well, either give it a go, or have a look at this picture which runs the command and also an example from the comment based help 😀

Get-WebFile

HELP METADATA and New-EnvironmentVariable

The function I wrote and decided to use during this post wasn’t an absolute requirement, like a majority of my functions. And like most functions, the first few times it is used won’t provide a return on investment. But as I spend allot of time working with software that likes to use Environment Variables, it made sense to write the function to be called and used many times that would set these values for me!

Now, in the interest having this function used by other people in my team, I thought it best for personal and professional expansion to write some function help to assist others when using my function. I have made it a personal goal moving forward to ensure all of my functions have this data available.

HELP METADATA – about_comment_based_help

Help meta data isn’t required to get a function up and running, but it should be a personal/professional requirement to ensure that when someone else uses your functions that they have the information available to their finger tips. This is where the function METADATA comes in (Or as it is correctly known, about_comment_based_help).

I normally only worry about the ‘Synopsis’, ‘Description’ and a series of ‘Examples’. But here is a list of what is possible as per the above link. You will need to have a read and decide what you think you need:

  • .SYNOPSIS
  • .DESCRIPTION
  • .PARAMETER
  • .EXAMPLE
  • .INPUTS
  • .OUTPUTS
  • .NOTES
  • .LINK
  • .COMPONENT
  • .ROLE
  • .FUNCTIONALITY
  • .FORWARDHELPTARGETNAME
  • .FORWARDHELPCATEGORY
  • .REMOTEHELPRUNSPACE
  • .EXTERNALHELP

Each of these Metadata headings need to be placed in a comment block at the beginning of your function script, be written in capitals and prefixed with a full stop ( . ). Here is the synopsis for my latest function, New-EnvironmentVariable. You will notice all of the info is placed between <# ..... #>

<# 
.SYNOPSIS
     Function to add System & User based variables  
.DESCRIPTION
     Function to add permanent System & User based variables where the New-Item command only adds user based environment variables for the current session    
.EXAMPLE
     CREATES a new user based environment variable called "MYVAR_NAME" with a value of 'ValueOfEnvVar' for the current user context. 
New-SVEnvironmentVariable -Name 'MYVAR_NAME' -Value 'ValueOfEnvVar' -Type User     
.EXAMPLE
       UPDATES a user based environment variable called "MYVAR_NAME" with a new value of 'MyNewValue' for the current user context.  
New-SVEnvironmentVariable -Name 'MYVAR_NAME' -Value 'ValueOfEnvVar' -Type User -OverWrite     
.NOTES
       N/A   
#>

Now, this allows me to call some help for the function and have a read about usage, this uses the Get-Help command specifically with the ‘-Examples’ switch – Here it is in action:


C:> Get-Help New-EnvironmentVariable -Examples

NAME
New-EnvironmentVariable

SYNOPSIS
Function to add System & User based variables


-------------------------- EXAMPLE 1 --------------------------

PS C:\>CREATES a new user based environment variable called "MYVAR_NAME" with a value of 'ValueOfEnvVar' for the current user context.

New-SVEnvironmentVariable -Name 'MYVAR_NAME' -Value 'ValueOfEnvVar' -Type User




-------------------------- EXAMPLE 2 --------------------------

PS C:\>UPDATES a user based environment variable called "MYVAR_NAME" with a new value of 'MyNewValue' for the current user context.

New-SVEnvironmentVariable -Name 'MYVAR_NAME' -Value 'ValueOfEnvVar' -Type User -OverWrite

Now, here is the whole function, have a go at loading the function and pulling some help – This will server 2 purposes – have you try giving it a go while also cutting this post nice and short!:


<# 
.SYNOPSIS
     Function to add System & User based variables 
.DESCRIPTION
     Function to add permanent System & User based variables where the New-Item comand only adds user based environment variables for the current session 
.EXAMPLE
     CREATES a new user based environment variable called "MYVAR_NAME" with a value of 'ValueOfEnvVar' for the current user context.     
New-SVEnvironmentVariable -Name 'MYVAR_NAME' -Value 'ValueOfEnvVar' -Type User 
.EXAMPLE
     UPDATES a user based environment variable called "MYVAR_NAME" with a new value of 'MyNewValue' for the current user context.     
New-SVEnvironmentVariable -Name 'MYVAR_NAME' -Value 'ValueOfEnvVar' -Type User -OverWrite 
.NOTES
     N/A
#>

function New-EnvironmentVariable {
[cmdletbinding()]
param(
[parameter(Mandatory=$true, Position='0')][String[]] $Name,
[parameter(Mandatory=$true, position='1')][String[]] $Value,
[parameter(Mandatory=$true, position='2')][ValidateSet('User','Machine')][String[]] $Type,
[parameter(Mandatory=$false, position='3')][Switch] $Overwrite
)

$Overwrite.ToBool | Out-Null

If ($Overwrite -eq $true)
    {
    Try
        {
            $ErrorActionPreference = 'stop'
            [Environment]::SetEnvironmentVariable("$Name","$Value","$Type")
            Write-Host "$Type variable $name has been written with a value of $value"
        }
    Catch
        {
            $ErrorMessage = $_.Exception.Message
            $FailedItem = $_.Exception.ItemName
            $ErrorMessage
            $FailedItem
        }
    Finally
        {
            # Don't need to say much here
        }
    }
Elseif ($Overwrite -eq $false)
    {
        $PathCheck = Test-Path ('Env:\'+$Name)
        IF ($PathCheck -eq 'true')
            {
                $UpperName = $name.ToUpper()
                Write-Host "$type variable $upperName already exists, use the overwite switch if you would like to change the value."
            }
        Elseif ($Overwrite -eq $false)
            {
                Try
                {
                    $ErrorActionPreference = 'stop'
                    [Environment]::SetEnvironmentVariable("$Name","$Value","$Type")
                    Write-Host "$Type variable $name has been written with a value of $value"
                }
                Catch
                {
                    $ErrorMessage = $_.Exception.Message
                    $FailedItem = $_.Exception.ItemName
                    $ErrorMessage
                    $FailedItem
                }
                Finally
                {
                    # Don't need to say much here
                }
            }
        }
    }

Good luck and happy scripting!

Connect-VCenter & Set-Creds


Set-Creds
Connect-Vcenter

Here are 2 seemingly useless functions that I’ve loved for some time now. Both tasks that are achieved are effectively not that difficult individually, or frankly by them self, but when repeated multiple times become very tiresome!

Set-Creds

The first function, Set-Creds, is a basic wrapper for the pre-existing Get-Credential function already supplied by Microsoft. This command can easily be run, the data stored in an encrypted Variable (to obfuscate the data from prying eyes) and then passed on to various  other commands where the current user shell context (typically the currently logged in user/password) is not suitable.

Here is the function it self.


function Set-Creds {
    param(
    [parameter(Mandatory=$false)]$VariableName='creds',
    [parameter(Mandatory=$False)][switch]$ClearVariable=$false
    )
    IF ($ClearVariable -eq $false)
        {
            Set-Variable -Name $VariableName -Value (Get-Credential) -Scope global
        }
    ElseIf ($ClearVariable -eq $true)
        {
            Remove-Variable -Name $VariableName -Scope global
        }
    }

Here is the code in action:

Set-Creds

Connect-VCenter

Here is another seemingly easy task to achieve. Loading the VMWare PowerCLI commands and then connecting to what ever VCenter you have in mind. This is actually again, a very simple task, but IT CAN BE MADE SO MUCH EASIER. How you might ask? Yes, that’s right, another Function!!!!

WARNING – This is a little lazy this code. I’ve not tested this on machines running anything less than PowerCLI version 5 – Prior to that, I would need to write some try/catch kinda stuff – So, its not going to break anything, but you might get a few errors.

function Connect-Vcenter {
    param(
        [parameter(Mandatory=$true)]$vcenter,
        [parameter(mandatory=$false)]$Credentials=$null,
        [parameter(Mandatory=$false)][switch]$Multiple=$false
    )
    #Load snap-in
& "C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1"

    if ($Credentials -eq $null)
        {
            $credentials = Get-Credential
        }
    if ($Multiple -eq $true)
        {
            # Connect up - Allows for multiple vcenter connections :O
            Connect-VIServer $vCenter -Credential $Credentials
        }
    else
        {
            #Connect to vCenter - But not if you are already connected
            if ($global:DefaultVIServers.Count -lt 1) 
                {
                    Connect-VIServer $vCenter -Credential $Credentials
                }
                else
                {
                    Write-Host "Already connected to a vcenter server!!"
                }
    }
}

Here we show the code in use, much the same as before…

Connect-Vcenter

All together now!

Right, now here is where things start to make a little more sense. Lets set the scene. You’ve arrived onsite at customers site. You need to connect to their VCenter server and do some work and you are in a hurry, your username and password aren’t on the same domain as the customers. Easy, now we just use the Set-Creds and the Connect-VCenter commands from within the first Powershell you can get open:

finaloutcome

So, what have we achieved? We’ve managed to load the VMWare PowerCLI into a standard Powershell Shell, create a variable with a SECURELY stored password and connect to the vcenter server all in a few simple commands.

Plenty of uses for this moving forward, I’m sure you’ll see me reference them in the future as I build up my online arsenal of functions. Which leads me to another point. How will having these functions make life easier if I am constantly having to load them into a shell? I’ve plans to write up how I make my own custom Powershell module and also customizing your $Profile to ensure they are always in there! But that is for a later date…

VMWare CPU Over Prescription!

Recently I was asked to look into performance issues on a VMware cluster. The IT department was adamant that the issues they were seeing were caused by poor SAN (Storage area network) I/O (Input/output). Having worked in infrastructure for many years, but most recently embarking on a journey into the world of DevOps (Development Operations), I was quick to put my hand up to look into the issues as I felt I had plenty of value to add!

INITIAL ANALYSIS:

Normally in a smaller environment, or more to the point, a smaller business, I would do a simple calculation to see how over provisioned the CPU is. This simple and extremely quick & mainly unscientific calculation came about for 2 reasons

  • Labor cost
  • Time frame (or lack of)

I call this calculation “CPU Contention Ratio” – Or CCR as it will be known moving forward. CCR is a crude method used for a quick analysis of the over prescription of physical CPU resource to virtual CPU resource. EG: How much physical CPU resource is actually backing the virtual CPU resource

Calculating CCR:
$a = (number of host logical CPU cores within the cluster)
$b = (number of virtual cores assigned to running Virtual Machines within the cluster)
$a / $b = (CCR)

Or, here is a function I’ve written to give us exactly that. Requirements to run this script are as follows:

  • VMWare PowerCLI installed
  • VMWare PowerCLI modules loaded into the current shell session (I’ve written a function for just that found here)
  • A Username & Password to the VCenter with at least read-only rights to the cluster and VM’s

function Get-VMCCR {
    param (
    $Datacenter,
    $Cluster
    )
    #Get the VM Host specific info and store in a variable
    $VMHosts = Get-Datacenter -Name $Datacenter | Get-Cluster -Name $Cluster | Get-VMHost
    $VMs = Get-Datacenter -Name $Datacenter | Get-Cluster -Name $Cluster | Get-VMHost | Get-VM | where {$_.powerstate -eq 'PoweredOn'}

    # Initilizing a new array 
    $VMHostCPUCounts = @()

    # Loop through the hosts to see if they have HyperThreading on and calculate as required
    foreach ($VMHost in $VMHosts)
        {
            IF ($VMHost.HyperthreadingActive -eq 'True')
                {
                    $VMHostCPUCount = "" | Select 'Name','PhysicalCPUCount','TotalLogicalCPUCount'
                    $VMHostCPUCount.Name = $VMHost.Name
                    $VMHostCPUCount.PhysicalCPUCount = $VMHost.NumCPU
                    $VMHostCPUCount.TotalLogicalCPUCount = ($VMHost.NumCPU)*2
                    $VMHostCPUCounts += $VMHostCPUCount
                }
            Else
                {
                    $VMHostCPUCount = "" | Select 'Name','PhysicalCPUCount','TotalLogicalCPUCount'
                    $VMHostCPUCount.Name = $VMHost.Name
                    $VMHostCPUCount.PhysicalCPUCount = $VMHost.NumCPU
                    $VMHostCPUCount.TotalLogicalCPUCount = $VMHost.NumCPU
                    $VMHostCPUCounts += $VMHostCPUCount
                }
        }



    # Dump the output to the console
    $Outputs = @()
    $Output = "" | Select 'Datacenter','Cluster','TotalLogicalCPUCount','AssignedVCPU','CCR'
    $Output.Datacenter = $Datacenter
    $Output.Cluster = $Cluster
    $Output.TotalLogicalCPUCount = ($VMHostCPUCounts.TotalLogicalCPUCount | Measure-Object -sum).sum
    $Output.AssignedVCPU = ($VMs.NumCPU | Measure-Object -Sum).sum
    $output.CCR = "{0:N2}" -f ((($VMHostCPUCounts.TotalLogicalCPUCount| Measure-Object -Sum).sum/($VMs.NumCPU | Measure-Object -Sum).sum))
    $Outputs += $Output
    
    return $outputs

}

And after running this function, I am output with the following:


Datacenter Cluster    TotalLogicalCPUCount AssignedVCPU CCR 
---------- -------    -------------------- ------------ --- 
Region01   Production                  136          434 0.31

As you can see, I have 0.31 of a physical core backing each virtual CPU assigned to a virtual machine within the cluster. In my opinion, this is to little. For real time applications, no less than about 0.9 CCR is acceptable and for general application servers that require performance, but are not real time, no less than 0.75 CCR is acceptable.

BUT ALAS – THIS WASN”T ENOUGH

So, now more analysis is required, as IT want a more scientific analysis of the exact issue and also proof that the storage isn’t the bottle neck as they are suggesting – BUT, more on the in-depth analysis and specifically VMWare statistics collection using VMWare PowerCLI in my next post.