HTTP Multipart Forms and on the Fly Compression in PowerShell.

I recently did some work with the vRealize Operations Manager REST API using PowerShell. Part of the work was to do with policy export and import, which raised 2 interesting challenges from a PowerShell point of view:

Firstly, HTTP mutlipart form data. We use an HTTP request to send a file to the API, easy on it’s own using the -infile parameter on Invoke-RestMethod, but as soon as you start adding other fields this becomes a bit more involved.

Secondly, the API exports and imports XML policy files in .zip format. Given that this work was part of a CI/CD pipeline in Gitlab, I really didn’t want to be committing .zip files to Git. I also didn’t want to write out files to disk, but rather compress and decompress on the fly as required.

This led to some interesting constructs inside my functions which I thought I would blog about.

HTTP Multipart Form Data

So, HTTP multipart forms, what’s the issue with these? The requirement was to upload a .zip file to the API, and at the same time optionally specify a “forceImport” parameter. The specifics don’t really matter, the point is we want to POST a file along with a bunch of other parameters. This is a one liner with curl:

curl "https://vrops01.lab.local/suite-api/internal/policies/import" -i -u admin:VMware1! -X POST -H "Content-Type: multipart/form-data" -H "X-vRealizeOps-API-use-unsupported: true" -F "policy=@c:\vROPSDemo\POLICY-TENANT01.zip" -F "forceImport=true" -k

Typically we would perform this sort of request with Invoke-RESTMethod (or potentially Invoke-WebRequest). To send a file, we would simply use the -InFile parameter. The complication arises when we want additional parameters to be sent in the same request, these then require a request body. PowerShell treats the -InFile and -Body parameters as mutually exclusive, i.e. one or the other. The end result is that natively these CMDlets do not support mutlipart/form-data request types. Not to worry though, there is always a workaround πŸ™‚

Going back to our vROPs example, let’s get some of the “setup” done before we get to the interesting multipart stuff:

## Set vROPs node IP
$vROPsNode = "10.107.9.202"

## Set node credentials
$vROPsUser = "admin"
$vROPsPass = "VMware1!"

## Path to policy file
$policyFile = "C:\temp_scripts\POLICY-TENANT01.zip"

## Get filename from path
$policyFilename = Split-Path $policyFile -Leaf

## Set authentication headers
$authHeader = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($vROPSUser + ":" + $vROPSPass))

## Define headers
$headers = @{}
$headers.Add("Content-Type", 'multipart/mixed')
$headers.Add("Accept", 'application/json')
$headers.Add("X-vRealizeOps-API-use-unsupported", 'true')
$headers.Add("Authorization", ("Basic " + $authHeader))

## Set target URI for this node
$Uri = ("https://" + $vROPSNode + "/suite-api/internal/policies/import")

All we are doing here is configuring our headers for the request we are about to send. The example vROPs being used is version 7.5 which means that the policy import is part of an unsupported API. This requires the “X-vRealizeOps-API-use-unsupported” header which we have specified. The other headers are probably familiar if you do much work with REST API’s.

Let’s get started with the multipart piece. First, we need to read our file into a byte array:

$policyBin = [System.IO.File]::ReadAllBytes($policyFile)

Next, we need to define what is called an HTTP boundary. When sending a multipart request, a boundary allows the server to see where one parameter starts and another ends within the request body. This can be any value at all as long as it’s unique within the request. I usually just set a GUID to be the boundary value:

$httpBoundary = [guid]::NewGuid().ToString()

To inject the file content into the request, we will need to encode our byte array as a string. These couple of lines will do the trick:

$encodingScheme = [System.Text.Encoding]::GetEncoding("iso-8859-1")
$encodedFile = $encodingScheme.GetString($policyBin)

Now, my preference is to define a request body template up front, then inject values into it. I find this makes code a little bit more transportable and easier to read, but be aware there are many other valid approaches to this. Here is the template we are going to use:

$httpTemplate = @"
--{0}
Content-Disposition: form-data; name=forceImport

true
--{0}
Content-Disposition: form-data; name=policy; filename={1}
Content-Type: multipart/form-data

{2}
--{0}--
"@

Here we can see the forceImport parameter and our file data fields. Let’s put some data into these to prepare the request:

$httpBody = $httpTemplate -f $httpBoundary, $policyFilename, $encodedFile

In PowerShell, -f allows you to perform string formatting. In this case, we are simply replacing {0}. {1} etc with the list of values in order (you can also use an array of values if you have a lot of them). I find this a simple way to inject content into an HTTP template.

Finally, the moment of truth:

Invoke-RestMethod -Uri $Uri -Method Post -ContentType "multipart/form-data; boundary=$httpBoundary;" -Body $httpBody -Headers $headers

If everything has gone to plan, the file content and the parameter have gone to the API. Again, don’t get hung up on the specifics of the API, this approach could be used anywhere with a similar scenario.

Adding on the fly Compression

While that was entirely complicated enough, let’s make it more difficult. While I wanted to source control an XML file, the import API only accepts file content as a .zip. I wanted to stay away from writing out temporary .zip files (and remembering to tidy them up) as much as possible. What if I could find a way to create a compressed byte array from the source XML in memory up front? That should work really well. And it did πŸ™‚

We need some code between opening the file and creating the HTTP request to do a bit of compression work for us. Rather than going through this line by line I’ll roll it out altogether:

## Path to policy file
$policyFile = "C:\temp_scripts\POLICY-TENANT01.xml"

## Load required .net assemblies
$assemblies = Add-Type -Assembly System.IO.Compression

## Declare new memory stream
[System.IO.MemoryStream] $memStream = New-Object System.IO.MemoryStream

## Declare new zip archive and set compression mode
$zipArchive = [System.IO.Compression.ZipArchive]::new($memStream, ([IO.Compression.CompressionMode]::Compress))

## Set new entry in this zip for policy file
$zipEntry = $zipArchive.CreateEntry("policyImport.xml") 

## Open stream to this file and write in contents
$fileStream = $zipEntry.Open()
$fileStream.Write($policyBin, 0, $policyBin.Length)

## Close the file steam
$fileStream.Close()

## Close the memoroy stream
$memStream.Close()

## Output byte array
$finalZip = $memStream.ToArray()


## Set encoding scheme
$encodingScheme = [System.Text.Encoding]::GetEncoding("iso-8859-1")
$encodedFile = $encodingScheme.GetString($finalZip)

As you can see this requires dipping into some .net to make this possible. Let’s take a look a the completed function to see how all of this hangs together:

function Import-VROPsPolicy {
    <#
    .SYNOPSIS
        Imports and XML policy to a vROPs node.

    .DESCRIPTION
        This function will take a previously exported vROPs policy and import it to the target node(s)
        The function will compress the XML in memory and inject it to the HTTP body.
        The function can optionally overwrite any existing policy.

    .PARAMETER vROPSNode
        The target vROPs node to perform an import on. Can be pipelined.

    .PARAMETER policyFile
        The XML policy file to import, which was exported from the same or another vROPs node.

    .PARAMETER forceUpdate
        Optional parameter. Will overwrite an existing policy with the new one.

    .PARAMETER Credential
        PowerShell credential object with appropriate permissions for policy import.

    .INPUTS
        System.String. vROPs node names can be piped to this function.

    .OUTPUTS
        None.

    .EXAMPLE
        Import-VROPsPolicy -vROPSNode vrops01.lab.local -policyFile c:\policies\sample.xml -forceUpdate -Credential $creds -Verbose

        Import the policy file sample.xml to the vROPs node vrops.lab.local and force overwrite. Uses credential object $creds and specifies verbose output

    .EXAMPLE
        $vROPSNodes | Import-VROPsPolicy -policyFile c:\policies\sample.xml -forceUpdate -Credential $creds

        Import the policy file sample.xml to all vROPs nodes within the $vROPSNodes array and force overwrite. Uses credential object $creds.

    .EXAMPLE
        $vROPSNodes | Import-VROPsPolicy -policyFile c:\policies\sample.xml -Credential $creds

        Import the policy file sample.xml to all vROPs nodes within the $vROPSNodes array (will not overwrite). Uses credential object $creds.

    .LINK

    .NOTES
        01           Alistair McNair          Initial version.

    #>

    [CmdletBinding()]
    Param
    (
        [Parameter(Mandatory=$true,ValueFromPipeline=$true)]
        [string]$vROPSNode,
        [Parameter(Mandatory=$true,ValueFromPipeline=$false)]
        [String]$policyFile,
        [Parameter(Mandatory=$true,ValueFromPipeline=$false)]
        [System.Management.Automation.PSCredential]$Credential,
        [Parameter(Mandatory=$false,ValueFromPipeline=$false)]
        [Switch]$forceUpdate
    )

    begin {

        Write-Verbose ("Starting function.")

        ## Ignore invalid certificates
        if (!([System.Management.Automation.PSTypeName]'TrustAllCertsPolicy').Type) {
            Add-Type @"
            using System.Net;
            using System.Security.Cryptography.X509Certificates;
            public class TrustAllCertsPolicy : ICertificatePolicy {
                public bool CheckValidationResult(
                    ServicePoint srvPoint, X509Certificate certificate,
                    WebRequest request, int certificateProblem) {
                    return true;
                }
            }
"@

            [System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy -ErrorAction SilentlyContinue

            [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

        } # if

        ## Validate specified policy file
        if (!(Test-Path $policyFile)) {

            throw ("Specified policy file " + $policyFile + " was not found.")
        } # if

        Write-Verbose ("Using policy file " + $policyFile)


        if ($forceUpdate.IsPresent) {
            Write-Verbose ("Policy overwrite has been specified.")
        } # if


        ## Create an HTTP boundary, required for multipart uploads
        $httpBoundary = [guid]::NewGuid().ToString()

        ## Define headers
        $headers = @{}
        $headers.Add("Content-Type", 'multipart/mixed')
        $headers.Add("Accept", 'application/json')
        $headers.Add("X-vRealizeOps-API-use-unsupported", 'true')

    } # begin


    process {


        Write-Verbose ("Processing vROPS node " + $vROPSNode)


        ## Read file out to binary
        $policyBin = [System.IO.File]::ReadAllBytes($policyFile)

        ## Load the required assemblies
        try {
            Add-Type -Assembly System.IO.Compression -ErrorAction Stop | Out-Null
        } # try
        catch {
            Write-Debug ("Failed to load assemblies.")
            throw ("Failed to load assemblies, the CMDlet returned " + $_.exception.message)
        } # catch

        ## Set encoding scheme
        $encodingScheme = [System.Text.Encoding]::GetEncoding("iso-8859-1")

        ## Declare new memory stream
        [System.IO.MemoryStream] $memStream = New-Object System.IO.MemoryStream

        Write-Verbose ("Compressing policy file.")

        ## Declare new zip archive and set compression mode
        $zipArchive = [System.IO.Compression.ZipArchive]::new($memStream, ([IO.Compression.CompressionMode]::Compress))

        ## Set new entry in this zip for policy file
        $zipEntry = $zipArchive.CreateEntry("policyImport.xml")

        ## Open stream to this file and write in contents
        $fileStream = $zipEntry.Open()
        $fileStream.Write($policyBin, 0, $policyBin.Length)

        ## Close the file steam
        $fileStream.Close()

        ## Close the memoroy stream
        $memStream.Close()

        ## Output byte array
        $finalZip = $memStream.ToArray()

        ## Configure a here string with HTTP header content
	    $httpTemplate = @"
--{0}
Content-Disposition: form-data; name=forceImport

{1}
--{0}
Content-Disposition: form-data; name=policy; filename=policyImport.zip
Content-Type: multipart/form-data

{2}
--{0}--
"@


        ## Inject content
        try {
            $httpBody = $httpTemplate -f $httpBoundary, $forceUpdate.IsPresent, $encodingScheme.GetString($finalZip)
            Write-Verbose ("HTTP request template configured.")
        } # try
        catch {
            Write-Debug ("Failed to set HTTP request.")
            throw ("Failed to configure HTTP request, the CMDlet returned " + $_.exception.message)
        } # catch

        ## Set target URI for this node
        $Uri = ("https://" + $vROPSNode + "/suite-api/internal/policies/import")


        ## Send request to import
        try {
            $policyImport = Invoke-RestMethod -Uri $Uri -Method Post -ContentType "multipart/form-data; boundary=$httpBoundary;" -Body $httpBody -Headers $headers -Credential $Credential -ErrorAction Stop
            Write-Verbose ("Policy import was successful")
        } # try
        catch {
            Write-Debug ("Failed to import policy file.")
            throw ("Failed to import policy file, the CMDlet returned " + $_.exception.message)
        } # catch


        ## The API always returns a 202 success code, even if import failed. We can't use this to indicate success.
        ## We need to check that at least 1 policy was created, updated or skipped. If all conditions are 0 then the import failed.
        if (($policyImport.'created-policies'.count -eq 0) -and ($policyImport.'skipped-policies'.count -eq 0) -and ($policyImport.'updated-policies'.count -eq 0)) {
            throw ("Policy import failed. Check that the XML is valid and the character encoding is UTF-8 without BOM.")
        } # if
        else {
            Write-Verbose ("Policy created count: " + $policyImport.'created-policies'.count)
            Write-Verbose ("Policy updated count: " + $policyImport.'updated-policies'.count)
            Write-Verbose ("Policy skipped count: " + $policyImport.'skipped-policies'.count)
        } # else


        Write-Verbose ("vROPS node complete.")

    } # process


    end {
        Write-Verbose ("Function complete.")
    } # end


} # function

As an aside, I found that the API in this version of vROPs always returned a 202 code, even if the whole thing was a smoking car crash. The function includes a bit of code to check that at least something was created, updated or skipped.

Conclusion

There are plenty of multipart examples out there to draw upon, but I wasn’t able to find anything hanging that together with the compression piece. This took a little bit of head scratching to get right, so hopefully it saves someone else some pain. As I mentioned, don’t get hung up on this being vROPs orientated, if you do work with REST API’s and PowerShell, then it’s not impossible you will come across a similar scenario.

The above function and other vROPs functions are now part of a Dot Source module hosted on Github here. This module will grow over time so worth keeping an eye on if you automate vROPs with PowerShell πŸ˜‰

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: