I recently did some work with the vRealize Operations Manager REST API using PowerShell. Part of the work was to do with policy export and import, which raised 2 interesting challenges from a PowerShell point of view:
Firstly, HTTP mutlipart form data. We use an HTTP request to send a file to the API, easy on it’s own using the -infile parameter on Invoke-RestMethod, but as soon as you start adding other fields this becomes a bit more involved.
Secondly, the API exports and imports XML policy files in .zip format. Given that this work was part of a CI/CD pipeline in Gitlab, I really didn’t want to be committing .zip files to Git. I also didn’t want to write out files to disk, but rather compress and decompress on the fly as required.
This led to some interesting constructs inside my functions which I thought I would blog about.
HTTP Multipart Form Data
So, HTTP multipart forms, what’s the issue with these? The requirement was to upload a .zip file to the API, and at the same time optionally specify a “forceImport” parameter. The specifics don’t really matter, the point is we want to POST a file along with a bunch of other parameters. This is a one liner with curl:
curl "https://vrops01.lab.local/suite-api/internal/policies/import" -i -u admin:VMware1! -X POST -H "Content-Type: multipart/form-data" -H "X-vRealizeOps-API-use-unsupported: true" -F "policy=@c:\vROPSDemo\POLICY-TENANT01.zip" -F "forceImport=true" -k
Typically we would perform this sort of request with Invoke-RESTMethod (or potentially Invoke-WebRequest). To send a file, we would simply use the -InFile parameter. The complication arises when we want additional parameters to be sent in the same request, these then require a request body. PowerShell treats the -InFile and -Body parameters as mutually exclusive, i.e. one or the other. The end result is that natively these CMDlets do not support mutlipart/form-data request types. Not to worry though, there is always a workaround 🙂
Going back to our vROPs example, let’s get some of the “setup” done before we get to the interesting multipart stuff:
## Set vROPs node IP
$vROPsNode = "10.107.9.202"
## Set node credentials
$vROPsUser = "admin"
$vROPsPass = "VMware1!"
## Path to policy file
$policyFile = "C:\temp_scripts\POLICY-TENANT01.zip"
## Get filename from path
$policyFilename = Split-Path $policyFile -Leaf
## Set authentication headers
$authHeader = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($vROPSUser + ":" + $vROPSPass))
## Define headers
$headers = @{}
$headers.Add("Content-Type", 'multipart/mixed')
$headers.Add("Accept", 'application/json')
$headers.Add("X-vRealizeOps-API-use-unsupported", 'true')
$headers.Add("Authorization", ("Basic " + $authHeader))
## Set target URI for this node
$Uri = ("https://" + $vROPSNode + "/suite-api/internal/policies/import")
All we are doing here is configuring our headers for the request we are about to send. The example vROPs being used is version 7.5 which means that the policy import is part of an unsupported API. This requires the “X-vRealizeOps-API-use-unsupported” header which we have specified. The other headers are probably familiar if you do much work with REST API’s.
Let’s get started with the multipart piece. First, we need to read our file into a byte array:
$policyBin = [System.IO.File]::ReadAllBytes($policyFile)
Next, we need to define what is called an HTTP boundary. When sending a multipart request, a boundary allows the server to see where one parameter starts and another ends within the request body. This can be any value at all as long as it’s unique within the request. I usually just set a GUID to be the boundary value:
$httpBoundary = [guid]::NewGuid().ToString()
To inject the file content into the request, we will need to encode our byte array as a string. These couple of lines will do the trick:
$encodingScheme = [System.Text.Encoding]::GetEncoding("iso-8859-1")
$encodedFile = $encodingScheme.GetString($policyBin)
Now, my preference is to define a request body template up front, then inject values into it. I find this makes code a little bit more transportable and easier to read, but be aware there are many other valid approaches to this. Here is the template we are going to use:
$httpTemplate = @"
--{0}
Content-Disposition: form-data; name=forceImport
true
--{0}
Content-Disposition: form-data; name=policy; filename={1}
Content-Type: multipart/form-data
{2}
--{0}--
"@
Here we can see the forceImport parameter and our file data fields. Let’s put some data into these to prepare the request:
$httpBody = $httpTemplate -f $httpBoundary, $policyFilename, $encodedFile
In PowerShell, -f allows you to perform string formatting. In this case, we are simply replacing {0}. {1} etc with the list of values in order (you can also use an array of values if you have a lot of them). I find this a simple way to inject content into an HTTP template.
Finally, the moment of truth:
Invoke-RestMethod -Uri $Uri -Method Post -ContentType "multipart/form-data; boundary=$httpBoundary;" -Body $httpBody -Headers $headers
If everything has gone to plan, the file content and the parameter have gone to the API. Again, don’t get hung up on the specifics of the API, this approach could be used anywhere with a similar scenario.
Adding on the fly Compression
While that was entirely complicated enough, let’s make it more difficult. While I wanted to source control an XML file, the import API only accepts file content as a .zip. I wanted to stay away from writing out temporary .zip files (and remembering to tidy them up) as much as possible. What if I could find a way to create a compressed byte array from the source XML in memory up front? That should work really well. And it did 🙂
We need some code between opening the file and creating the HTTP request to do a bit of compression work for us. Rather than going through this line by line I’ll roll it out altogether:
## Path to policy file
$policyFile = "C:\temp_scripts\POLICY-TENANT01.xml"
## Load required .net assemblies
$assemblies = Add-Type -Assembly System.IO.Compression
## Declare new memory stream
[System.IO.MemoryStream] $memStream = New-Object System.IO.MemoryStream
## Declare new zip archive and set compression mode
$zipArchive = [System.IO.Compression.ZipArchive]::new($memStream, ([IO.Compression.CompressionMode]::Compress))
## Set new entry in this zip for policy file
$zipEntry = $zipArchive.CreateEntry("policyImport.xml")
## Open stream to this file and write in contents
$fileStream = $zipEntry.Open()
$fileStream.Write($policyBin, 0, $policyBin.Length)
## Close the file steam
$fileStream.Close()
## Close the memoroy stream
$memStream.Close()
## Output byte array
$finalZip = $memStream.ToArray()
## Set encoding scheme
$encodingScheme = [System.Text.Encoding]::GetEncoding("iso-8859-1")
$encodedFile = $encodingScheme.GetString($finalZip)
As you can see this requires dipping into some .net to make this possible. Let’s take a look a the completed function to see how all of this hangs together:
function Import-VROPsPolicy {
<#
.SYNOPSIS
Imports and XML policy to a vROPs node.
.DESCRIPTION
This function will take a previously exported vROPs policy and import it to the target node(s)
The function will compress the XML in memory and inject it to the HTTP body.
The function can optionally overwrite any existing policy.
.PARAMETER vROPSNode
The target vROPs node to perform an import on. Can be pipelined.
.PARAMETER policyFile
The XML policy file to import, which was exported from the same or another vROPs node.
.PARAMETER forceUpdate
Optional parameter. Will overwrite an existing policy with the new one.
.PARAMETER Credential
PowerShell credential object with appropriate permissions for policy import.
.INPUTS
System.String. vROPs node names can be piped to this function.
.OUTPUTS
None.
.EXAMPLE
Import-VROPsPolicy -vROPSNode vrops01.lab.local -policyFile c:\policies\sample.xml -forceUpdate -Credential $creds -Verbose
Import the policy file sample.xml to the vROPs node vrops.lab.local and force overwrite. Uses credential object $creds and specifies verbose output
.EXAMPLE
$vROPSNodes | Import-VROPsPolicy -policyFile c:\policies\sample.xml -forceUpdate -Credential $creds
Import the policy file sample.xml to all vROPs nodes within the $vROPSNodes array and force overwrite. Uses credential object $creds.
.EXAMPLE
$vROPSNodes | Import-VROPsPolicy -policyFile c:\policies\sample.xml -Credential $creds
Import the policy file sample.xml to all vROPs nodes within the $vROPSNodes array (will not overwrite). Uses credential object $creds.
.LINK
.NOTES
01 Alistair McNair Initial version.
#>
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true,ValueFromPipeline=$true)]
[string]$vROPSNode,
[Parameter(Mandatory=$true,ValueFromPipeline=$false)]
[String]$policyFile,
[Parameter(Mandatory=$true,ValueFromPipeline=$false)]
[System.Management.Automation.PSCredential]$Credential,
[Parameter(Mandatory=$false,ValueFromPipeline=$false)]
[Switch]$forceUpdate
)
begin {
Write-Verbose ("Starting function.")
## Ignore invalid certificates
if (!([System.Management.Automation.PSTypeName]'TrustAllCertsPolicy').Type) {
Add-Type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy -ErrorAction SilentlyContinue
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
} # if
## Validate specified policy file
if (!(Test-Path $policyFile)) {
throw ("Specified policy file " + $policyFile + " was not found.")
} # if
Write-Verbose ("Using policy file " + $policyFile)
if ($forceUpdate.IsPresent) {
Write-Verbose ("Policy overwrite has been specified.")
} # if
## Create an HTTP boundary, required for multipart uploads
$httpBoundary = [guid]::NewGuid().ToString()
## Define headers
$headers = @{}
$headers.Add("Content-Type", 'multipart/mixed')
$headers.Add("Accept", 'application/json')
$headers.Add("X-vRealizeOps-API-use-unsupported", 'true')
} # begin
process {
Write-Verbose ("Processing vROPS node " + $vROPSNode)
## Read file out to binary
$policyBin = [System.IO.File]::ReadAllBytes($policyFile)
## Load the required assemblies
try {
Add-Type -Assembly System.IO.Compression -ErrorAction Stop | Out-Null
} # try
catch {
Write-Debug ("Failed to load assemblies.")
throw ("Failed to load assemblies, the CMDlet returned " + $_.exception.message)
} # catch
## Set encoding scheme
$encodingScheme = [System.Text.Encoding]::GetEncoding("iso-8859-1")
## Declare new memory stream
[System.IO.MemoryStream] $memStream = New-Object System.IO.MemoryStream
Write-Verbose ("Compressing policy file.")
## Declare new zip archive and set compression mode
$zipArchive = [System.IO.Compression.ZipArchive]::new($memStream, ([IO.Compression.CompressionMode]::Compress))
## Set new entry in this zip for policy file
$zipEntry = $zipArchive.CreateEntry("policyImport.xml")
## Open stream to this file and write in contents
$fileStream = $zipEntry.Open()
$fileStream.Write($policyBin, 0, $policyBin.Length)
## Close the file steam
$fileStream.Close()
## Close the memoroy stream
$memStream.Close()
## Output byte array
$finalZip = $memStream.ToArray()
## Configure a here string with HTTP header content
$httpTemplate = @"
--{0}
Content-Disposition: form-data; name=forceImport
{1}
--{0}
Content-Disposition: form-data; name=policy; filename=policyImport.zip
Content-Type: multipart/form-data
{2}
--{0}--
"@
## Inject content
try {
$httpBody = $httpTemplate -f $httpBoundary, $forceUpdate.IsPresent, $encodingScheme.GetString($finalZip)
Write-Verbose ("HTTP request template configured.")
} # try
catch {
Write-Debug ("Failed to set HTTP request.")
throw ("Failed to configure HTTP request, the CMDlet returned " + $_.exception.message)
} # catch
## Set target URI for this node
$Uri = ("https://" + $vROPSNode + "/suite-api/internal/policies/import")
## Send request to import
try {
$policyImport = Invoke-RestMethod -Uri $Uri -Method Post -ContentType "multipart/form-data; boundary=$httpBoundary;" -Body $httpBody -Headers $headers -Credential $Credential -ErrorAction Stop
Write-Verbose ("Policy import was successful")
} # try
catch {
Write-Debug ("Failed to import policy file.")
throw ("Failed to import policy file, the CMDlet returned " + $_.exception.message)
} # catch
## The API always returns a 202 success code, even if import failed. We can't use this to indicate success.
## We need to check that at least 1 policy was created, updated or skipped. If all conditions are 0 then the import failed.
if (($policyImport.'created-policies'.count -eq 0) -and ($policyImport.'skipped-policies'.count -eq 0) -and ($policyImport.'updated-policies'.count -eq 0)) {
throw ("Policy import failed. Check that the XML is valid and the character encoding is UTF-8 without BOM.")
} # if
else {
Write-Verbose ("Policy created count: " + $policyImport.'created-policies'.count)
Write-Verbose ("Policy updated count: " + $policyImport.'updated-policies'.count)
Write-Verbose ("Policy skipped count: " + $policyImport.'skipped-policies'.count)
} # else
Write-Verbose ("vROPS node complete.")
} # process
end {
Write-Verbose ("Function complete.")
} # end
} # function
As an aside, I found that the API in this version of vROPs always returned a 202 code, even if the whole thing was a smoking car crash. The function includes a bit of code to check that at least something was created, updated or skipped.
Conclusion
There are plenty of multipart examples out there to draw upon, but I wasn’t able to find anything hanging that together with the compression piece. This took a little bit of head scratching to get right, so hopefully it saves someone else some pain. As I mentioned, don’t get hung up on this being vROPs orientated, if you do work with REST API’s and PowerShell, then it’s not impossible you will come across a similar scenario.
The above function and other vROPs functions are now part of a Dot Source module hosted on Github here. This module will grow over time so worth keeping an eye on if you automate vROPs with PowerShell 😉
.The Dot Source 