Installing vRealize Orchestrator 7.6

VMware vRealize Orchestrator has changed quite a lot, and it now has a brand new HTML5 client, which is the default one. You can still use legacy client, but there can be some compatibility issues between workflows done with new vs. legacy client.

Do note, this is not a "best practice" guide, more a documentation on how I did installation in my home lab

Orchestrator comes as an ova file, so deploying it is quite straight forward, and I'm not going to through that process.

But, after you have done the initial deployment, this is what I did.

I first logged in to https://<appliance ip>:5480, to check if there are any updates available. Well, there was none at that time.

Next I went to Control Center. You can find it by going to front page of your appliance, https://<appliance ip>:8281/vco/ (or by going to http://<appliance ip>, which will redirect you there) and selecting "Start the Control Center"

It will ask you for you root / pass, that you provided during deployment of .ova

First we configure a host name:

Then authentication. Since I do not have vRealize Automation, I did choose vSphere as authentication mode, and gave it my vCenter address:

You need to click "accept certificate" first.

Give account that you use to connect to vCenter:

 After that, you need to select Admin group that has access to vRO, and you are done.

Now you should see vRO Control Center

If you want, you can test that authentication works, by going "Configure Authentication Provider -> Test Login"

Now you should be able to login to your Orchestrator Client, go to front page of your appliance, and select "Start the orchestrator client"

For the first time, it gave me this:

But after logging out of my vCenter, I was able to log in succesfully. It should give you your vCenter login page:

And after giving credentials,we are logged in to our brand new HTML5 based orchestrator client!

My HomeLab setup

On end of last year, I started to slowly build up up a home lab to be able to test all kinds of things, and to study more some of our own applications.

And don't get me wrong here, we have a great demo environments, but they are something that you are not supposed to break down. And sometimes, to really understand that how things work, you actually need to break them.

So I decided that I finally need to build up my own home lab environment.

And here is my current physical setup:

VMware (all in version 6.7):

•  Virtual vCenter Appliance
• "Production" Cluster:
• 1x Intel NUC8i5BEK, i5-8259U @ 2.3 GHz, 64GB RAM
• 1x Intel NUC7i3BNK, i3-7100U @ 2.4 GHz, 32GB RAM
• 1x Omen by HP laptop, i7-8750H @ 2.20GHz, 16GB RAM. This one I occasionally boot up from USB-disk containing ESX installation, if I need temporarily more compute power, or to test some DR scenarios etc.

Hyper-V, standalone host, Win 2016 Core

• ASUS UX303L laptop, Intel® Core™ i7 4510U @ 2GHz, 8GB. This is my old laptop with broken display. Only sad thing with this is that it wont boot if it's not attached to some display while booting, have not figured out how to bypass that. This one uses local SSD in laptop, so no connection to shared storage.

Network:

• HP OfficeConnect Switch 1820-24G. This is reasonably priced (web)managed switch, and at least in my home lab 1 Gbps connections are fast enough. Still for my Storage I did a 2 Gbps trunk.

Storage:

• QNAP TS879 Pro, equipped with 8x 3TB disks. And configured to RAID-10. So more than enough of capacity and performance for my needs. This is my main storage for the VMware environment. Disks are both iSCSI and NFS (Why both? Well, because I can). Connected to my switch with 2x links, so I have 2 Gbps connection for storage.
• Buffalo LinkStation LS220D. This is my secondary storage for homelab and primary storage for my personal files. And I use it as a secondary backup target for my homelab, and use my QNAP as a primary backup target for files in this one.

Firewall: 

• pfSense, running in virtual machine. Only one node, but since I now have to hosts, I'm going to make this one HA

So, what do I do with this environment then, and what do I have running in there?

Well, besides the basic infra services that you just have to have (DNS, AD etc) I have installed for example:

• VMware Orchestrator 7.6 (<- I hopefully have some time to blog about this later)
• A lot of Veeam stuff, including:
• Veeam Backup & Replication 9.5U4b and V10 beta1 (<- Yes, it's out!)
• Veeam Availability Orchestrator v2 (<- I'm planning to blog about this also)
• Veeam One v9.5
• Veeam Backup for Office 365 v3

I have also been playing around with Minio lately. (Hint, it has a lot to do with Backup & Replication)

Best here is, that I can pretty much test anything I want to, and if (read: when) I mess up with something, it only affects me, not anyone else.

And, hopefully this will help me to write a lot more blog posts.

Your blog will reboot in 3..2..1..NOW!

Wow, my blog has been silent for a bit over three years now. 

I just lost my motivation (and time) to write anything.

But I'm trying to start it all over again, let's see what happens. I already have some ideas on what to write about. 

So, now with a new name, and also with a new Instagram account (since I lost my access to old one, sigh....)

Feel free to start following, I will do my best to make this blog active again.

vRealize Orchestror, "Hello World!" and VM query

In first vRealize Orchestartor post, we connected Orchestrator to vCenter

Now it's time to create our first workflows.

With new tool, you always need to do a 'Hello World' first, so let's do it.

I have created a folder called 'vLAB', and under that, we create new workflow.

Give it a name 'Hello World'

In workflow editor, on Schema tab, drag'n'drop 'Scriptable task' to workflow, between start and end icons.

Click Pencil icon on top of 'Scriptable task' to edit it.

In script editor, on Scripting tab, write:
System.log("Hello World!");

And hit Close.

Now we can run our Workflow, by clicking 'Run' button, on Schema tab of Workflow editor.

And in Log tab, we can see output! We have successfully run our first workflow!

Cool! Now let's do something more productive. Let's get all VM's that we have in our VMware environment.

So, create new Workflow named 'Get All Virtual Machines' and add action 'getAllVMS', that is built-in action, and then add Scriptable Task, so it should look like this:

getAllVMs action will return all VMware Virtual Machines, so we need to set up Attribute (predefined variable) to our workflow. So, go to 'General' tab in workflow editor, and add new Attribute by clicking 'A+' icon

It will create attribute named 'att0'. It's a good idea to rename it, so click that name to rename it, we will name it 'AllVMs'

Then click on 'string', to change type of this Attribute. Set type to 'Array of...' and search 'VC:VirtualMachine'.

Now we need to direct values from 'getAllVMs' action to our attribute 'AllVMs'. So go back to 'Schema' tab, and hit Pencil icon on top of 'getAllVMs' action.

Go to 'Visual binding' tab, select 'actionResult' and 'drag' it to top of 'AllVMs', to connect it.

Next, edit 'Scriptable Task', and go to 'Visual Binding'.

Drag 'AllVMs to empty space in center of windows, at 'IN' side, so it should look like this:

Now we can use information inside 'AllVMs' in our Scriptable task.

Go to 'Script' tab, and write:

System.log(AllVMs);

(Do note, since this is javacript, almost everything is case sensitive....)

And now we can run our workflow, and we should get list of our VMs in 'Log' tab.

So, we have made our first two Workflows. 

In next posts, I'll show how to utilize that data, to do actions against those VMs.

Exporting documents from PasswordState

I have been using PasswordState for couple of years, and I have to say that I like it.

But there is one feature missing. Since you might want to take offline copy of everything that you have in your password management solution to some offsite place - in a form that does not require you to set-up your servers in case of disaster - you might need to copy out more than just your passwords.

Exporting passwords is easy, built-in feature. But exporting documents that you might have (like certificate files etc) is not that easy.

You have to use API-interface to download, and read file names from database. A bit tricky, and if you need to do that regularly, you need to have at least a (PowerShell) script for you to do that.

So here is what I did:

 
#SQL Functionality from:
#https://cmatskas.com/execute-sql-query-with-powershell/


Param(
 [string]$apikey,
 [string]$exportpath 
 )
 
if(-not$apikey) { Throw "Mandatory paramter Apikey not specified" }
if(-not$exportpath) { Throw "Mandatory paramter exportpath not specified" }
if(!(test-path $exportpath)){Throw "Exportpath " +$exportpath+ " does not exist"}

$ItemDocPath = $exportpath + "\PWItemDocuments"
$ListDocPath = $exportpath + "\PWListDocuments"


if(!(test-path $ItemDocPath)){New-Item -ItemType directory $ItemDocPath}
if(!(test-path $ListDocPath)){New-Item -ItemType directory $ListDocPath}



#ENVIRONMENT VARIABLES, MODIFY TO MATCH YOUR ENVIRONMENT
$Server = "yoursqlinstance"
$Database = "yourdatabase"
$PasswordStateSiteUrl = "https://yourpasswordstateurl/"
#ENVIRONMENT VARIABLES, MODIFY TO MATCH YOUR ENVIRONMENT

#DO NOT CHANGE THESE
$QueryListDocuments = $("SELECT * FROM ["+$Database+"].[dbo].[PasswordListDocuments]")
$QueryItemDocuments = $("SELECT * FROM ["+$Database+"].[dbo].[PasswordDocuments]")
$QueryPasswordLists = $("SELECT [PasswordListID],[PasswordList],[Description] FROM ["+$Database+"].[dbo].[PasswordLists]")
$QueryPasswords = $("SELECT [PasswordID],[Title],[PasswordListID] FROM ["+$Database+"].[dbo].[Passwords]")
#DO NOT CHANGE THESE

function ExecuteSqlQuery ($Server, $Database, $SQLQuery) {
    $Datatable = New-Object System.Data.DataTable
    $Connection = New-Object System.Data.SQLClient.SQLConnection
    $Connection.ConnectionString = "server='$Server';database='$Database';trusted_connection=true;"
    $Connection.Open()
    $Command = New-Object System.Data.SQLClient.SQLCommand
    $Command.Connection = $Connection
    $Command.CommandText = $SQLQuery
    $Reader = $Command.ExecuteReader()
    $Datatable.Load($Reader)
    $Connection.Close()
    
    return $Datatable
}

$PasswordLists = New-Object System.Data.DataTable
$PasswordLists = ExecuteSqlQuery $Server $Database $QueryPasswordLists

$Passwords = New-Object System.Data.DataTable
$Passwords = ExecuteSqlQuery $Server $Database $QueryPasswords

$ListDocuments = New-Object System.Data.DataTable
$ListDocuments = ExecuteSqlQuery $Server $Database $QueryListDocuments

$ItemDocuments = New-Object System.Data.DataTable
$ItemDocuments = ExecuteSqlQuery $Server $Database $QueryItemDocuments



foreach ($document in $ListDocuments) {
 $command = 'curl ' + $PasswordStateSiteUrl+'api/document/passwordlist/"'+$document.DocumentID+'"?apikey='+$apikey+' -OutFile "'+$ListDocPath+'\'+$document.DocumentID+'_'+$document.DocumentName+'"'
 invoke-expression $command

 $temp = $passwordlists|Where-Object {$_.PasswordListID -eq $document.PasswordListID}
 $pwlistname = $temp.PasswordList
 
 [array]$PWListDocuments += New-Object -TypeName PSObject -Property @{
   "OriginalFileName" = $document.DocumentName
   "DocumentID" = $document.DocumentID
   "Description" = $document.DocumentDescription
   "PasswordList" = $pwlistname
   "Modified" = $document.Modified
   "ModifiedBy" = $document.ModifiedBy
  }
 
 
 }

 
foreach ($document in $ItemDocuments) {
 $command = 'curl ' + $PasswordStateSiteUrl+'api/document/password/"'+$document.DocumentID+'"?apikey='+$apikey+' -OutFile "'+$ItemDocPath+'\'+$document.DocumentID+'_'+$document.DocumentName+'"'
 invoke-expression $command
 
 $temp = $Passwords|Where-Object {$_.PasswordID -eq $document.PasswordID}
 $PasswordTitle = $temp.Title
 $PasswordListID = $temp.PasswordListID
 
 $temp = $passwordlists|Where-Object {$_.PasswordListID -eq $PasswordListID}
 $pwlistname = $temp.PasswordList
 
 
 [array]$PWDocuments += New-Object -TypeName PSObject -Property @{
   "OriginalFileName" = $document.DocumentName
   "DocumentID" = $document.DocumentID
   "Description" = $document.DocumentDescription
   "PasswordItem" = $PasswordTitle
   "PasswordList" = $pwlistname
   "Modified" = $document.Modified
   "ModifiedBy" = $document.ModifiedBy
   
   
   
   
  }
 
 }

 
if ($PWListDocuments){
 $outputfile = $exportpath+'\PWListDocuments.csv'
 $PWListDocuments|Export-CSV -useCulture -NoType -Encoding UTF8 $outputfile
 }

 if ($PWDocuments)
 {
  $outputfile = $exportpath+'\PWItemDocuments.csv'
  $PWDocuments|Export-CSV -useCulture -NoType -Encoding UTF8 $outputfile
 }

So, this script does following things when you run it:

• Needs two parameters:
• apikey: you need to have a system wide API-key in your PasswordState
• exportpath: root path, where documents etc are exported
• this path needs to exist
• It creates two documents under exportpath, 'PWItemDocuments' and 'PWListDocuments'.
• Does four queries to your PasswordState database to read information that we need
• Exports all documents that are attached in password lists to 'PWListDocumnts' folder (with ID-number as a prefix)
• Exports all documents that are attached in password items to 'PWItemDocuments' folder (with ID-number as a prefix)
• Creates two .csv files to exportpath, with following info:
• PWListDocuments.csv
• Original filename
• Document ID in PasswordState
• Description of document
• Password list name where document is attached to
• Modified date
• Modifier
• PWItemDocuments.csv
• Original filename
• Document ID in PasswordState
• Description of document
• Password item name where document is attached to
• Password list name where that item is
• Modified date
• Modifier

Reason that I'm adding document ID as a prefix, is that you might have duplicate filenames in your attachments, and this way we can get all of those exported.

vRealize Orchestrator, connecting to vCenter

I have been working with vRealize Orchestrator for almost a year now, and I think that it's a hidden jewel in VMware stack. But I have to admit, it's not easiest tool to start playing with, so I'm planning to write about some things that I have done with it, starting with very basic stuff, and maybe at some point some more complicated (and more useful) stuff.

One of the first things to do, is to add your vCenter server to your Orchestrator. In Orchestrator, go to a workflow: "Library / vCenter / Configuration / Add a vCenter Server instance".  Right click workflow and select "Start workflow..."

I have vCenter appliance in address vcsa01.vlab.dom, so I add that to first field. Then I change last option (Ignore certificate warnings) to Yes and click next



On next page, we set-up account to be used when connecting to vCenter. I'm using my vCenter servers SSO-domain, so account is vlabsso.dom\administrator, you could also use AD service account dedicated for this. I also chose to use this account always when using orchestrator, so answer to first question is 'No'



After submitting this worklow, if everything goes well, we have added our vCenter to Orchestrator. You can add multiple vCenters to one Orchestrator.

To see that vCenter was added succesfully, go to inventory tab, and see that you can browse your vCenter environment.



Now we have succesfully added vCenter to our Orchestrator.

Next post: first workflows

Sysinternals 20yo party at Helsinki / Finland

This is mostly my personal notebook for today, but maybe you might be interested as well...

Sysinternals share: \\live.sysinternals.com\Tools\

Like almost always with video meetings, we start the day with debugging Skype meeting. We got good live experience, on how not to start a video conference: in a hurry.

After about 10 minutes, we finally are hearing Mark Russinovich giving welcome speech, but it ended too early, since we lost audio again...



After that hassle, we luckily have rest of the speakers on site. Next up, Aaron Margosis, giving opening speech.

Some examples of Sysinternals influence on Windows ecosystem


Tricks & tricks:
use '-ct | clip' -> it will format output to tab delimited and copies it to clipboard -> paste to excel
-nobanner switch is coming on future versions





PsExec -sid -> run as local system and don't wait for command to finish

du -> quite familiar from *nix, has nice features

streams -d * -> unblock files 'downloaded'


Now some security with Paula Januszkiewicz, maybe most interesting session, only small parts written here..



Logs all process activity to dedicated event log branch



Parsing logfiles



Extract hashes with takdefence pyhton tool (http://www.tekdefense.com/tekcollect/)


Check those hashes against https://www.virustotal.com/ database with API (tool: http://www.woanware.co.uk/forensics/virustotalchecker.html)




Configuring:









And lot's of cool demos of using sysmon !






https://github.com/gentilkiwi/mimikatz

After the lunch, Tim Mangan about Process Monitor and problem debugging on AppV

"Apps suck" - I love this guy already.

So, it's a lot ot App-V related stuff, so not in very high on my interest list, but still interesting to listen. Not much notes though.





Process monitor, use filter (ex. filter safe items etc), use highlight (ex. highlight result 'Success' and look for something that is not highlighted), save to PML with all data...


Short brake, then, Brian Catlin and Process Explorer.











Virtual vs. Physical address space





Daniel Pearson - LiveKd, ProcDump & NotMyFault



Whee, no powerpoints, only live demos

Procdump, "ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike."

Example, procdump notepad -> does .dmp of notepad process
To delay and do multiple dumps, procdump -s 5 -n 3
-c 25 -> take dump when CPU value is 25%, if you add -s 5, it has to be for 5 seconds of that CPU usage, can be used to monitor CPU spike and dump situation of that spike.
get examples, -? -e
procdump -e 1 -f "" -> filter for exceptions (monitor them all, does not dump)

You can use WinDBG to debug dump files.

LiveKD

https://technet.microsoft.com/en-us/sysinternals/livekd.aspx

You can "attach" to live windows

NotMyFault (my all time favorite tool)

Shortly, you can crash (cause BSOD), in multiple ways with this tool.



"How it's done in Enterprise environment" - Petri Paavola

- There is usually only one, or at best couple of guys who can really do troubleshooting.
- Overloads that dude

In client environment, you can use procmon remotely with PsExec -> get trace of error to experts.



Network tracing, use netsh since it's built-in.

netsh trace start ....

In client environment -> use PsExec to run it remotely.

Then use netmon / message analyzer to analyze that trace.

You can also use ProcMon to monitor network traffic.

WPA, Windows Performance Analyzer -> analyze slow boot times.


Mikko Järvinen - Troubleshooting extravaganza

Stories about client troubleshooting.. no notes, but nice stories :)



Nice day, most sessions were really deep dives, it's easy to understand that those guys are MVP's