New in Veeam Backup & Replication v10: Linux Repositories with XFS support

One of enhancements in Veeam Backup & Replication v10, is possibility to use XFS file system and FastClone, which is similar that Veeam has utilized with ReFS for years already.

But how to set-up it? First of all, you need to have Linux server that meet's requirements.

Basically what you need is:

• Ubuntu 18.0.4 or later. For other distributions, Fast Clone support is experimental.
• XFS file system
• Cyclic redundancy check (CRC) is enabled
• The minimum supported data block size is 1 KB. The maximum supported block size is 4KB

And you can format your disk like this:

mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/sdb1

This is how it looks:

Now we can add our repository to Veeam Backup & Replication, in console, go to Backup Infrastructure -> Backup Repositories -> Right click -> Add backup repository.

(Wan't to do it with PowerShell? Scroll to end of page)

Select "Direct attached storage"

Select Linux

Give name and description for your repository

If you have already added your Linux machine as managed server to Veeam, you can just select it from drop down menu, if not, click "Add new"

Enter FQDN or IP-address of Linux machine. Notice, that you need to have SSH and Perl installed.

Select credentials to use, typically it is best to use dedicated service account. If you have not previously entered account info, you can do it with "Add..."

Validate SSH key fingerprint

And in a short moment you see that server has been added

Now back in "New Backup Repository" wizard, your newly added server will be automatically selected.

Click browse, to select path that is used for this repository

Select folder..

Select "Use fast cloning on XFS volumes" checkbox

Select mount server. Notice, that you cannot use Linux repository as a mount server, it has to be a Windows machine. I'm using my Backup & Replication machine in this example.

You can review your settings

And when you click Apply, your new repo will be added

Finally you will see a summary page that confirms that new repository has been successfully created

We can, of course, do the same thing by using PowerShell, with just couple of lines.

First we get our credentials and then add Linux machine to Backup & Replication. If you already have Linux machine added, you can skip this step. As you can see, I was a bit lazy, and used screenshot from previous post, since process of adding Linux machine does not differ, if it's used as a Proxy or a Repository

Then we configure it as a repository. As you can see, we first read information about mount server and Linux server to variable, and then use them as parameters when we add repository.

You also need to set folder, type and use "-EnableXFSFastClone" to enable Fast Cloning in XFS volumes.

Actual code:

$credential = Get-VBRCredentials -Name <YOURCREDENTIALNAME>
Add-VBRLinux -Name <FQDN OF LINUX MACHINE> -Credentials $credential
$mountserver = Get-VBRServer -Name <YOURMOUNTSERVER>
$reposerver = Get-VBRServer -Name <YOURREPOSERVER>
Add-VBRBackupRepository -Name <YOURREPONAME> -Description <YOURDESCRIPTION> -Server $reposerver -MountServer $mountserver -Folder <YOURFOLDER> -Type LinuxLocal -EnableXFSFastClone

New in Veeam Backup & Replication v10: Linux Proxies

One of the many new features in Veeam Backup & Replication v10, is Linux proxy.

You can use Linux proxies in your VMware environment.

Adding new Linux proxy is really simple.

(Wan't to do it with PowerShell? Scroll to end of page)

In Backup & Replication console, go to Backup Infrastructure -> Backup Proxies -> Right click -> Add VMware backup proxy.

If you have added your Linux machine as a managed server to Backup & Replication, select it from drop down list, otherwise select "Add New..."

Select "Linux"

Enter FQDN or IP-address of Linux machine. Notice, that you need to have SSH and Perl installed.

Select credentials to use, typically it is best to use dedicated service account. If you have not previously entered account info, you can do it with "Add..."

Validate SSH key fingerprint

And in a short moment you see that server has been added

Now back in "New VMware proxy" wizard, your newly added server will be automatically selected.

Do note, that "Transport mode" and "Connected datastores" selections are grayed oyt. That is, because Linux proxy only works in "Virtual appliance" mode

If in VMware VM settings, you do not have disk.enabledUUID=TRUE parameter, you will get following warning. Select "Yes" to pick VM from vCenter manually

Select "Browse..." to select your VM

You can use search to find your machine

And you will see your machine selected

You will also get a warning that your Linux machine will be rebooted. This is because Veeam add's that missing parameter to VM's configuration

You can set Throttling settings if needed

And when you click "Apply" your proxy will be configured

And finally you get summary that states that Linux Proxy has been successfully created

And that's it! Your new Linux proxy is now ready to be used in backup jobs. If you are using automatic selection in job settings, it will automatically be used in next backup and replication jobs!


Looking for PowerShell magic? Well, here we go:


First we get our credentials and then add Linux machine to Backup & Replication. If you already have Linux machine added, you can skip this step

Same as with wizard, if you VM does not have disk.enabledUUID=TRUE parameter in place, you will get error when you try to add Linux proxy:

So we will first search our VM, and give it as a parameter to PowerShell command:

Code:

 

$credential = Get-VBRCredentials -Name <YOURCREDENTIALNAMEHERE>
Add-VBRLinux -Name <FQDN OF LINUX MACHINE> -Credentials $credential

$vientity = Find-VBRViEntity -Name "<VMNAME IN VSPHERE>"
Get-VBRServer -Name "<FQDN OF LINUX MACHINE>" | Add-VBRViLinuxProxy -ProxyVM $vientity

Exporting documents from PasswordState

I have been using PasswordState for couple of years, and I have to say that I like it.

But there is one feature missing. Since you might want to take offline copy of everything that you have in your password management solution to some offsite place - in a form that does not require you to set-up your servers in case of disaster - you might need to copy out more than just your passwords.

Exporting passwords is easy, built-in feature. But exporting documents that you might have (like certificate files etc) is not that easy.

You have to use API-interface to download, and read file names from database. A bit tricky, and if you need to do that regularly, you need to have at least a (PowerShell) script for you to do that.

So here is what I did:

 
#SQL Functionality from:
#https://cmatskas.com/execute-sql-query-with-powershell/


Param(
 [string]$apikey,
 [string]$exportpath 
 )
 
if(-not$apikey) { Throw "Mandatory paramter Apikey not specified" }
if(-not$exportpath) { Throw "Mandatory paramter exportpath not specified" }
if(!(test-path $exportpath)){Throw "Exportpath " +$exportpath+ " does not exist"}

$ItemDocPath = $exportpath + "\PWItemDocuments"
$ListDocPath = $exportpath + "\PWListDocuments"


if(!(test-path $ItemDocPath)){New-Item -ItemType directory $ItemDocPath}
if(!(test-path $ListDocPath)){New-Item -ItemType directory $ListDocPath}



#ENVIRONMENT VARIABLES, MODIFY TO MATCH YOUR ENVIRONMENT
$Server = "yoursqlinstance"
$Database = "yourdatabase"
$PasswordStateSiteUrl = "https://yourpasswordstateurl/"
#ENVIRONMENT VARIABLES, MODIFY TO MATCH YOUR ENVIRONMENT

#DO NOT CHANGE THESE
$QueryListDocuments = $("SELECT * FROM ["+$Database+"].[dbo].[PasswordListDocuments]")
$QueryItemDocuments = $("SELECT * FROM ["+$Database+"].[dbo].[PasswordDocuments]")
$QueryPasswordLists = $("SELECT [PasswordListID],[PasswordList],[Description] FROM ["+$Database+"].[dbo].[PasswordLists]")
$QueryPasswords = $("SELECT [PasswordID],[Title],[PasswordListID] FROM ["+$Database+"].[dbo].[Passwords]")
#DO NOT CHANGE THESE

function ExecuteSqlQuery ($Server, $Database, $SQLQuery) {
    $Datatable = New-Object System.Data.DataTable
    $Connection = New-Object System.Data.SQLClient.SQLConnection
    $Connection.ConnectionString = "server='$Server';database='$Database';trusted_connection=true;"
    $Connection.Open()
    $Command = New-Object System.Data.SQLClient.SQLCommand
    $Command.Connection = $Connection
    $Command.CommandText = $SQLQuery
    $Reader = $Command.ExecuteReader()
    $Datatable.Load($Reader)
    $Connection.Close()
    
    return $Datatable
}

$PasswordLists = New-Object System.Data.DataTable
$PasswordLists = ExecuteSqlQuery $Server $Database $QueryPasswordLists

$Passwords = New-Object System.Data.DataTable
$Passwords = ExecuteSqlQuery $Server $Database $QueryPasswords

$ListDocuments = New-Object System.Data.DataTable
$ListDocuments = ExecuteSqlQuery $Server $Database $QueryListDocuments

$ItemDocuments = New-Object System.Data.DataTable
$ItemDocuments = ExecuteSqlQuery $Server $Database $QueryItemDocuments



foreach ($document in $ListDocuments) {
 $command = 'curl ' + $PasswordStateSiteUrl+'api/document/passwordlist/"'+$document.DocumentID+'"?apikey='+$apikey+' -OutFile "'+$ListDocPath+'\'+$document.DocumentID+'_'+$document.DocumentName+'"'
 invoke-expression $command

 $temp = $passwordlists|Where-Object {$_.PasswordListID -eq $document.PasswordListID}
 $pwlistname = $temp.PasswordList
 
 [array]$PWListDocuments += New-Object -TypeName PSObject -Property @{
   "OriginalFileName" = $document.DocumentName
   "DocumentID" = $document.DocumentID
   "Description" = $document.DocumentDescription
   "PasswordList" = $pwlistname
   "Modified" = $document.Modified
   "ModifiedBy" = $document.ModifiedBy
  }
 
 
 }

 
foreach ($document in $ItemDocuments) {
 $command = 'curl ' + $PasswordStateSiteUrl+'api/document/password/"'+$document.DocumentID+'"?apikey='+$apikey+' -OutFile "'+$ItemDocPath+'\'+$document.DocumentID+'_'+$document.DocumentName+'"'
 invoke-expression $command
 
 $temp = $Passwords|Where-Object {$_.PasswordID -eq $document.PasswordID}
 $PasswordTitle = $temp.Title
 $PasswordListID = $temp.PasswordListID
 
 $temp = $passwordlists|Where-Object {$_.PasswordListID -eq $PasswordListID}
 $pwlistname = $temp.PasswordList
 
 
 [array]$PWDocuments += New-Object -TypeName PSObject -Property @{
   "OriginalFileName" = $document.DocumentName
   "DocumentID" = $document.DocumentID
   "Description" = $document.DocumentDescription
   "PasswordItem" = $PasswordTitle
   "PasswordList" = $pwlistname
   "Modified" = $document.Modified
   "ModifiedBy" = $document.ModifiedBy
   
   
   
   
  }
 
 }

 
if ($PWListDocuments){
 $outputfile = $exportpath+'\PWListDocuments.csv'
 $PWListDocuments|Export-CSV -useCulture -NoType -Encoding UTF8 $outputfile
 }

 if ($PWDocuments)
 {
  $outputfile = $exportpath+'\PWItemDocuments.csv'
  $PWDocuments|Export-CSV -useCulture -NoType -Encoding UTF8 $outputfile
 }

So, this script does following things when you run it:

• Needs two parameters:
• apikey: you need to have a system wide API-key in your PasswordState
• exportpath: root path, where documents etc are exported
• this path needs to exist
• It creates two documents under exportpath, 'PWItemDocuments' and 'PWListDocuments'.
• Does four queries to your PasswordState database to read information that we need
• Exports all documents that are attached in password lists to 'PWListDocumnts' folder (with ID-number as a prefix)
• Exports all documents that are attached in password items to 'PWItemDocuments' folder (with ID-number as a prefix)
• Creates two .csv files to exportpath, with following info:
• PWListDocuments.csv
• Original filename
• Document ID in PasswordState
• Description of document
• Password list name where document is attached to
• Modified date
• Modifier
• PWItemDocuments.csv
• Original filename
• Document ID in PasswordState
• Description of document
• Password item name where document is attached to
• Password list name where that item is
• Modified date
• Modifier

Reason that I'm adding document ID as a prefix, is that you might have duplicate filenames in your attachments, and this way we can get all of those exported.

Migrating VMs with Veeam Backup & Replication

I have been involved in a project, where we need to migrate hundreds of VMware VM's from one datacenter to another.

When thinking, that how we should do that migration, we decided to test if Veeam Replication functionality would be good for us. And it is. In our case, downtime for servers outside office hours, is mainly not an issue, so it did this even easier.

Why did we end up using Veeam?

• We are already using it as our backup solution, and also doing replication for DR purposes
• We have limited network capacity between datacenters, and Veeam compresses data while doing replication
• Network labels and subnets are different on datacenters, and Veeam can do IP address re-map, and also change VM's network, while doing replication, so it makes it faster to bring those servers back online at new DC.

How are we doing it?

We are having migration script, that does following steps

• Reads list of VMs that are in specific Veeam Replication job, used in migrations
• Shuts down those servers in old datacenter (tries to do it gently, but if it does not succeed, does brutal poweroff, none of which have happened so far)
• Changes some settings on those VM's before migration
• Starts Veeam Replication job to move servers to new datacenter

And when that is done, we manually do failover in Veeam, to start servers in new DC.

One notable thing about replication in our case. It would have been possible (and was our original idea), to migrate most of the data in beforehand, and when doing actual migration, just replicate changed data, and make actual migration time really short. Only problem is, that on old environment we noticed that CBT is corrupted on some machines, and it forced us to disable CBT on migration job.

Here is the script that we are using. It needs to run on Veeam Backup&Replication server, since we have not yet upgraded to v9.

This script can be run as a Scheduled task, so you can build your Replication job to be ready, and let this script do the work at night.

#Change these settings
$logpath = "c:\scripts\migration_log\"
$vCenter = "oldvcenter.vmware.local"
$vCenterUser "vmware\administrator"
$vCenterPass "P@ssw0rd"
$ReplicaJob = "VeeamReplicaJobName"

#Make sure that VMware snapins are loaded
if ( (Get-PSSnapin -Name VMware* -ErrorAction SilentlyContinue) -eq $null )
{
    Add-PsSnapin VMware*
}
#Add Veeam functions
& 'C:\Program Files\Veeam\Backup and Replication\Backup\Install-VeeamToolkit.ps1'


#This fuction is grabbed from: http://ict-freak.nl/2009/10/05/powercli-enabledisable-the-vm-hot-add-features/
Function Enable-vCpuHotAdd($vm){
    $vmview = Get-vm $vm | Get-View 
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="vcpu.hotadd"
    $extra.Value="true"
    $vmConfigSpec.extraconfig += $extra
    $vmview.ReconfigVM($vmConfigSpec)
}

#This fuction is grabbed from: http://ict-freak.nl/2009/10/05/powercli-enabledisable-the-vm-hot-add-features/
Function Enable-MemHotAdd($vm){
    $vmview = Get-vm $vm | Get-View 
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $extra = New-Object VMware.Vim.optionvalue
    $extra.Key="mem.hotadd"
    $extra.Value="true"
    $vmConfigSpec.extraconfig += $extra
    $vmview.ReconfigVM($vmConfigSpec)
}

#https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010048
Function Enable-AutoUpdate($vm){
 $vmview = $vm | Get-View
 $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
 $vmConfigSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo
 $vmConfigSpec.Tools.ToolsUpgradePolicy = "UpgradeAtPowerCycle"
 $vmview.ReconfigVM($vmConfigSpec)
}

#Starts logging of all activities in this script
$timestamp = get-date -Format yyyy.dd.MM.HH.mm.ss
$logfile = $logpath + $timestamp +".log"
start-transcript -path $logfile -append

#Connects to vCenter server
Connect-VIServer $vCenter -User $vCenterUser -Password $vCenterPass

#Gets list of VMs from replication job in Veeam 
$VMlist = Get-VBRJob -Name $ReplicaJob|Get-VBRJobObject|Where-Object {$_.type -like "Include"}|Select Name

write "VMs in replica job"
write $VMlist

# shutdown logic idea from: http://www.virtu-al.net/2010/01/06/powercli-shutdown-your-virtual-infrastructure/
$VMs = get-vm $VMlist.name
foreach($VM in $VMs)
{
 write "Shutting down VM " $VM.name
 $VM | Shutdown-VMGuest -Confirm:$false
}

#Set the amount of time to wait before assuming the remaining powered on guests are stuck 
$waittime = 1800 #Seconds

$Time = (Get-Date).TimeofDay
do {
    #Wait for the VMs to be Shutdown cleanly
    sleep 1.0
    $timeleft = $waittime - ($Newtime.seconds)
    $numvms = (Get-VM $VMlist.name | Where { $_.PowerState -eq "poweredOn" }).Count
    Write "Waiting for shutdown of $numvms VMs or until $timeleft seconds"
    $Newtime = (Get-Date).TimeofDay - $Time
    } 
 until ((@(Get-VM $VMlist.name | Where { $_.PowerState -eq "poweredOn" }).Count) -eq 0 -or ($Newtime).Seconds -ge $waittime)
 
 
$ForceShutdown = Get-VM $VMlist.name | Where { $_.PowerState -eq "poweredOn" }
 
if ($ForceShutdown) {
 Write "Starting forced poweroff for following servers"
 write $ForceShutdown
 Stop-VM $ForceShutdown -Confirm:$false
}

#Enable Memory hotadd, cpu hotadd and VMware tools autoupdate 
foreach($VM in $VMs)
{
 write "Change settings on VM: $VM"
 Enable-MemHotAdd $VM
 Enable-vCpuHotAdd $VM
 Enable-AutoUpdate $VM
}

#Finally start replication job
Start-VBRJob -Job $ReplicaJob

Modifying .XML files with Powershell

I was asked to help on one application upgrade recently, and it required modifying hundreds of XML-files. All changes were similar, so it was perfect place to do some script automation.

Note, this is not the only way to modify XML-files, this is my way :)

How do I modify XML-files with PowerShell? That's exactly the question that I had in my mind. But first, the XML file that I had. This is not the actual file, but it has same elements so we can use this as an example:

<WEBSETTINGS>
   <Locations>
    <Url value="~/Change/needed.html" />
    <LogDirectory value="\Logs" />
    <NotNeeded value="This we need to remove" />
  </Locations>
  <Settings>
    <Version value="1.0.0" />
    <SmtpServer value="smtp.example.com" />
  </Settings>
</WEBSETTINGS>

Now that we have a XML file, how do we read that with powershell? Like this:

$XML = New-Object -TypeName XML
$XML.Load("C:\XML-example\example.xml")

That will read file C:\XML-example\example.xml to variable called $XML. And now we can browse through XMLwith that variable, like you can see in this screenshot:

Wow, that's easy.

Now let's try to modify our XML. And that's as easy as setting a value to variable, like this:

$XML.WEBSETTINGS.Locations.Url.value = "http://scattereditnotes.blogspot.com"

And that's it, we have changed value of attribute "Url".

Then we add totally new attribute to our XML, it's a little bit more work, but still quite easy. So, let's add attribute called 'NewAttribute' with value 'New Value', and we add that under section 'Settings'. And code to do so:

$NewRow='<NewAttribute value="New Value"/>'
$NewXmlAttribute = $XML.CreateDocumentFragment()
$NewXmlAttribute.InnerXml = $NewRow
$node=$XML.SelectSingleNode('//Settings')
$node.AppendChild($NewXmlAttribute)

Now, let's remove one not needed attribute. Since we know exactly what attribute we want to remove, it's easy:

$RemoveAttribute = $XML.WEBSETTINGS.Locations.NotNeeded
$XML.WEBSETTINGS.Locations.RemoveChild($RemoveAttribute)

So, now we have done following things, we read XML-file to variable, modified value of attribute, added new attribute, and deleted one. Only one thing to do, save it.

$XML.Save("C:\XML-example\example.xml")

Whole code is here:

#Read XML-file to variable
$XML = New-Object -TypeName XML
$XML.Load("C:\XML-example\example.xml")
#Modify Attribute
$XML.WEBSETTINGS.Locations.Url.value = "http://scattereditnotes.blogspot.com"
#Add new attribute to section Settings
$NewRow='<NewAttribute value="New Value"/>'
$NewXmlAttribute = $XML.CreateDocumentFragment()
$NewXmlAttribute.InnerXml = $NewRow
$node=$XML.SelectSingleNode('//Settings')
$node.AppendChild($NewXmlAttribute)
#Remove attribute
$RemoveAttribute = $XML.WEBSETTINGS.Locations.NotNeeded
$XML.WEBSETTINGS.Locations.RemoveChild($RemoveAttribute)
#Save changes back to XML-file
$XML.Save("C:\XML-example\example.xml")

Nice, but, my task was to modify hundreds of .XML-files. Files were in directories under main directory, one file per directory. And I also wanted to make a backup file of all the original files.

And here is the final code:

#Get all directories to variable
$Directories = get-childitem|where {$_.Attributes -eq 'Directory'}|select FullName
#Go through all directories
foreach ($Directory in $Directories)
{
 #Read XML-file to variable
 $XML = New-Object -TypeName XML
 $XML.Load($Directory.FullName+"\example.xml")
 #Create backup of file
 $XML.Save($Directory.FullName+"\example.xml.backup")
 #Modify Attribute
 $XML.WEBSETTINGS.Locations.Url.value = "http://scattereditnotes.blogspot.com"
 #Add new attribute to section Settings
 $NewRow='<NewAttribute value="New Value"/>'
 $NewXmlAttribute = $XML.CreateDocumentFragment()
 $NewXmlAttribute.InnerXml = $NewRow
 $node=$XML.SelectSingleNode('//Settings')
 $node.AppendChild($NewXmlAttribute)
 #Remove attribute
 $RemoveAttribute = $XML.WEBSETTINGS.Locations.NotNeeded
 $XML.WEBSETTINGS.Locations.RemoveChild($RemoveAttribute)
 #Save changes back to XML-file
 $XML.Save($Directory.FullName+"\example.xml")
}

This is what our XML-file looks like after our modification.

<WEBSETTINGS>
  <Locations>
    <Url value="http://scattereditnotes.blogspot.com" />
    <LogDirectory value="\Logs" />
  </Locations>
  <Settings>
    <Version value="1.0.0" />
    <SmtpServer value="smtp.example.com" />
    <NewAttribute value="New Value" />
  </Settings>
</WEBSETTINGS>

And that is so beautiful.