How VMware snapshots affect Veeam backup efficiency?

Recently I was given task to take over one VMware environment which has Veeam backup in use. When going through the environment, I found out that some backup jobs take way more time that what I have usually seen and started investigating this issue.

First thing that I found was, that there are lot of snapshots in that environment, also in those servers that are backed up with Veeam. Little search at Veeam's Forums and I found this topic: http://forums.veeam.com/veeam-backup-replication-f2/can-snapshots-slow-down-running-backup-t22521.html

So, it's quite obvious, if you have snapshots on those VM's that you are backing up with Veeam, it will slow it down because consolidation is slower with nested snapshots. (So, backup itself is still fast, but 'cleaning up' after backup is slower)

But, is it really that much slower?

I had to follow the situation for a while since I was quite curious to see that how big impact does snapshots have. And the impact, it's way bigger than what I suspected.

This table shows impact of removing snapshots from VM's:

VM	Snapshot size in GB	Average time to do consolidation with old snapshots	Average time to do consolidation with just Veeam snapshot
Machine 1	9	1 minute	15 seconds
Machine 2	78	30 minutes	30 seconds
Machine 3	8	2 minutes	under 10 seconds
Machine 4	25	12 minutes	under 10 seconds
Machine 5	16	2 minutes	under 10 seconds
Machine 6	31	10 minutes	under 10 seconds
Machine 7	4	1,5 minutes	15 seconds
Machine 8	87	30 minutes	under 10 seconds
Machine 9	34	15 minutes	under 10 seconds
Machine 10	23	10 minutes	under 10 seconds

Conclusions? Don't keep snapshots on your environment if you don't need them, it will slow down things. And this is just one thing that it slows down, it of course affects to VM performance also, but there are lot of good blog articles about that already. With only 10 VMs, time saved with just removing old snapshots was almost 2 hours. You can just imagine that how much time you can save in you backup jobs if you have ten's or even hundreds of VMs that have (forgotten) snapshots.

And of course, impact is also dependent on your overall storage performance. In this case, underlying storage is quite reasonable, so with slow storage, impact might be even bigger.

Modifying .XML files with Powershell

I was asked to help on one application upgrade recently, and it required modifying hundreds of XML-files. All changes were similar, so it was perfect place to do some script automation.

Note, this is not the only way to modify XML-files, this is my way :)

How do I modify XML-files with PowerShell? That's exactly the question that I had in my mind. But first, the XML file that I had. This is not the actual file, but it has same elements so we can use this as an example:

<WEBSETTINGS>
   <Locations>
    <Url value="~/Change/needed.html" />
    <LogDirectory value="\Logs" />
    <NotNeeded value="This we need to remove" />
  </Locations>
  <Settings>
    <Version value="1.0.0" />
    <SmtpServer value="smtp.example.com" />
  </Settings>
</WEBSETTINGS>

Now that we have a XML file, how do we read that with powershell? Like this:

$XML = New-Object -TypeName XML
$XML.Load("C:\XML-example\example.xml")

That will read file C:\XML-example\example.xml to variable called $XML. And now we can browse through XMLwith that variable, like you can see in this screenshot:

Wow, that's easy.

Now let's try to modify our XML. And that's as easy as setting a value to variable, like this:

$XML.WEBSETTINGS.Locations.Url.value = "http://scattereditnotes.blogspot.com"

And that's it, we have changed value of attribute "Url".

Then we add totally new attribute to our XML, it's a little bit more work, but still quite easy. So, let's add attribute called 'NewAttribute' with value 'New Value', and we add that under section 'Settings'. And code to do so:

$NewRow='<NewAttribute value="New Value"/>'
$NewXmlAttribute = $XML.CreateDocumentFragment()
$NewXmlAttribute.InnerXml = $NewRow
$node=$XML.SelectSingleNode('//Settings')
$node.AppendChild($NewXmlAttribute)

Now, let's remove one not needed attribute. Since we know exactly what attribute we want to remove, it's easy:

$RemoveAttribute = $XML.WEBSETTINGS.Locations.NotNeeded
$XML.WEBSETTINGS.Locations.RemoveChild($RemoveAttribute)

So, now we have done following things, we read XML-file to variable, modified value of attribute, added new attribute, and deleted one. Only one thing to do, save it.

$XML.Save("C:\XML-example\example.xml")

Whole code is here:

#Read XML-file to variable
$XML = New-Object -TypeName XML
$XML.Load("C:\XML-example\example.xml")
#Modify Attribute
$XML.WEBSETTINGS.Locations.Url.value = "http://scattereditnotes.blogspot.com"
#Add new attribute to section Settings
$NewRow='<NewAttribute value="New Value"/>'
$NewXmlAttribute = $XML.CreateDocumentFragment()
$NewXmlAttribute.InnerXml = $NewRow
$node=$XML.SelectSingleNode('//Settings')
$node.AppendChild($NewXmlAttribute)
#Remove attribute
$RemoveAttribute = $XML.WEBSETTINGS.Locations.NotNeeded
$XML.WEBSETTINGS.Locations.RemoveChild($RemoveAttribute)
#Save changes back to XML-file
$XML.Save("C:\XML-example\example.xml")

Nice, but, my task was to modify hundreds of .XML-files. Files were in directories under main directory, one file per directory. And I also wanted to make a backup file of all the original files.

And here is the final code:

#Get all directories to variable
$Directories = get-childitem|where {$_.Attributes -eq 'Directory'}|select FullName
#Go through all directories
foreach ($Directory in $Directories)
{
 #Read XML-file to variable
 $XML = New-Object -TypeName XML
 $XML.Load($Directory.FullName+"\example.xml")
 #Create backup of file
 $XML.Save($Directory.FullName+"\example.xml.backup")
 #Modify Attribute
 $XML.WEBSETTINGS.Locations.Url.value = "http://scattereditnotes.blogspot.com"
 #Add new attribute to section Settings
 $NewRow='<NewAttribute value="New Value"/>'
 $NewXmlAttribute = $XML.CreateDocumentFragment()
 $NewXmlAttribute.InnerXml = $NewRow
 $node=$XML.SelectSingleNode('//Settings')
 $node.AppendChild($NewXmlAttribute)
 #Remove attribute
 $RemoveAttribute = $XML.WEBSETTINGS.Locations.NotNeeded
 $XML.WEBSETTINGS.Locations.RemoveChild($RemoveAttribute)
 #Save changes back to XML-file
 $XML.Save($Directory.FullName+"\example.xml")
}

This is what our XML-file looks like after our modification.

<WEBSETTINGS>
  <Locations>
    <Url value="http://scattereditnotes.blogspot.com" />
    <LogDirectory value="\Logs" />
  </Locations>
  <Settings>
    <Version value="1.0.0" />
    <SmtpServer value="smtp.example.com" />
    <NewAttribute value="New Value" />
  </Settings>
</WEBSETTINGS>

And that is so beautiful.

VMworld 2014, day after

How do you get from beginning of autumn to end of autumn in four hours? Fly from Barcelona to Helsinki :)

So, VMworld is now behind and it's time to summarize my feelings right after the event.

Was it worth it? 

As a whole, absolutely. But I have to open this a little bit.

I'm 100% technical guy, so I'm quite bad target for a marketing propaganda. And on a conference like that, there is lot's of propaganda for you to watch and hear. For example partner day did not give me much, BUT, as a first timer, it was not total waste either (I'll explain this later)

Some of the brake-out sessions were awesome (example: VAPP2979 - Advanced SQL Server on vSphere Techniques and Best Practices), but some were not that great. Even some deepdive sessions feel more like a marketing sessions than technical sessions. But mostly those sessions that I attended, were good.

In Solutions Exchange there were lots of vendors marketing their own products and solutions. I had some good discussions with different vendors over there. But did I really get much out from it? To be honest, maybe not us much us I could have.

There was also place where to do Hands-On-Labs, aka. HOL's. Of course, you can do those anytime, anywhere, if you just have internet connection. But do you have time to do those at work? Or do you want to spend your evenings doing these at home? Most of us don't have that time. So, VMworld was good place to to those also.

And of course, certification center. This year it was possible to do with tests with 50% discount.

How it felt for a first timer?

First of all, this was first big IT-congress for me, so in that way, everything was new and exciting. And for being there for the first time, that partner day was not that waste. It was a good day to walk around and find out where everything is. And to do HOL's, because even in VMworld, you don't have time to do those because there is so many interesting sessions.

Schedule building was challenge, but I think that I did it quite well. One sad thing was, that it was only two weeks before conference that I had possibility to register to event, and start scheduling break-out sessions. So, some of the most interesting sessions were already full. So, book early, get to the best sessions.

Schedule was also challenge from different angle. Congress center was huge, and it took time to walk from one place to another. For example, if you had 30 minutes time between sessions, it didn't make sense to walk to Solutions Exchange because you would need to go back to sessions hall almost immediately. For that reason I skipped some sessions to crawl around Solutions Exchange. And on one day, I actually missed warm meal because I had built myself too tight schedule.

Some days were also quite long, on Wednesday last session ended at 18:30 (and after that, there was VMworld party :) )

Again on next year?

I missed some very interesting session because they were fully booked, so why not. Tt's hard to say at this point if is it possible for me to go there next year. But one thing is sure, if I'm going, I have to be able to get my conference registration done way earlier than this year.

This years event was great for me, I learned some new things and got lot's of small tips. And, I also made some new friends and got new contacts, things that are often underestimated, but are something that money cannot buy.

VMworld 2014 Europe, day four

(Note! This blog post is a personal notepad for the day, so it might be a little bit confusing to read :))

Last day starting, first session:

STO1153 - Performance Best Practices to Run Virtualized Applications on Virtual SAN

Some benchmark info, tested with 32 node cluster.

What marketing slides say, when you scale out VSAN, you get linear performance boost..

Some slides:

Some testing applications that had been used:

VIP 2.0 (Beta): VSAN Qualification Assessment

• Can be used to collect and analyze information from your current environment, and gives you suggestion on which VMs will work well on VSAN and how to size VSAN environment.

When using SSD cache, you need to give it some time to 'warm up'

Tools for VSAN, VSAN Observer (KB 2064240)

When configuring for performance:

• Check read & write performance of SSD disks
• Controllers, queue depth is important

VAPP2979 - Advanced SQL Server on vSphere Techniques and Best Practices

One of the sessions that I have been waiting for, let's hope that this is good.

Virtualizing business critical applications

http://www.vmware.com/solutions/business-critical-apps

Design for Performance, not just Capacity!

Storage multipahting, minimum of four paths from ESX server to storage array.

Dedicated datastores for DB servers is recommended.

Configure max queue depth if needed (KB1267)

When to use RDM:

• shared disk failover clusters
• SAN management tools in use

On all other situations, use VMFS!

Always create vmdk files with eager zeroed thick disks.

Ensure that blocks are aligned at both the ESXi and Windows levels.

Small LUNs = Better performance

Use multiple vSCSI adapters to evenly distribute workload.

Configure ESXi Large Pages (2MB) when running Tier 1 applications. Improves performance.

Use larges pages on Windows guest level also.

Make sure that there is no swapping in either host or guest level!

• set memory reservation for VMs to avoid ballooning/swapping
• do not overcommit memory 

ESXi memory features that help avoiding swapping:

• Transparent Page Sharing
• Ballooning
• Do not turn those on when running Tier 1 applications like SQL!

Use VMXNET3 Paravirtualized adapter to increase performance

Tune Guest OS network buffers, maximum ports

Cores per Socket? Keep default, so 1 core per socket.

PVSCSI adapters

• Less CPU overhead
• Not supported for ANY type of Windows Clustering configuration!
• Applies also on Always-on clusters, but there is no technical limitation to use it on Always-on

Memory optimization on SQL side:

Check block alignment on disks!

Consolidation options.

Scale out approach gives some benefits. Don't build up too huge VMs for SQL.

Use DRS as your insurance policy, but don't rely on it for resource planning.

VMware support for MS Clustering on vSphere, KB1037959

So far, that was most interesting breakout session that I have attended.

INF2427 - DRS: Advanced Concepts, Best Practices and Future Directions

Mechanisms:

• Initial placement -> maximize cluster utilization
• Load balancing
• Resource control (reservations, shares, limits, resource pools)
• Cluster rules (like. affinity / anti-affinity)

DRS Load balancing:

• Goal -> keep the cluster imbalance within a target threshold
• Does not try to completely balance the cluster
• Challenges
• Dynamic load in VMs
• Balance both memory and CPU
• Goal should be achieved with the least impact on VM performance

Measuring imbalance

No load-balancing if imbalance < target threshold

Load balancing Key points:

• Cluster imbalance is not a bad thing
• constraints may prevent DRS from load-balancing the cluster
• you should look at wheter VMs are happy
• Don't use too aggressive migration threshold settings -> will lead to more vmotions in system
• Default setting of 3 works great in most cases

CPU and Memory over-commitment

• Is over-commitment good?
• Yes, as long as the VMs get what they want/need
• Enables efficient utilization of resources
• don't go too far with over-commitment
• Use reservations on important servers

Options: Control CPU Over-commitment

• Option: MaxVcpusPerClusterPct

Control MEM over-commitment

• Option: MaxClusterMemOvercommitPct & MaxHostMemOvercommitPct

CPU Ready Time tips

• Ready time is a good indicator of contention
• Reality: Some ready time is ok in virtualized environments
• DRS demand calculation takes ready time into account
• High CPU ready? -> Sometimes manual vMotion is answer to get improved performance
• Don't use power management options in bios (or give control to OS)

vSphere 60 new things:

•  Network awareness
• Network bandwidth reservations for VMs
• DRS Placement Across clusters
• Support placement of VMs across clusters

• Provide placement recommendations for cross-VirtualCenter vMotions.
• Improved Static Memory Overhead Estimation
• Reduces overhead memory on VMs

Future Directions

https://labs.vmware.com/flings/proactive-drs

MGT2487 - Built to be Extended: vCloud Automation Center

It's now vRealize Automation....

For some reason, I had hard time writing up anything about this session to blog :)

INF1601 - Taking reporting and command line automation to the next level with PowerCLI

Last session of this day, and actually last session of VMworld for me. They are already shutting things down at congress center.

PowerCLI 5.8, what's new:

Best practices:

• Learn RegEx expression!
• Retrieve information from outside, and use it on your scripts (fetch data from external web-pages and use that information on your scripts)
• From idea to Function
• Start with simple script, but if possible, develop and generalize your script so that you can share it to community

Reporting:

• vSphere reporting is safest place to start :)
• vCheck (http://virtu-al.net)
• Statistical reporting, Get-Stat, Get-StatInterval
• Stats toolbox: http://www.lucd.info/2014/09/02/stats-toolbox/

Desired State Configuration (DSC)

• Introducd with PS v4
• Two modes: push & pull
• Main purpose: capture configuration drifs
• Also:
• initial deployment
• monitoring

That's it, VMworld is over.

I will write up some thoughts about the event later on.

VMworld 2014 Europe, day three

(Note! This blog post is a personal notepad for the day, so it might be a little bit confusing to read :))

Morning started again with General Session. Maybe biggest thing for me was enhancements to Fault Tolerance and vMotion. But, all this was already announced in VMworld 2014 US..

BTW. you can watch keynote speeches and many other videos from VMworld here


HBC2638 - Ten Vital Best Practices for Effective Hybrid Cloud Security

If you have tools like IPS/IDS, AV, automation etc used in your private cloud, extend those components to public cloud too.

Make sure that path between public and private cloud is secure.

How does public cloud do secure data destruction for failed devices?

Examples of commonly ignored threats

• Social engineering
• 'Watering holed' (social meetings, pubs, bars etc.)
• Logical
• Crack / Penetration
• DDOS
• Physical

What to do:

• Design review
• Penetration testing
• Use NIDS (Network Intrusion Detection System) + HIDS (Host-based Intrusion Detection System)
• Host based AV / Filtering

Educate, not only you IT staff, but also your other employees!

Threat your credentials like cattle, not pets

Threat all systems like cattle, not pets.

Automate us much as possible, treat all manual system interaction as a liability

INF3014-SPO Scripting & Automation

Good discussion about scripting and automation in general, discussion on recursive treewalks with scripts and collecting performance data.

Request for all: share you scripts to community!

Vagrant, development environment, worth checking out

• automated setup of operatins systems & applications
• testing your automation

vCenter server simulator, this is something that I had never heard of, but immediately found some links like this

NET1974 Multi-Site Data Center Solutions with VMware NSX

Yet another NSX session, all of these are big sessions, so interest on NSX is huge. In a nutshell, everything told is in a topic, you can build quite nice multi-site solution with NSX. :)

STO2997-SPO The vExpert storage game show EMEA 

Jeopardy style gameshow, PureStorage vs. VMware. Quite fun to watch and lot's of small information. But impossible to even try to write up this stuff.

Last session of this day: SDDC1176 - Aske the Experts vBloggers

Link tip: http://vmwaretips.com/

Interesting chat with four vBloggers, and hosted by VCDX, lot's of VMware experience on same stage.

On those last two sessions of day, lot's of little and interesting things, but nothing to write on a blog about :)

Day will end with a VMworld 2014 Europe party...

VMworld 2014 Europe, day two

(Note! This blog post is a personal notepad for the day, so it might be a little bit confusing to read :))

Day started with a General Session. Lot's of hype and marketing jargon, also some new product releases (most of them already done at VMworld US)

Good list of new releases can be found here.

Lots or rebranding

STO 2197 - Storage DRS: Deepdive and Best Practices:

Storage DRS basic functionality:

• Ease of Storage Management
• Initial placement (automatically select datastore)
• Out of Space Avoidance
• IO Load Balancing
• Virtual Disk Affinity (Anti-Affinity)
• Datastore maintenance mode

Storage DRS can handle space over-commit either in Hypervisor level or in Datastore level through VASA integraion..
Best practice: never combine those!

Many new features are in vSpere 6.0

• Brake large I/O to 32KB
• Reservations to storage I/O
• If reservation cannot be satisfied, Storage DRS will kick in.
• Fine grain automatic actions

Best practices:

• avoid mixing of vSphere LUNs and non-vSphere LUNs on the same physical storage
• Configure host IO queue size with highest allowed value
• Keep congestion threshold conservatively high
• More datastores = better IO control, bigger datastore = better storage management

After that session, I went to do VCP5-DCV test, and...... Finally, achieved VCP status! :)

NET1743 - VMware NSX - A Technical Deep Dive

VMware is talking a lot about NSX, but at least in Finland, when can you buy it? How much does it cost? Technology is very interesting, might even say very cool.

I'm not going to cover this NSX  session, since lot of that information is already available on VMware blogs and documents. One link: www.vmware.com/products/nsx/resources.html

NET1586 - Advanced Network Services with NSX

NSX: offers:

• Switching / DCHP server-or-relay / DNS
• Routing / NAT
• Firewalling
• Load Balancing
• L2 / L3 VPN

Firewall is centrally managed, distributed FW. Works in front of VMNIC.

Security within the same IP-subnet.

Buzzword: micro-segmentation!

FW rules are not dependent on IP-addresses, you can use object names VMware to build rules. And those rules can be dynamic.

INF1469 - Extreme Performance Series: Monster VM Performance

Do idle vCPU cause co-stop? No they don't, there is no co-stop overhead on idle vCPUs.

Single threaded app -> multiple vCPU -> might even hurt performance

Over-provisioning vCPUs is not free -> Only configure as many vCPUs as the app needs!

Is %CSTP (co-stop) indicating a performance issue?

• %RDY + %CSTP = less than a few percent per vCPU should be ok

To avoid co-stop, address the slowest vCPU

• co-stop begins if Progress(vCPU) - Progress(vCPU_slowest) > costop_threshold
• Possible causes for vCPU_slowest
• High ready time (%RDY)
• High wait time (%VMWAIT = %WAIT - %IDLE)
• Snapshot, delta-disks, linked-clones
• VMM-level synchronization (memory manipulation: page sharing, NUMA remapping etc)

NUMA

• non-uniform memory access system architecture
• each node consists of CPU cores and memory
• pCPU can cross NUMA nodes, but at a performance cst

vNUMA

• Problem
• BIG VM has more vCPUs than # cores per NUMA node
• Guest app/OS has no knowledge of underlying NUMA
• Solutions:
• Expose virtual NUMA to VM
• Guest app/os achieves optimal placement of processes and memory

vNUMA is enabled:

• HW version higher than 8
• VM is wide (More vCPU per VM than # cores per NUMA node)
• Hyperthreading does not count
• by default, is configured with VMs with more than 8vCPU

How to ensure optimal vNUMA?

• Size VM to be multiples of pNUMA size
• 8/16/24... vCPU VMs for 8core/node
• 10/20/30... vCPU VMs for 10 cores/node

Virtual Sockets & Virtual Cores per Socket

• Add sockets, don't cores (if licensing allows you to)

Memory overcommitting = not recommended. Leave 10% room for ESXi.

• On huge VMs (large memory), consider using memory reservation to avoid creating huge swap files (.vswp) on datastore

Now the day is over (1800 local time), and it's time to celebrate my VCP status! 

VMworld 2014 Europe, day one

First day on VMworld 2014 Europe, and first time in VMworld ever.

Congress center (Fira Barcelona Gran Fia) is huge so there is going to be a lot of walking during next days.

First day is only for partners, and most of the sessions are marketing material. So, it was a great day to do at least one HOL (Hands-On-Lab). I tested how OpenStack can be used to manage vSphere, and there was also NSX in use (LAB: "OpenStack with VMware vSphere and NSX, SPL-SDC-1420). Very interesting lab I must say. Also it was good to explore the whole area, so on next days I will know where to find everything.

Technical sessions will start tomorrow and I’m also going to do (or at least trying to J) VCP exam, an hopefully gaining (finally) VCP status.

Windows 2008 R2 guest crashing on VMware or Hyper-V 2012 R2 with Intel E5 v2 series processor

Virtual Machine reboots with BSOD suddenly, with no reason? Hosts have Intel E5 v2 processor? And, if you have Hyper-V, you can find event like this:

"Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin

Source:        Microsoft-Windows-Hyper-V-Worker

Date:          x.x.2014 xx:xx:xx

Event ID:      18560

Task Category: None

Level:         Critical

Keywords:     

User:          NT VIRTUAL MACHINE\<guid>

Computer:      hyper-v-hostname

Description:

'<vmname>' was reset because an unrecoverable error occurred on a virtual processor that caused a triple fault. If the problem persists, contact Product Support. (Virtual machine ID )"

VMware has a good Knowledge Base artice about this issue: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2073791

But, it seems that this issue is also affecting Hyper-V 2012 R2 hosts.

I have had this problem on both platforms. At least Dell has published a BIOS upgrade that fixes this issue, quote from R720 BIOS 2.2.3 release notes:
"What’s New

===========

* Updated Intel Xeon processor E5-2600 V2 product family microcode to 427.

Fixes

======

* Corrected an issue where a complex sequence of internal processor events may result in unexpected page faults or use of incorrect page translations If EPT (Extended Page Tables) is enabled.   Due to this erratum a guest may crash or experience unpredictable system behavior."

So, if you run in to this problem, look for BIOS update from your HW vendor!

Duplicate VMs in SCVMM 2012 R2 after Live Migration

After migrating VMs cluster-to-cluster (or from stand-alone host to cluster), you might see duplicate VMs in SCVMM. Both VMs are in running state, properties are identical, you can connect to both of those etc. Everything works, but it does not look good.

This is how it looks like:

So, how to remove duplicates.

Way to detect which one is 'original' and which one is 'duplicate', select VM, check on recent job and see which one has been live migrated. (Actually, it does not seem to matter if you remove duplicate or original, it is just a SCVMM database entry that we are removing. VM actually keeps running even if you remove both of those)

Open properties of that VM and start doing some change, for example like this:

And click 'View Script'

It opens notepad, where we can see ID of this VM in SCVMM database.

And rest is done in SCVMM Powershell. First, use get-vm and check that we get two VMs with different ID.

get-vm <servername>|fl id

Then, we remove duplicate VM with ID that we just checked.

get-vm <servername>|where ID -eq "<ID>"|remove-vm -force

And now important thing, you have to use "-force" switch. Remove-vm (or to be precise, Remove-SCVirtualMachine) without -force, removes all files and configurations of that VM, but when you use -force, it only removes entry from SCVMM database. Obvious, right?

Here is documentation: http://technet.microsoft.com/en-us/library/hh801721.aspx and quote from that page: 'NOTE: When used with the Force parameter, Remove-SCVirtualMachine only deletes the virtual machine from the VMM database. It does not delete the virtual machine itself.'

And why this happens? Well, I don't know. It seems to be a bug in SCVMM. For some reason, sometimes SCVMM 'detects' that new VM has been added to cluster. It happens 15-30 seconds before Live Migration is finished. And when it happens, you get duplicate VMs.