My HomeLab setup

On end of last year, I started to slowly build up up a home lab to be able to test all kinds of things, and to study more some of our own applications.

And don't get me wrong here, we have a great demo environments, but they are something that you are not supposed to break down. And sometimes, to really understand that how things work, you actually need to break them.

So I decided that I finally need to build up my own home lab environment.

And here is my current physical setup:

VMware (all in version 6.7):

•  Virtual vCenter Appliance
• "Production" Cluster:
• 1x Intel NUC8i5BEK, i5-8259U @ 2.3 GHz, 64GB RAM
• 1x Intel NUC7i3BNK, i3-7100U @ 2.4 GHz, 32GB RAM
• 1x Omen by HP laptop, i7-8750H @ 2.20GHz, 16GB RAM. This one I occasionally boot up from USB-disk containing ESX installation, if I need temporarily more compute power, or to test some DR scenarios etc.

Hyper-V, standalone host, Win 2016 Core

• ASUS UX303L laptop, Intel® Core™ i7 4510U @ 2GHz, 8GB. This is my old laptop with broken display. Only sad thing with this is that it wont boot if it's not attached to some display while booting, have not figured out how to bypass that. This one uses local SSD in laptop, so no connection to shared storage.

Network:

• HP OfficeConnect Switch 1820-24G. This is reasonably priced (web)managed switch, and at least in my home lab 1 Gbps connections are fast enough. Still for my Storage I did a 2 Gbps trunk.

Storage:

• QNAP TS879 Pro, equipped with 8x 3TB disks. And configured to RAID-10. So more than enough of capacity and performance for my needs. This is my main storage for the VMware environment. Disks are both iSCSI and NFS (Why both? Well, because I can). Connected to my switch with 2x links, so I have 2 Gbps connection for storage.
• Buffalo LinkStation LS220D. This is my secondary storage for homelab and primary storage for my personal files. And I use it as a secondary backup target for my homelab, and use my QNAP as a primary backup target for files in this one.

Firewall: 

• pfSense, running in virtual machine. Only one node, but since I now have to hosts, I'm going to make this one HA

So, what do I do with this environment then, and what do I have running in there?

Well, besides the basic infra services that you just have to have (DNS, AD etc) I have installed for example:

• VMware Orchestrator 7.6 (<- I hopefully have some time to blog about this later)
• A lot of Veeam stuff, including:
• Veeam Backup & Replication 9.5U4b and V10 beta1 (<- Yes, it's out!)
• Veeam Availability Orchestrator v2 (<- I'm planning to blog about this also)
• Veeam One v9.5
• Veeam Backup for Office 365 v3

I have also been playing around with Minio lately. (Hint, it has a lot to do with Backup & Replication)

Best here is, that I can pretty much test anything I want to, and if (read: when) I mess up with something, it only affects me, not anyone else.

And, hopefully this will help me to write a lot more blog posts.

Windows 2008 R2 guest crashing on VMware or Hyper-V 2012 R2 with Intel E5 v2 series processor

Virtual Machine reboots with BSOD suddenly, with no reason? Hosts have Intel E5 v2 processor? And, if you have Hyper-V, you can find event like this:

"Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin

Source:        Microsoft-Windows-Hyper-V-Worker

Date:          x.x.2014 xx:xx:xx

Event ID:      18560

Task Category: None

Level:         Critical

Keywords:     

User:          NT VIRTUAL MACHINE\<guid>

Computer:      hyper-v-hostname

Description:

'<vmname>' was reset because an unrecoverable error occurred on a virtual processor that caused a triple fault. If the problem persists, contact Product Support. (Virtual machine ID )"

VMware has a good Knowledge Base artice about this issue: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2073791

But, it seems that this issue is also affecting Hyper-V 2012 R2 hosts.

I have had this problem on both platforms. At least Dell has published a BIOS upgrade that fixes this issue, quote from R720 BIOS 2.2.3 release notes:
"What’s New

===========

* Updated Intel Xeon processor E5-2600 V2 product family microcode to 427.

Fixes

======

* Corrected an issue where a complex sequence of internal processor events may result in unexpected page faults or use of incorrect page translations If EPT (Extended Page Tables) is enabled.   Due to this erratum a guest may crash or experience unpredictable system behavior."

So, if you run in to this problem, look for BIOS update from your HW vendor!

Duplicate VMs in SCVMM 2012 R2 after Live Migration

After migrating VMs cluster-to-cluster (or from stand-alone host to cluster), you might see duplicate VMs in SCVMM. Both VMs are in running state, properties are identical, you can connect to both of those etc. Everything works, but it does not look good.

This is how it looks like:

So, how to remove duplicates.

Way to detect which one is 'original' and which one is 'duplicate', select VM, check on recent job and see which one has been live migrated. (Actually, it does not seem to matter if you remove duplicate or original, it is just a SCVMM database entry that we are removing. VM actually keeps running even if you remove both of those)

Open properties of that VM and start doing some change, for example like this:

And click 'View Script'

It opens notepad, where we can see ID of this VM in SCVMM database.

And rest is done in SCVMM Powershell. First, use get-vm and check that we get two VMs with different ID.

get-vm <servername>|fl id

Then, we remove duplicate VM with ID that we just checked.

get-vm <servername>|where ID -eq "<ID>"|remove-vm -force

And now important thing, you have to use "-force" switch. Remove-vm (or to be precise, Remove-SCVirtualMachine) without -force, removes all files and configurations of that VM, but when you use -force, it only removes entry from SCVMM database. Obvious, right?

Here is documentation: http://technet.microsoft.com/en-us/library/hh801721.aspx and quote from that page: 'NOTE: When used with the Force parameter, Remove-SCVirtualMachine only deletes the virtual machine from the VMM database. It does not delete the virtual machine itself.'

And why this happens? Well, I don't know. It seems to be a bug in SCVMM. For some reason, sometimes SCVMM 'detects' that new VM has been added to cluster. It happens 15-30 seconds before Live Migration is finished. And when it happens, you get duplicate VMs.