Sunday, September 8, 2019

Using F-Secure Server Protection with Veeam Secure Restore

Do note, this is not a "best practice" guide, rather an example how I did it in my homelab, and it demonstrates how easy it is to integrate AV scanner to Secure Restore feature in Veeam Backup & Replication, Update 4


Veeam introduced feature called "Secure Restore", with version 9.5 Update 4 of Backup & Replication, that was released on January 2019.

It allows you to scan machine data with antivirus software before restoring it to the production environment. And you can also use this when testing your backups with Sure Backup.

You can integrate basically any AV software with Veeam, as long as your AV supports command-line scanning.

How Secure Restore works, and how to integrate AV software, is documented here: https://helpcenter.veeam.com/docs/backup/vsphere/av_scan_about.html?ver=95u4

But I wanted to test how it's actually done. So I got my a self a trial version of F-Secure Server Protection. It's part of their cloud managed "Protection Service for Business", aka. PSB.

I installed that AV product to one of my Backup Repository servers, that also acts as a mount server.

To configure Veeam integration, I need to modify AntivirusInfos.xml file, that is located under "%ProgramFiles%\Common Files\Veeam\Backup and Replication\Mount Service" folder.

After reading documentation, I knew that I need at least path to AV scanner, required command line parameter(s), exit codes and a string that is shown when infection is found.

First, I started to figure out that how command line scanning works. I found out, that there is a fsscan.exe, located in "C:\Program Files (x86)\F-Secure\PSB\" directory. When you run it, it show's you command line parameters.


With this, I knew that I need to use --target parameter, to specify what to scan.

For exit codes, I found this page: https://community.f-secure.com/t5/Business-Suite/On-demand-scanner-fsav-exit/ta-p/20254 that shows exit codes for fsav.exe, but I did not find anything for fsscan.exe. But since fsav.exe is old name for fsscan.exe, I decided to test if those exit codes still work.

I then downloaded EICAR test file, to test what happens when I manually run command line scan. (It's a bit tricky to download these days, you have to disable all your AV protection to be able to download it).

First I tested scanning with folder that had no infected files in it, to see how it looks.


After that, I copied EICAR test file to couple of folders, among with other clean files and run a test again.

When comparing clean and infected results, I searched for a unique string that I can use to detect when infection is found. That string seems to be "Infections found.", with a dot (".").

So now I had a bare minimum information I needed to add to configuration XML file. Since I wanted to to this with minimum effort, I decided to use "IsPortableSoftware='true'" parameter, so I can leave "RegPath='' ServiceName=''" parameters empty.

I used these parameters:
AntivirusInfo Name='F-Secure Server Protection'
IsPortableSoftware='true'
ExecutableFilePath='C:\Program Files (x86)\F-Secure\PSB\fsscan.exe'
CommandLineParameters='--target %Path%'
RegPath=''
ServiceName=''
ThreatExistsRegEx='Infections\s+found\.'
IsParallelScanAvailable='false'

And also added ExitCodes, which you can see from full code snip.

My addition to XML files looks like this:

 
<AntivirusInfo Name='F-Secure Server Protection' IsPortableSoftware='true' ExecutableFilePath='C:\Program Files (x86)\F-Secure\PSB\fsscan.exe' CommandLineParameters='--target %Path%' RegPath='' ServiceName='' ThreatExistsRegEx='Infections\s+found\.' IsParallelScanAvailable='false'>  
           <ExitCodes>  
                <ExitCode Type='Success' Description='No threats detected'>0</ExitCode>  
                <ExitCode Type='Error' Description='Fatal error; unrecoverable error.'>1</ExitCode>  
                <ExitCode Type='Infected' Description='Virus threat was detected'>3</ExitCode>  
                <ExitCode Type='Infected' Description='Riskware (potential spyware) found'>4</ExitCode>  
                <ExitCode Type='Warning' Description='Suspicious files found; these are not necessarily infected by a virus'>8</ExitCode>  
                <ExitCode Type='Error' Description='Scan error, at least one file scan failed'>9</ExitCode>                 
           </ExitCodes>  
      </AntivirusInfo>  


After adding those lines to my AntivirusInfos.xml, I started doing some tests.

I created some dummy virtual machines, one with EICAR test file, one with "Potentially Unwanted Application" test file from AMTSO, and one clean machine. All those machines are dummy machines with no operating system, there is only a 4GB disk in each of those machines with some random files in there.

To test, I started full VM recovery for all of those three test machines. Only thing I need to do while restoring, is to check Secure Restore option:


I set it to abort restoring, if infection is found. With both F-Secure and Microsoft Defender, they do not report you about found infections during AV scan with a string that could be parsed, so it always does a full scan, so "Scan entire image" check box does not have effect with those two AV scanners.

And how does it look like? With a clean machine, it looks like this:



Everything is nice and green, and restore was successful.

But with infected machines, it looks different. Both EICAR and "Potentially Unwanted Application" are detected:





I was expecting that with "Potentially Unwanted Application", F-Secure would have used Exit Code 4, but it seems to detect it as an infection instead of riskware.

I also tested those same machines with Microsoft Defender. I expected to get same results, but I did not! With clean machine, and machine with EICAR test file, results were the same, but with "Potentially Unwanted Application", Microsoft Defender did not detect anything harmful! I'm not sure if this is some setting error in my servers MS Defender or not, but quite a worrying anyways. To be sure, I actually tested it with two different servers, and with same results.




As a last thing, I wanted to see if there is any performance difference between F-Secure Server Protection and Windows Defender.

So I restored a real Windows 10 machine, that has some EICAR files in it.

With F-Secure Server Protection, it lasted about 10 minutes:


And with Windows Defender, it lasted a bit over 20 minutes:



So, actually quite a big difference.

As you can see, it's really easy to integrate you AV scanner to be used with Secure Restore functionality in Veeam Backup & Replication!

Saturday, August 31, 2019

Connecting vRealize Orchestrator 7.6 to vCenter

Do note, this is not a "best practice" guide, more a documentation on how I did configuration in my home lab

Most likely the first thing that you want to do with your Orchestrator, is to add your vCenter instance to your Orchestrator, so you can run workflows against it.

To do that, let's log in to your Orchestrator HTML5 Client.




Go to workflows. We need to search the right workflow for us. Use two keywords, "add" and "vcenter", and we should be able to find a correct workflow

Click "RUN" on "Add a vCenter Server instance" workflow to start it.

Give your vCenter IP or FQDN to form. You might also want to check "Do you want to ignore certificate warnings? If you select Yes, the vCenter Server instance certificate is accepted silently and the certificate is added to the trusted store", at least if you are using self-signed certificates.


Go to "Set the connection properties" tab, and give username and password that are used to connect to vCenter. 


Click "Run", and wait for workflow to run.


If everything goes as expected, you should see that status is "completed".

You can now go to Administration -> Inventory, and you should be able to see your vCenter inventory from here:


We have now succesfully added our vCenter instance to Orchestrator!



Tuesday, August 27, 2019

Installing vRealize Orchestrator 7.6

VMware vRealize Orchestrator has changed quite a lot, and it now has a brand new HTML5 client, which is the default one. You can still use legacy client, but there can be some compatibility issues between workflows done with new vs. legacy client.

Do note, this is not a "best practice" guide, more a documentation on how I did installation in my home lab

Orchestrator comes as an ova file, so deploying it is quite straight forward, and I'm not going to through that process.

But, after you have done the initial deployment, this is what I did.

I first logged in to https://<appliance ip>:5480, to check if there are any updates available. Well, there was none at that time.

Next I went to Control Center. You can find it by going to front page of your appliance, https://<appliance ip>:8281/vco/ (or by going to http://<appliance ip>, which will redirect you there) and selecting "Start the Control Center"




It will ask you for you root / pass, that you provided during deployment of .ova

First we configure a host name:



Then authentication. Since I do not have vRealize Automation, I did choose vSphere as authentication mode, and gave it my vCenter address:


You need to click "accept certificate" first.

Give account that you use to connect to vCenter:


 After that, you need to select Admin group that has access to vRO, and you are done.


Now you should see vRO Control Center



If you want, you can test that authentication works, by going "Configure Authentication Provider -> Test Login"


Now you should be able to login to your Orchestrator Client, go to front page of your appliance, and select "Start the orchestrator client"

For the first time, it gave me this:

But after logging out of my vCenter, I was able to log in succesfully. It should give you your vCenter login page:


And after giving credentials,we are logged in to our brand new HTML5 based orchestrator client!







Wednesday, August 21, 2019

My HomeLab setup

On end of last year, I started to slowly build up up a home lab to be able to test all kinds of things, and to study more some of our own applications.

And don't get me wrong here, we have a great demo environments, but they are something that you are not supposed to break down. And sometimes, to really understand that how things work, you actually need to break them.

So I decided that I finally need to build up my own home lab environment.

And here is my current physical setup:


VMware (all in version 6.7):

  •  Virtual vCenter Appliance
  • "Production" Cluster:
  • 1x Omen by HP laptop, i7-8750H @ 2.20GHz, 16GB RAM. This one I occasionally boot up from USB-disk containing ESX installation, if I need temporarily more compute power, or to test some DR scenarios etc.


Hyper-V, standalone host, Win 2016 Core

  • ASUS UX303L laptop, Intel® Core™ i7 4510U @ 2GHz, 8GB. This is my old laptop with broken display. Only sad thing with this is that it wont boot if it's not attached to some display while booting, have not figured out how to bypass that. This one uses local SSD in laptop, so no connection to shared storage.

Network:

  • HP OfficeConnect Switch 1820-24G. This is reasonably priced (web)managed switch, and at least in my home lab 1 Gbps connections are fast enough. Still for my Storage I did a 2 Gbps trunk.
Storage:
  • QNAP TS879 Pro, equipped with 8x 3TB disks. And configured to RAID-10. So more than enough of capacity and performance for my needs. This is my main storage for the VMware environment. Disks are both iSCSI and NFS (Why both? Well, because I can). Connected to my switch with 2x links, so I have 2 Gbps connection for storage.
  • Buffalo LinkStation LS220D. This is my secondary storage for homelab and primary storage for my personal files. And I use it as a secondary backup target for my homelab, and use my QNAP as a primary backup target for files in this one.
Firewall: 
  • pfSense, running in virtual machine. Only one node, but since I now have to hosts, I'm going to make this one HA

So, what do I do with this environment then, and what do I have running in there?

Well, besides the basic infra services that you just have to have (DNS, AD etc) I have installed for example:
I have also been playing around with Minio lately. (Hint, it has a lot to do with Backup & Replication)

Best here is, that I can pretty much test anything I want to, and if (read: when) I mess up with something, it only affects me, not anyone else.

And, hopefully this will help me to write a lot more blog posts.


Sunday, August 11, 2019

Your blog will reboot in 3..2..1..NOW!



Wow, my blog has been silent for a bit over three years now. 

I just lost my motivation (and time) to write anything.

But I'm trying to start it all over again, let's see what happens. I already have some ideas on what to write about. 

So, now with a new name, and also with a new Instagram account (since I lost my access to old one, sigh....)

Feel free to start following, I will do my best to make this blog active again.