Office 365 backups

We have until now carried out on-site backups of our Office 365 mail, calendar and contacts using CodeTwo Backup, which I agree is a bit perverse.  The space we are using on our trust Nimble CS220 is starting to be of concern, so I'm giving in and looking at cloud backup solutions.

There are lots of these several of which I've trialed before but am looking at again:

1. CloudAlly - http://www.cloudally.com/
2. Dell EMC Spanning - http://spanning.com/ 
3. SkyKick - https://www.skykick.com/ (but sold through partners, not direct I think)
4. CloudFinder - http://cloudfinder.com/office-365-backup/
5. Datto Backupify - http://www.datto.com/backupify
6. UpSafe - http://www.upsafe.com/

My high-level requirements are:

1. At least daily backups of Mail, Calendar and Contacts
2. Quick search to find any of the above.  In particular, ability to find Calendar entries by date of event (not date it was last modified, which is no use at all - do you hear me CodeTwo?)
3. Ability to restore individual mails, calendar items and contact to the original mailbox (and less importantly, to another mailbox)
4. Storage located in the EU (UK ideally, given potential Brexit, but EU will do)
5. Unlimited storage per user
6. A modest cost per user (no more than £3 per user per month)
7. Reasonable and credible assurances around data security (ideally as part of the terms and conditions)

I am in contact and/or trialing the systems above, and will update this post shortly to detail the differences, costs and my overall rating.

CloudAlly

Pros

• Easy set up.  I had backups running in less than 5 minutes.
• Well priced - $30 per user per year is very competitive.
• Covers lots of services in one product.  Sharepoint Online; Google Apps; Box; OneDrive and more.
• Allows items to be restored from a specific snapshot (backup run) so the backup set isn't one big dumping ground.  This is useful for those "It was there on the 5th April" restore requests.
• Backups can be run on demand as well as on a daily schedule.

Cons

• Can't search calendar entries by date of event - this is a common problem, and a real issue for me.  Someone asks me "I had a meeting in my calendar for this morning but I can't recall what it was, can you look in the backup as it has gone" - in this case unless they can recall when they modified it it can be very hard to track down.
• The UI isn't to my liking.  It isn't clear what is going on.  For example, I manually started a backup and the user now shows the total for his mailbox accurately in the user list.  However, the total for amount backed up for the whole of my Office 365 still says 0.  It might be that the backup is still running, but I can't really tell.  Also, you can list accounts that have backups, but from that list you can't get to restores.  To do that you have to navigate to the restore section and find the user there.  As I say, not my favorite UI.
• Not a lot in the way of reporting, no clear way to see if anything is going wrong.

Spanning

Pros

• Very fast set up
• Allows items to be looked at by date of backup.
• Provides a summary of backup issues (though there are often many during initial backup due to MS throttling, etc.)
• Nice UI.  Easy to see what is backed up for whom and how much.
• Well regarded by my peers.  Relatively big player in the market with huge backing (Dell/EMC)
• EU datacenter option

Cons

• Doesn't backup contacts yet - Spanning don't seem able to tell me when it will (but insist it will eventually)
• At the high end in terms of cost, even after haggling.
• Calendar entries cannot be searched for by event date

SkyKick

Pros

• Nice UI with reasonable reporting.
• Sold through channel so easy invoicing.

Cons

• Can't restore individual calendar items!
• Search is hard to use (IMO) and doesn't offer any advanced filters, just keywords.
• No way to kick off a backup of one mailbox manually

CloudFinder

Pros

• Very fast to set up
• Nice, useful dashboard showing current backup status
• Simple but useful reporting
• Seems quite fast to backup
• Search is fast and seems accurate

Cons

• No way to search for calendar events by date of event - only by date of backup (which is of very little use to me)

Backupify

Pros

• Big player in the market.
• UI is relatively clear and functional
• Backs up three times a day and allows manually initiated backups
• *Can* search for calendar events based on date of the event
• Sold through the channel so easy invoicing.

Cons

• Considers the backup as a 'single dump' of all data per person, so there's no way to look at the data by backup date.
• No easy to access trial
• Sold through the channel, so a lot slower to get info on than the direct, self-service alternatives

AD Domain Administrator password and name change

Changing the AD domain administrator username is good practice, as is a regular change to the password.  Our password had remained the same for a good long time due to fear of breaking things (and an apparently inaccurate belief that one system in particular would break).

In practice it isn't so bad:

1. Get Service Credentials Manager from http://www.cjwdev.co.uk/Software/ServiceCredMan/Info.html - very handy tools to trawl the network getting the details of all services and scheduled tasks along with their login account details
2. Run the above - if your network is small the free version of the tool is sufficient.  Larger networks would benefit from the ability to export the output to Excel, so the paid version is a better option.  What comes out of this will help you plan the change - if nothing or little runs as the domain administrator then the process is generally quick and smooth.
3. In Active Directory Users and Computers, right click on the administrator account and click on Rename - enter the new name.  Log our and back in as this user, if you are logged in as it (though you should be using individual admin accounts that are separate from your day-to-day logins).
4. Again in ADUC right click on the administrator account and select Reset Password.  At this point the change is done.
5. Wait for AD to propagate the change to other DCs (if you have any), or force it with 'repadmin /syncall <other DC name> /Adep'
6. Now go to any system with services or scheduled tasks that run as the administrator account.  Ideally, create new account for these (local ones if possible, or domain ones with no or restricted logon rights) and use one account per service 'group' (so one for the Veeam services, for example).  Restart said services and make sure things all still work.
7. Also look out for applications which cache credentials in their config.  Backup Exec, for example, or Veeam ONE - again, these shouldn't be the domain administrator account, but if they are they will need updating (or ideally replacing with dedicated accounts).

Note that using dedicated accounts for services might require that they have local admin rights.  In our case we had a couple of services on the Domain Controllers (not ideal, but needs must..) so adding the account to the local administrators group wasn't possible via the GUI.  To do this from the command line :

net localgroup Administrators /add DOMAIN\USERNAME

Also a lot quicker than doing it via the GUI on the non-DC systems.

Cisco Meraki Wifi with 802.1x, Microsoft NPS RADIUS and Windows 7

Most of the set up for 802.1x authentication of Wifi clients for Meraki APs using Microsoft NPS RADIUS is pretty straight forward.  There are numerous guides on the web such as Meraki's own:

https://documentation.meraki.com/MR/Encryption_and_Authentication/Configuring_RADIUS_Authentication_with_WPA2-Enterprise

One thing that isn't well documented is how to make Windows 7 clients actually work.  The problem is actually not Meraki related, but due to the RADIUS server not having enrolled a certificate for RAS/IAS use in the domain certificate store and is only seen when you have an AD CA configured. The RADIUS server delivers a certificate which Windows 7 will reject *even if it has been told to ignore server certificate validity* in the SSID config.

The solution is to open the Certificate Template MMC snap-in on the AD CA server system, make a copy of the RAS/IAS template and make sure enrolment and auto-enrolment is enable in it's permissions for 'RAS and IIS Servers'.  This will result in the RADIUS server enrolling a new certificate which will be used when RADIUS clients connect - this certificate is properly formed and will work for Windows 7.  You can check that this has worked by looking in the Issued Certificates list in the CA MMC plug-in on the AD CA Server - there should be a certificate for the RADIUS server using the template you created above.

Once you do this Windows 7 should start to connect successfully.

This week's niggles

• Outlook 2016 not previewing Office files.  Very annoying but easily fixed with a registry change http://www.outlook-tips.net/tips/outlook-2016excel-files-wont-preview/
  
• Bluetooth problems in Windows 10 - hard to tell which particular thing fixed it, but my fully up to date Windows 10 with the 1511 update seems rock solid in this respect now.  I will test the Wifi stability issues over the weekend at home.

Embracing the Cloud - Cisco Meraki and VMWare's stingy Essentials license

I hate the term 'Cloud' as it refers broadly to an approach that has been around for a long time, so it really is a buzzword and nothing more, but I'll concede that it is simpler to just use the term in the interest of brevity...

In considering replacement network hardware for our cheapish-and-cheerfulish Netgear kit we're faced with a few core options.  HP, Dell, Cisco Meraki are in the mix, as are some other less well known players, but for a single IT staff resourced company Meraki is so tempting as to be worth taking the risks associated with a cloud managed system.  I have a Meraki AP and an 8 port Meraki switch that I got for free by attending some webinars (cunning strategy, IMO) and they really are so very easy to provision and manage and look good too (not that it matters when they will be mostly hidden away).

There are lots of nay-sayers around using Cloud for one thing or another, and the privacy and security oriented part of me (with it's roots in my early days as a *nix sysadmin) agrees with them all.  However, we have taken the plunge for many management and commercial reasons, not least of which being the trend in our industry and our lean approach to IT staffing.  So the fact that Meraki is cloud managed is OK with me, in this context.  Horror stories about delayed renewals taking networks down merely re-enforce the need to manage one's contracts properly (and are increasingly older stories, so I suspect it is more well managed in general now).

Having said that, we are not a huge operation IT-wise, and our environment on-premise is mostly virtual.  This puts us in the VMWare Essentials Plus license band, which suits us almost perfectly, being affordable (though not cheap) and allowing us a HA cluster at our larger site and a single server at the smaller site (so resilience between sites at the application level and at one site at the infrastructure level).

I have always used LAGs to mitigate against bandwidth problems for the VMWare physical connections, though of course the ability for a LAG to mitigate the problem is a lot more limited than people often think.  The added resilience is handy too.  On our NetGear switches we just set up a LAG and ignore LACP and it just works.  On the Meraki switch we are told (by Meraki) that this will not work - their documented method is to enable LACP and use a VMWare distributed switch, which we can't do as the Essentials Plus license doesn't include that feature.  Upgrading to a full vSphere license is pretty costly (looks like somewhere around the £5-6K mark, with an associated annual maintenance increase - a lot for one tiny feature IMO).

Meraki theorise that doing the same as we do now, but turning RSTP off on the LAG will work.  They warn about the chance of loops, but from my understanding of vSwitches spanning tree is not necessary as the vSwitch prevents loops itself.

Of course, I can't commit to the new hardware without being sure that this will work, so it's time to vmotion all the guests onto one of the two cluster nodes and move the freed up host to my 8 port meraki switch for some testing (I could dredge up enough hardware to build a test system, just about, but I would feel better using my live host and it would be a lot quicker - 1 man IT dept, remember...)

Mind you, I do have an apprentice starting soon, so if I don't get to this before then they could learn a lot by building out a VMWare host....  Hmm.

Whichever I decide I'll post the results of the test here as I cannot find any posts on the Internet saying whether this works (which makes me wonder if many people use Essentials Plus)

UPDATE: I did the testing with the production ESXi host and couldn't persuade the Meraki and the server to work reliably with an aggregated link - ports would shutdown (presumably because the Meraki was trying to use LACP and VMware wasn't) and some VLANs worked but others didn't.  What did work was to use "Route based on the originating virtual port" - sadly the Essentials license doesn't include the variant of this that takes load into account, so there's no clever load balancing, but there is some arbitrary spread and at least the links are redundant.  The nice thing about this is it requires no configuration on the switch other than making the ports into trunks.

Domino 9.0.1FP4 vs Backups Exec 2014, round 2

After resisting for a long time I gave up and removed Backup Exec from our production Domino server and set up a separate server just for backups.  I didn't want to purchase another Windows license, and whilst I love linux I didn't want to wrestle with IBM's Linux requirements and their Domino install process (none of which is that bad, but there's only one of me here).

So I popped some extra disk space into my network monitor machine, which is recently new and has lots of spare resources and stuck Domino 9.0 on there, which is fully supported by BE.

1. I now have a pull-only Domino server, with a 30 min schedule.  This gives me a 30 minute 'quick recovery' window is someone screws up (well, kinda)
2. The production Domino server no longer gets constant errors reported in the DDM - looks like all the fixups and compact clashes were down to BE
3. No more disk errors, no more database corruption
4. Backups no longer get exceptions and run a bit faster too

So I wish I had done this earlier.

The network monitor is another post waiting to be written.  PRTG is a lovely NMS...

HP Intelligent Provisioning nightmares

Much like https://supertekboy.com/2013/06/03/hp-intelligent-provisoning-firmware-update-failed/#.VebpBPZVjX8 I am once again unable to get the HP Intelligent Provisioning to apply updates to a brand new HP sever (just a little DL60 this time).

It escapes me just how HP can ship hardware with this kind of problem.  I just want my SmartUpdate DVD back, at least it worked!

Tried static IP addresses, DHCP, separate network interface, different port on the back, nothing works.

Sigh.

[UPDATE: giving up and downloading the HP SP DVD]

[UPDATE: amusingly whilst trying to login to the HP Support site, which doesn't seem to want to work, the server started updating. I had to use the 2nd NIC and specifically go into the provisioning set up to change to that NIC even though it was already selected]