The Cloud isn't everything

So what do you do when the cloud document collaboration system you rely on stops allowing logins on a Sunday and support only communicate via email ?  On a Sunday when your users need access...

Email them repeatedly and hope they either fix is as a matter of course or actually respond to the emails and look into the problem.

Luckily I keep an onsite backup so at least we can get to the files as a last resort.

This kind of thing is never mentioned when people are pushing 'cloud' solutions. I start from an assumption that things will go wrong hence the onsite backup but many people just drink the koolaid and suffer the consequences.

So much for being on holiday!

[UPDATE: turns out that they had been making a change for another customer on Friday night and they accidentally changed our SAML configuration data, breaking our account.  It took a lot of back and forth to get this checked.]

IBM Domino 9.0.1FP4 and Backup Exec 2014

And here's why you need to keep some kind of configuration/change log, even if you only ever refer to it once a year and there's only one of you in the IT team...

Whilst we don't use IBM (ne Lotus) Domino quite as much as we used to (now we live in the brave new cloud world of Office 365 for email...) we do have several key systems that rely on it.

Now that IBM have finally patched the various SSL and related vulnerabilities it pays to keep the servers on pretty recent versions, especially in an environment like ours which are small enough to be manageable by one person who knows reasonably well how it all hangs together (ehem, that'll be me I guess).

So after letting Domino 9.0.1 FP4 bed in at other IBM customers for a little while I decided to get us updated, particularly as we are seeing some odd issues with database files on one of the servers.

Of course it didn't (but should have) cross my mind that IBM might do something that breaks Backup Exec's backup agent.  It really should have.  Backup Exec is a ropey product at the best of times, and when you put that together with the increasingly niche (but still quite common) Domino you're asking for trouble.

So I updates and dutifully logged a help desk ticket, tagged as a Domino ticket and marked as a 'change'.

My backups then began to fail.  Part of the reason for that was the fact that I had disabled the Backup Exec agent service to allow the FP install and then forgotten to turn it back on again.  I turned it back on, and now the backups hang on any attempt to access Domino databases.

Turns out IBM changed something and the Backup Exec agent can't get to the databases any more.

So I compared the backup logs with the 'change log' (er, spiceworks help desk with a custom field to indicate that a ticket was actually a change - it kinda works).  And the two things correlated.

Before I made this connection symantec support were kind enough to spend the day on the phone trying to get it to work before I finally pointed out that perhaps it was a Domino version issue.  Only version 9.0 of Domino is officially supported.

So I went back to 9.0.1 (no patches) and lo and behold it works again.

So surely I can get back up to FP3, which was, after all, working (though my suspicions were roused about the database corruption I had been seeing...).  Er, no.  Back up to FP3 and it fails again.  So back to 9.0.1 we go and it's working again.

Maybe I can sneak up to FP2, which for some reason requires that I pause the Windows Management Instrumentation service (thank you Google) to install.  But no, so back to 9.0.1...

This took most of the day.  Now I can try to cram the day's work into the last 45 mins before I have to leave ;-)

2 years, four months post treatment

Well I'm still here....

Unless the worst happens I'm switching the blog from health matters to IT matters, so if you subscribe feel free to un-subscribe unless you want to hear my random unedited musing...