I get to the office on Tuesday morning and I see a message from the “manager” of our transaction system about an error she was getting with the software. The error led me to believe that the SSL certificate for the server had expired. I checked the web site for this particular service and, in fact, it had expired the previous afternoon (oops). So I dug up the documentation on setting up a renewed certificate for this software and got to work.
I renewed the certificate through GoDaddy – not our normal certificate provider, but that’s another post altogether – and installed it as described in the provider’s documentation. The web server was happy, but the services for the software would not start. After a little more troubleshooting I found that the certificate was “not yet valid” according to the software. I know the certificate is valid and I know I can’t change the date on the certificate, so the only thing left to do is to call their support.
I had a coworker call support and give them the low down as I was already late for a meeting. The agent double checked all my work and everything looked good. However, he did notice that the “Not Valid Before” date in the certificate was listed in GMT. He then informed us that this system has a known issue that basically ignores the timezone of the certificate and assumes that it is the local time zone of your server. So while the certificate was valid as of 8 am local time, their software would not see it as valid until after 1 pm. ACHIEVEMENT: BUG FOUND! The agent informed us that we would have to wait until after 1 pm for the software to begin working again.
I learned all this after returning from my meeting and obviously, this was not an acceptable answer. We can’t have a production system down half of the day. So I placed another call to support to try to get a resolution. I was connected to a second agent who quickly got up to speed on the situation, got in touch with the first agent, and provided me a “loaner certificate” so we could get the services back online.
I expressed my dissatisfaction with the way the ticket was handled and that their system does not calculate the time zone correctly, but was appreciative that he was able to quickly resolve the issue on the second call.
Lessons Learned:
- Don’t let your certificates expire.
- Renew your certificates and give them some “breathing room” before activating them.
- Don’t take no for an answer.