Contact Us: 1-800-634-1990

|

Totango Blog

Stay in the latest updates.

Get the latest tips and advice delivered straight to your inbox.

Public Service Announcement: Totango Critical Functions Temporarily Disabled

Edit: 8/23/13 3:19pm PT

Post Mortem Analysis

What happened?
We had database hiccups over the weekend prior to a major sub-system upgrade. This resulted in inaccurate data reporting of several sub-systems that influenced critical system functionalities.

In more detail:  We were planning for a major upgrade of our indexing infrastructure to support the current and future loads of data and also to lay down the groundwork for functionality planned for the next few weeks. We had been testing the indexing infrastructure for several weeks without any problem.

On Friday, our databases experienced severe overloads which reflected in the resulting hiccups. The upgrade which took place on Sunday only amplified the effect of the database overload.

The impact of technical issues were immediately identified, however, it took us time to build a fix procedure, implement and deploy. Users also started experiencing inaccurate information on Monday morning (PST).

Lessons Learned

  1. We are tightening up even further the monitoring thresholds in general and we’re implementing close immediate monitoring for future upgrades.

  2. No data was lost. However, a procedure for history recovery is being pushed forward to fill in any missing data points

  3. We are committed to maintaining an open and transparent dialogue with our users (even though there were some voices of concern). Our users greatly appreciated the gesture and we plan to continue to do so should another event like this arise.

Again, we apologize for any inconvenience caused by this maintenance and thank you for your patience and understanding.

#####Original Post#####

Dear Totango user,

Several critical Totango functions are currently disabled due to a longer than expect infrastructure deployment. There’s no data loss.

What’s going on?

Over the weekend, we conducted a major upgrade to the underlying infrastructure of our system. We expected it to take some time to crunch the numbers to be ready in time for the beginning of the week, but as of this moment, it is still processing.

How will this affect me?

As a result active lists are returning inaccurate results and therefore triggers and tags are inaccurate as well. As a preemptive measure, we’ve temporarily disabled triggers and Salesforce sync to preserve the integrity of your data. We are looking for this to be resolved in the next few hours. We will provide a full report and analysis of the chain of events after this is resolved.

We apologize for any inconvenience caused by this unexpected delay in the upgrade. If you have any questions, please do not hesitate to email us.

Update (12:15pm PT): Still working on resolving this issue, estimated to be completed by or before end of today.

Update (12:46pm PT): Deploying fix.

Update (2:29pm PT): Numbers are still crunching!

Update (2:57pm PT): 50% of accounts back to normal…stay tuned!

Update (1:07am PT): Your critical Totango functions should be back to normal. We appreciate your patience while we got everything back up to speed. If you experience any issues, please let us know at support@totango.com.

Post mortem analysis will be published soon.

Guy Nirpaz

Guy Nirpaz is a Silicon Valley-based Israeli entrepreneur and CEO of Totango, a Customer Success software platform. A pioneer in the Customer Success field, Guy established the Customer Success Summit and is a well-regarded industry speaker and community contributor. Guy loves people and technology and has dedicated his career to improving the way in which business is done through innovation. Fun Facts: Guy moonlights as the lead guitarist in a rock band based out of his garage in Palo Alto and used to command a tank battalion...as well as having grown oranges.

You might also like
You might also like