Less Annoying CRM Apology Letter – or – How it’s done

Monday, August 24th, 2015

I’ve been a LessAnnoyingCRM user for about a year now and I love it. (NOTE: If you use this Less Annoying CRM affiliate link you’ll get 60 days free use instead of the normal 30)

Recently they had some issues on their site, and the level of transparency was amazing.

If you ever have to issue an apology letter to your customers, you can’t go wrong using this as a template

 

Less Annoying CRM server issues – apology and post-mortem

from: Tyler King <tyler@lessannoyingcrm.com>
to: “malcolm.b.anderson@pragmaticagility.com” <malcolm.b.anderson@pragmaticagility.com>
date: Mon, Aug 17, 2015 at 8:08 AM

You may have noticed that the Less Annoying CRM website had some problems last Friday and over the weekend. We take great pride in the speed and reliability of our site, and it’s a huge embarrassment when we let you down like this. I’m writing to apologize, and to explain what happened.

Let me start by saying that I’m sorry. The entire LACRM team is sorry. This has been a stressful couple of days for us as we worked to fix the server issues, yes, but I know that’s not even close to what our customers experienced when they weren’t able to access their crucial data when they needed it.

So what happened? First, in case you’re worried, let me assure you that no data was lost (other than if you tried to save anything when the site was down, including logged emails). The site is running smoothly and we have no indication that any of the previous problems are still present, so performance should be back to normal.

If you’re the type of person who is interested in the nitty gritty, I’ve included both a short and a long description of the issues, as well as what actions we took to solve them, below. Please let me know if you have any questions. Once again, please accept my deepest personal apology.

Thanks,
Tyler
CEO of Less Annoying CRM

—The short story—
Friday morning (US time), the site went down due to a database failure. No data was lost, but it took about one hour for us to bring up our backup server, and the site was inaccessible during that time. When the site came back up, it was clear that our failover database still wasn’t behaving correctly, and there were performance issues for the rest of the day. We believed that we had fixed the issue Friday night, but unfortunately it occurred again on Saturday morning, so the site was slow most of the day on Saturday. The fix we attempted Saturday night worked temporarily, but the problem was back Sunday morning. On Sunday we decided to take more drastic actions which required us to put the site into “read only” mode for the day while we built an entirely new database from scratch. The new database went live Sunday night, and we haven’t had any problems since then. We’ll continue to monitor the situation, and we’ve already taken actions to strengthen our infrastructure for the future.

—The long (more technical) story—
Our databases crashed Friday morning (US time) for unknown reasons. We keep a failover database ready at all times so that it’s easy to recover from this type of scenario. Switching to the failover database is supposed to be nearly instant, but (again, for unknown reasons) it took much longer than it typically does. We suspect that something was wrong with the hardware in question. Because of this extra delay, it took us about one hour from the start of the downtime before the site was live again.

The failover complete, the site continued to have intermittent database problems throughout Friday. The site was still generally accessible and there was no more extended downtime, but we noticed continued issues with slowness, and occasional brief outages that quickly self-corrected. After looking into the issue, we believed that the problem was caused by a problem with the database server related to the same issue that caused it to take so long to fail over initially. Friday night we switched to an entirely new database server, and the problems subsided.

Unfortunately, Saturday morning, the problems returned. This time they followed a pattern. First, the site would experience major slowdowns caused by database issues. After some period of time (normally approximately 30 minutes) the database would calm down and the site would perform normally for about 20-30 minutes, and then the cycle would start over. The site was generally available during this time, but during the bad periods it would run very slowly (10-20+ seconds per page load).

We spent all day on Saturday working on diagnosing the problem. We tested countless possible solutions, and eventually were able to eliminate certain possibilities. For example, we confirmed that the problem wasn’t caused by user behavior (i.e. a user surge of traffic that we couldn’t handle). We also tried stopping all of our various services to see if that would prevent the database from choking and nothing worked. So that left us with one possibility: the hardware was faulty. Saturday night, we attempted to restart the database using a new virtual machine. After this, the site seemed to be running smoothly until Sunday morning when the problems appeared again.

Leading up to Sunday, we had been trying to find a solution that wouldn’t result in major disruptions to the CRM (the site was still running, it was just slow). Sunday morning we decided to bite the bullet and completely rebuild the database on all new hardware, along with some other changes (such as switching to solid state drives). Unfortunately this required us to put the site into a read-only mode so that no new data could be written while we created the new database. The site was in read-only mode for most of the day on Sunday, until the new database was ready Sunday night. As soon as the new database was ready and tested, we turned off read-only mode and everything was back to normal.

We will continue to monitor things to make sure that there aren’t lingering issues, but the fact that the site is running smoothly under the high load of Monday morning traffic is a great sign. The changes we made on Sunday are already a big step towards preventing this type of thing from happening again in the future, but our technical team will also be making other changes to how the database works in the future to make it even more reliable.

Once again, I’m very sorry about this problem. I know that this explanation doesn’t make up for the inconvenience we put you through, but I always want to be as transparent as possible any time there is a problem. Please let me know if you have any questions.

3 RSpec Book chapter-5 “output” solutions

Monday, September 15th, 2014

I’m doing some volunteer work for an organization as a Ruby Programmer and as such, I’m having to come up to speed on RSpec in a hurry.

 

Well today I got stuck in the middle of chapter 5.  I’m talking “thrashing all over the internet” kind of stuck here.  Sadly my google-foo was failing me until just a moment ago (about 1am on a Saturday night, Sunday morning.)

 

Thankfully Carmen Berros not only had my exact problem, she took the time to write up and link to her friends suggestion.

 

If you are looking to understand what all the big deal is with BDD, even if you are not a ruby programmer, “The RSpec book” (affiliate link) is wonderful, and well worth your time

 

 

Another problem I ended up having with this chapter were my RSpec tests passing, but my Cucumber tests failing.

That solution turned out to be to change the name of “output” to “terminal_output” in

features/step_definitions/codebreaker_steps.rb

 

Lastly, if you haven’t found the solution to the dreaded “should is depreciated” messages, the solution is hinted at here  (this also goes in features/step_definitions/codebreaker_steps.rb)

I’ve included the two lines that I tested

#output.should_receive(:puts).with(‘Welcome to Codebreaker!’)
expect(output).to receive(:puts).with(‘Welcome to Codebreaker!’)

#output.should_receive(:puts).with(‘Enter guess:’)
expect(output).to receive(:puts).with(‘Enter guess:’)

 

If this helps you in your RSpec journey, think about leaving a comment to let us know why you’re taking the time to learn RSpec.

Mobile Apps & TDD – It doesn’t have to be that hard.

Tuesday, September 2nd, 2014

Was having a conversation a couple of days ago and the subject of TDD with Mobile Development came up.
The bottom line was that, “most companies that are doing mobile apps could use TDD training.”

What happens is, they hear that “mobile is different” and then don’t write their code in a way that is testable.

This leaves them with an app who’s internal logic can only be validated by white box testing on as many different devices and operating systems as possible.

There are even people making the case that TDD doesn’t work for mobile because TDD only tests your code’s internal logic.

That’s a punch line.  Testing your code’s internal logic is what Test Driven Design  is intended to provide.

For those of you who would like to dig deeper on why TDD in the mobile space, read Robert van Loghem’s article “Why TDD + Continuous Testing is Imperative for Mobile Apps”

Robert references an old Object Mentor article that is no longer available except through the “Wayback Machine.”  You can read “Continuous Testing Explained” here

 

Can you fill in this line? “If … you might be doing #redneck-agile.”

Wednesday, August 6th, 2014

I just ran across this gem in an agile open group on LinkedIn ‘Discover how to develop “bug free” software projects using Agile based project management tool’ followed by a link to a sales page.

I couldn’t help myself, I started laughing.   I could just imagine it, somewhere, some poor PMP is excited, because finally, they can “buy agile.”

I could also imagine the Jeff Foxworthy of geeks, drawling ‘If your senior managers, keep equating “the right tools” to “agile” … you might be doing redneck agile’ (Except the JFoG in my head was much funnier)

 

Heinlein once said “I’ve found out why people laugh. They laugh because it hurts so much . . . because it’s the only thing that’ll make it stop hurting.”

He also made reference to the coercive effects of humor.  You can influence people’s behavior more by making their behavior a punchline, than you can by all out education.

Just maybe there is some concise wisdom out there that can be used to help people lighten up and understand that making mistakes on the road to agile is a part of being on the road to agile.

So I’m asking for help from the agile community.  There’s a bunch of y’all out there that are funnier than I am.  Can you craft your painful, yet funny experiences into the form of a #redneckagile joke?

 

Thanks

Malcolm

 

 

 

3 Secrets To An Agile Mindset

Wednesday, April 23rd, 2014

Recently I was asked how to position oneself from being a project manager to being a product owner and I came up with the following list

 

* Read the agile manifesto and underlying principles once a day for 30 days and then monthly until you leave the industry (it’s 267 words, it won’t take that long – 3 – 5 minutes each time.)
agilemanifesto.org and http://agilemanifesto.org/principles.html

* Watch Fred Kofman’s talk ” Doing Your Job May Be Hazardous To Your Career”
https://www.youtube.com/watch?v=cgJFLR2f4rY (13:55)

* Take Steve Blank’s “How to build a startup” course (it’s free) (That’s about an hour a week for about 8 weeks)at
https://www.udacity.com/course/ep245

 

In the process though I realized that this list applies to anyone who is doing anything with agile software development, and possibly anything to do with starting their own business.

 

If this post has been useful to you, please comment below and forward it to your friends.

If this was forwarded to you, the original can be found at

pragmaticagility.com/3-secrets-to-an-agile-mindset or at bit.ly/3sec2am

What kind of Scrum Master are you?

Tuesday, November 19th, 2013

I’ve noticed that there seem to be 2 types of Scrum Masters out there, change agent servant leaders and paper pushers with agile titles.

 

Some companies understand that the Scrum Master is a valuable useful position that make a difference.

Reviewing Michael James’s Scrum Master Checklist is a useful, (and humbling) bi-annual practice.

Scrum Master is a full time leadership position.

(By “leadership”, I use the military understanding of the word, meaning, “willing to do what needs to get done in order to fulfill the mission while keeping the big picture in mind.”

This is contrasted by what seems to be the standard civilian misunderstanding, “willing to give orders about this quarters work, while being incapable of providing direct value.”)

These types of Scrum Masters positions tend to be found in companies where the company mission statements are an accurate reflection of the company rules that are whispered in the halls.

Scrum Masters in these positions have the opportunity to make a difference by being change agent servant leaders.

 

 

Other companies understand that they need some magical fairy dust called “agile” or some vague bad thing will happen to the company in the future.

In order to get the magical agile fairy dust, these companies change their project manager’s titles to “Scrum Master,” because it is well known that Scrum Masters produce magical agile fairy dust.

These companies know that a Scrum Master is there to push their teams to show daily progress during the daily status meeting (weirdly called a “standup meeting”)

They also know that the Scrum Masters are there to turn those status meetings into magical burn down charts to be shared at executive briefings.

Lastly, Scrum Masters are there to schedule meetings for the team.

These types of Scrum Master positions tend to be found in companies where the company mission statement bear no resemblance to the company rules that are whispered in the halls.

Scrum Masters in these positions have little opportunity to make a difference because they have been reduced to the role of paper pushers with agile titles.

 

 

Are you allowed making a difference in your company?

 

 

Stay On Mission.

 

If you resonate with this agile slice, please add your voice to the conversation.

If this agile slice sounds silly, please take some actions to test a particular idea’s validity, and then share your results, good or bad, with the community.

If something strikes you as golden, then I’m glad we were able to provide value to your life, please let us know what difference it has made for you or your company.

Why empirical evidence doesn’t matter

Friday, November 15th, 2013

One of the most interesting phenomena I’ve discovered while doing agile coaching is the various knee jerk reactions when paired programming is brought up in the fortune 1000.

 

The reactions are interesting because studies show that paired programming makes a  huge difference.

We’re talking 50 – 85 percent reduction in new bugs, for a cost of 15% up front developer productivity.

Given the cost of finding and fixing bugs, the 15% up front cost is almost free.

 

Up until this week, the lack of interest in (or near violent reactions to) pair programming did not make sense.

 

Then, this week I came across the Semmelweis Reflex.

According to Wikipedia:

The Semmelweis reflex or “Semmelweis effect” is a metaphor for the reflex-like tendency to reject new evidence or new knowledge because it contradicts established norms, beliefs or paradigms.

The term originated from the story of Ignaz Semmelweis, who discovered that childbed fever mortality rates reduced ten-fold when doctors washed their hands with a chlorine solution between patients. His hand-washing suggestions were rejected by his contemporaries, often for non-medical reasons. For instance, some doctors refused to believe that a gentleman’s hands could transmit disease (see Contemporary reaction to Ignaz Semmelweis).

 

All of the sudden, the whole “Paired Programming Reflex” (if you will) started to make sense; empirical evidence doesn’t matter.

Just as a gentleman’s hands can’t transmit disease, having 2 developers working together must destroy half their productivity.

At this point, pairing is no longer just a funny idea.  Pairing is moved from adjective dangerous, it noun danger.

If you listen closely, the next sound to be heard is Fight or Flight reaction being engaged.

The idea, and the bearer of that idea, must be evaded, or destroyed.

 

Stay On Mission

 

If this sounds like a situation that a friend of yours has run into, please forward this to them.

If this explains a situation that you have run into personally, please leave a comment describing your experience.

Three 20 year-olds create a healthcare.gov miracle

Tuesday, November 12th, 2013

Up until the lurching non-roll out of Healthcare.gov, no one outside of the software community had ever heard of “Agile Development.”

The Wall Street JournalThe New Yorker, and Bloomberg have changed that.

Well, that and the miracle in San Francisco.

You see three 20 year-olds, in only 3 days, built a usable front-end to Healthcare.gov.

Miracle?

Yes, 3 people, in 3 days did what 55 independent contracting companies couldn’t do in 3 years with over 500 million lines of code.

There’s a miracle in there somewhere.

If nothing else, if you could get that kind of productivity out of your programmers, you would call it a miracle too.

Over the next couple of days and weeks, there’s going to be a lot of made of these 3, but how many smart business owners are stopping to ask, “How can I get that kind of productivity from my teams?”

In the coming days and weeks, I’ll be exploring that smart question.

Please pass this on to the smart business owners in your life.

Resource: 7 Agile Myths

Monday, April 1st, 2013

Ade Shokoya has a great article called “7 Agile Myths That Could Cost You Your Job & Reputation” which is an excerpt out of his book “Waterfall to Agile: A Practical Guide to Agile Transition.”

 

This short article is a insiders look at 7 common misconceptions that can hinder an agile adoption.

 

I’ve only got a minor nit to pick with his article.  Ade’s first myth is “Agile is a Methodology.”  This is either a myth or a truth depending on how you define methodology.

Like I said, it’s a nit.  His point in here is that Agile is an approach, not a step-by-step method.

 

Ade’s article is worth reading simply for his opening about “What animal is the king of the jungle.”

 

This excerpt from the article is one of my pet peeves, because I’ve seen so many companies slap the agile label on their waterfall process, and then complain about their lack of results.

Myth 4 – Labelling It ‘Agile’ Makes It Agile

As Agile practices grow in popularity, more and more organisations are becoming aware of the benefits Agile has to offer.

But rather than making the necessary organisational, cultural, and environmental changes required to achieve those benefits, many organisations are simply slapping “Agile” related labels on what they’re already doing, thinking that makes them Agile.

And when things don’t turn out as expected, Agile gets the blame.

But just as calling a dog a cat doesn’t make it so, re-labelling traditional software development methods “Agile” does not make them Agile. It takes more than that.

The Entrepreneurial Developer

Friday, March 29th, 2013

Welcome to Agile Slices; a series of quick pragmatic articles of about 200 words, intended to help you, your team and your business, harness the power of agility.

 

The Entrepreneurial Developer

 

When I first started programming, more than 20 years ago, I was hired by a guy to teach him how to program. After 2 weeks he looked at me and said, “You know, I see what you do, and I know I could do what you do. I just don’t want to, how about I hire you to do the programming that I want done?”

 

Over the course of the next two years I became the effective CIO for Athena Software. Athena Software was an award winning shrink wrap company that produced handicapping software for American football and basketball. It’s ironic that I’ve never been a big sports fan.

 

During that time, I was the business analyst, the architect, the database administrator, the programmer, the tester, the release manager and the support team.

 

It was obvious that programming was just one piece of the puzzle. An important piece, but not the be all end all of making a profitable company.

 

Later, when I got out into “the real world” I was surprised to find out that my skill set was not “normal.” Normal turned out to be testers who could’t code, developers who couldn’t test, and database schemas that were … lets call it, sub-optimal.

 

Since that time, I’ve come to describe myself as an entrepreneurial developer and I’ve found many others who share that mindset.

 

Entrepreneurial team members are an asset to your company because they focus on the entire value stream and have a primary interest in making the company more money. Entrepreneurial team members can be difficult. They’re opinionated, they tend not to tow the party line, and tend to ask uncomfortable questions. Worse, not only can they be a pain in the neck, if you’re not careful, they will be the first ones hired away if you don’t pay them enough, or if the work environment becomes unfriendly.

 

For all that, you want to have them on your team because they are self driven, they get stuff done and they have ideas for bringing more money into your company.

 

Stay On Mission.

 

If you have already used this slice, please add your voice to the conversation. If it sounds silly, please take some actions to test a particular idea’s validity, and then share your results, good or bad, with the community. If something strikes you as golden, then I’m glad I was able to provide value to your life, please let us know what difference it has made for you or your company.