My Agile Failures - Deliver Software Frequently without Enough Safenet

"Deliver software Frequently" becomes symbol being agile. As my previous post, I am talking about that delivery software to production is not finish line because feedback loop has not finished just with that. This post I will talk about deliver software frequently without building enough safe-net or not enough infrastructure.

Recently, I have been push software from releasing every week to several times a week. But in the same time, there was an incident happened. Of course, there are many reason behind that. There are always many reason behind an incident. But the most important lesson that I learn from that is safe-net for delivering quickly.

The basic assumption that a team should have is that if there is a failure, what we can do. How can we achieve rollback without customer notice or with minimal impact. "Fail more often" doesn't mean that you can fail more often. But it is a mindset that

. We gona to fail and we should have a way and capacity to quickly recover
. We should learn from our failure

Failure doesn't necessary a system failure, it can be that metrics shows that business hypothesis failed.

And I personally found "Post Mortem" is a good way keep us learning from failure in real time. Dont let similar bad things happen more than three times.

What is your failure experience with delivering software more often?

Separate Deployment and Release for Web Applications

Background

Most organization has treated deployment as release. Of course, there are many advantages doing that. Like, it is easier to maintain and manage. And it will be very efficient for small team when one feature delivery requires time. So when new version get deployed into server, everyone will see the changes.

But when team grows, it becomes biggest problem if we still want to delivery features to customers as early as possible. There are few problems as my discovery:

1. Feature deployed into server, but product team and sales team are not ready for end users to use yet.
2. Some features will only bring value into certain group of customers and not bring any value to other group of customers
3.  Some features are designed for working with certain user group but not for others yet. Release into everyone will cause
4. Frequent change of user inteface are note acceptable to most customer although it will be liked by most of customers

What we do

Separate Release from Deployment

First way that I can thought of will be separate release from deployment. There are few pratices out there:

1. Feature Flipping/Toggling
2. Early Access Program
3. Any other you are using? 

Do you have any ideas how to make it better? There must be some very good idea out there. Please let me know so that I can compare features.

Feature Flipping/Toggling

With feature flipping/toggling, you will single version of software but be able to allow certain user group to see new features like evanglist, trial users, users from certain region or country using GeoIP, users with certain behaviors.

It is easy to manage source code. But it is very hard to keep all software engineers to use feature flipping properly. If not, the code grows into a big mess. 

Early Access Program

There will be different version of software installed on server. Some users can be configured to access to newest version/some version of software.  It is hard to manage bugs found or some urgent feature needs to be in production.

The advantage of this is that you don't need to worry too much problems may be caused by feature flipping (architecture design and clean up of flags)

Why we will separate release from deployment

With separating release from deployment, we can

• Minimize impact while doing deployment 
• Minimize risk for deployment and release
• Support continuous deployment without degrade too much of user experience because of frequent change
• Customer can get feature as early as possible
• A very good communication channel between customers and engineers
• We still deliver much better product to most end user after some iterations with user feedback

Side Note

In both solution that I have found,  my feel is that most important part is to keep communication channel wide open between engineers and customers. The communication channel should be easily accessible for both engineers and customers.  Only with that, your team can be Agile.

Why you won't do continuous deployment

Introduction

Continue deployment is very popular around world at the moment. It has been seen as one corner stone of agile practices. But I do believe it takes very long time for organization to be able deploy often. 

 

My view about continue deployment is that you should NOT do continuous deployment unless you have a safe net. 

Safe net means differently from business to business. But there are several things that are essential across different business types. 

Monitoring

Monitoring becomes more and more important for IT businesses. Without any data supporting your theory, you may make wrong decision. The decision may cost your company as highly competition from anywhere around the world with low entry requirement for an IT business. 

There are tons of article talking about monitoring. But I do believe that every business should encourage monitoring across all layers:

• Operation/System Services including databases
• Application Layer error
• User Interactions

Everything has its important in helping us making right decision and fix problems as quick as possible. I believe that nothing can be 100% right unless we put them in production. With fast growing amount of data, it is not possible for teams to test most systems with scale which is same as production. Expecting fail and have a safe net to catch failure as early as possible and design a system to make problem to be fixed as soon as possible. 

If you don't have monitoring in place, do NOT do continuous deployment. 

Build Pipeline

Environments always should be first problem you solve if you want team to be agile. I have been experience in lots of companies that they are experience as following order:

1. Use product as development sandbox

2. Have staging environment and staging environment db box is used for development

3. Have continuous build and automatic acceptance testing environments

4. Then have dedicated dev, testing and staging environment

5. Then have dedicated preview sites

6. Have gateway function to select users for preview features (Not done in any company yet, only read in paper now)

Without achieving (4),  there are not enough testing before going to production. In that case, please do NOT use continuous deployment unless your system in VERY small. If you have done continuous deployment without reach (4), please let me know.I would love to know that how can you minimize risk while not having such environments. 

Testing and Quality Control

Unit testing, functional testing/integration testing and acceptance tests are essential for your continuous deployment. 

Some people out there are fans of unit testing or functional testing. I don't believe one type of test is enough for continuous deployment.  

Functional testing and Acceptance Tests will tell us whether system did what we expect. As every software engineer knows, there are many cases that you can't put into functional test cases. It tests from user point of view. But it never tests from software engineer point of view. There are lots of place that we can miss with functional testing. I have see that some one promote functional testing and they promote technical stories in the same time. It is kind of making sense because they want to cover in more granular level. 

Unit testing is essential as most agree. It give us confident in a granular level so that we will be very confident as software engineer. And another advantage is that we have static about how much coverage of testing is at the moment. It is very important thing for me. Someone says 80% is ideal line that you can have enough confidence. I would say sometimes 70% maybe enough too. And progressively working towards 80% is best way to do. But you will find that after 65%, it normally becomes very hard to do. And it is worth to write test coverage as requirement in stories from that time.  

With unit testing only, we can assure that system does what user or product team wants. And we haven't tested in an integrated environment yet. With those reasons, functional and acceptance tests are very important too.

Feature Flipping

I have not done any feature flipping in any organization yet. But I believe that feature flipping is important for continuous deployment. There are frequent requests about hide certain features for different kinds of reasons. Normally reaction at the moment, we just delay deployments. But it will prevent other important features to be released too. When team grows, the problems will occur more and more. And I am pushing feature flipping at the moment. But it may result in messy code and it needs frequent clean up are always concerns.