Why you won't do continuous deployment

Introduction

Continue deployment is very popular around world at the moment. It has been seen as one corner stone of agile practices. But I do believe it takes very long time for organization to be able deploy often. 

 

My view about continue deployment is that you should NOT do continuous deployment unless you have a safe net. 

Safe net means differently from business to business. But there are several things that are essential across different business types. 

Monitoring

Monitoring becomes more and more important for IT businesses. Without any data supporting your theory, you may make wrong decision. The decision may cost your company as highly competition from anywhere around the world with low entry requirement for an IT business. 

There are tons of article talking about monitoring. But I do believe that every business should encourage monitoring across all layers:

• Operation/System Services including databases
• Application Layer error
• User Interactions

Everything has its important in helping us making right decision and fix problems as quick as possible. I believe that nothing can be 100% right unless we put them in production. With fast growing amount of data, it is not possible for teams to test most systems with scale which is same as production. Expecting fail and have a safe net to catch failure as early as possible and design a system to make problem to be fixed as soon as possible. 

If you don't have monitoring in place, do NOT do continuous deployment. 

Build Pipeline

Environments always should be first problem you solve if you want team to be agile. I have been experience in lots of companies that they are experience as following order:

1. Use product as development sandbox

2. Have staging environment and staging environment db box is used for development

3. Have continuous build and automatic acceptance testing environments

4. Then have dedicated dev, testing and staging environment

5. Then have dedicated preview sites

6. Have gateway function to select users for preview features (Not done in any company yet, only read in paper now)

Without achieving (4),  there are not enough testing before going to production. In that case, please do NOT use continuous deployment unless your system in VERY small. If you have done continuous deployment without reach (4), please let me know.I would love to know that how can you minimize risk while not having such environments. 

Testing and Quality Control

Unit testing, functional testing/integration testing and acceptance tests are essential for your continuous deployment. 

Some people out there are fans of unit testing or functional testing. I don't believe one type of test is enough for continuous deployment.  

Functional testing and Acceptance Tests will tell us whether system did what we expect. As every software engineer knows, there are many cases that you can't put into functional test cases. It tests from user point of view. But it never tests from software engineer point of view. There are lots of place that we can miss with functional testing. I have see that some one promote functional testing and they promote technical stories in the same time. It is kind of making sense because they want to cover in more granular level. 

Unit testing is essential as most agree. It give us confident in a granular level so that we will be very confident as software engineer. And another advantage is that we have static about how much coverage of testing is at the moment. It is very important thing for me. Someone says 80% is ideal line that you can have enough confidence. I would say sometimes 70% maybe enough too. And progressively working towards 80% is best way to do. But you will find that after 65%, it normally becomes very hard to do. And it is worth to write test coverage as requirement in stories from that time.  

With unit testing only, we can assure that system does what user or product team wants. And we haven't tested in an integrated environment yet. With those reasons, functional and acceptance tests are very important too.

Feature Flipping

I have not done any feature flipping in any organization yet. But I believe that feature flipping is important for continuous deployment. There are frequent requests about hide certain features for different kinds of reasons. Normally reaction at the moment, we just delay deployments. But it will prevent other important features to be released too. When team grows, the problems will occur more and more. And I am pushing feature flipping at the moment. But it may result in messy code and it needs frequent clean up are always concerns. 

Feature Team or/and Component Team for Agile Development

Introduction

Question arised when senior manager want to put out a component based team structure for engineering/development team. I can see some benefits from team structure. But in the same time, I feel there is something wrong with my gut feeling. What is wrong and what would be better format? 

After a careful thought, I believe that mixed team structure will be best for current business that I work at. 

My Thought

My thought the mixed team structure will be best fit for the business. There will be three product/feature teams and 2 component teams. 

My reason is as following: 

• There is a component requires high availability, scalability and stability which requires different skill set than just software engineering and needs to understand infrastructure. 
• There is another component similar to a component above which requires specific attention and skill sets. They will create a platform to make entry skills much lower for product teams
• But for fast delivery, to minimize handover waste, product teams will be formed to delivery end to end development except first one above. 
• Also using product/feature team to deliver features will minimize complexity of dependency management
• Product team will make handling defects easier 
• Product team will make integration testing easier and earlier in stages as features developed in one team instead of across different teams

But there are difficulties:

• For some business, building cross functional team is very hard
• Discipline and architecture guide to access codes concurrently
• A shift to share responsibility of design (I don't remember where I read an article. But consult community with member from different product teams and component teams may be an answer. Have no experience with that)
• Developing and co-ordinating common functions
• For other business but not mine
• Infrastructure reuse (I do think there is innovation solution for this most of time)
• How to build a platform to make learning of some skills easier for product/feature team as some of frequent used skills are hard to learn

I believe there are someone out there that have more experience than I do and do have different opinion on this. Please comment on this and I do believe there will be a very good debate and discussion. 

Scrum Log Jeff Sutherland: An Alternative to Kanban: One-Piece Continuous Flo...

Scrum Log Jeff Sutherland: An Alternative to Kanban: One-Piece Continuous Flo...:     Jim Coplien I'm pleased to be visiting here as a guest this week at Jeff's invitation. The topic at hand is kanban . Unfortunatel...

"Bureaucratic Hierarchy" or"Flat" for Start-Ups

Introduction

I have been wondering for what fits better for start-ups. Will that be "Bureaucratic Hierarchy" or "Flat"? It has always impression that "flat" organization will fit more for start-ups. But at this time, which one is better in general?

As agile coach and SCRUM master for start-ups, I have been thinking the problem in a scope for Agile and Start-Ups. It is worth for me to share some thought with you guys.

For you reference, it is interesting to look at organization charts.

Bureaucratic Hierarchy

I have been experiencing "Bureaucratic Hierarchy" since I have been in IT industry since 1997.  As a software Engineer and Project manager working in China, it has been the mindset that you either move to management or you get fired after 40. The mindset was embedded in everyone who were working in IT industry at that time. From my person opinion, it has been caused by

. No one want to work more than 12 hours a day from day to day
. Software Engineer doesn't produce any value other than code
. Organization runs like traditional enterprises

.com bubble partially explain that the organization structure can't support needs of startups which includes

. Innovation
. Motivation
. New skills
. Competition of resources
. Slow response to market

Flat Organization

Every company start with small and very flat organization. But most of small companies that I have worked had  very flat structure. Every senior engineers have worked as project manager, support team members, and infrastructure team members. With new technologies, all those things are quite easy to achieve. That has caused that turn around of a feature is very quick and feature delivery is mostly quite close to what is required. 

Scale up has been concerns across different start-ups. Most of organizations has been trying to get managers from big company which were transforming organization into very "Bureaucratic Hierarchy" ones. 

But if you look out, you will find that several organization continue road with "Flat" are quite successful, like Google, Valve, Apple and others.  

In the same time, there are bad sides about "flat organization" like innovation costs are very high esp for Google. 

Problems with Startups

Any solution should start with problems. What are most frequent problems seen in Start-Ups these days? From my personal experience, they should be as following:

1. Urge to make money or prove in good track to  make money
2. Stay survive in such hard economics time
3. Respond to market quickly to fullfil customer request
4. Keep competition strength over new comers
5. Competition about human resources

"Flat" or "Bureaucratic Hierarchy"

I am personally in favor of "Flat" organization with my strong Engineering background. And I totally believe Engineering should be ones driving company directions instead of marketing and sales because Engineering make things impossible as possible and provide unique opportunities for start-ups.

But in practice, I believe different organization may have their own version of organization structure to fit their own purpose.  My preference is put up here for debate. 

Let's have1 and 2 problems as last to discuss because the difficulty and too many variable from company to company. 

Responding to market quickly to fullfil customer request. As a start-up, making existing customer happy and keep advantage at existing market are super important at this time. Bureaucratic Hierarchy will make decision making harder and there is larger chance to miss opportunities. 

Keeping competition strength over new comers is very hard thing to do because entry requirement for IT start-ups are very low at the moment and it is very easy for new start-ups. Innovation is key strength as technology start-up to keep their strength over new comers. From my point of Bureaucratic Hierarchy will kill innovation because as my experience, there is no motivation from engineering team to do any innovation in that case.  

Competition about human resources are across world for start-ups. What to solve? Solution 1 will be making more money and have more people either by buying company or recruiting new employee. Solution 2 will be keeping team small and employee up to date. With such financial hard time, I believe more in second option as it will bring more innovation into team at the same time and it will keep cost of running very low. But for solution 1, it has been preference for companies which want to grow quickly. I don't believe in such solution because that fast growing won't bring best out of people. Communication will shut-down whatever you try with my experience. Component teams seem to be best solution. But cost in problem solving and development will grow at least in double size because hand-over required and problem investigation needs to involve different teams. In summary, I believe "flat" is better choice for solving this kind of problem. 

Let's think about "Staying Survive in such hard time". It is highly related to "making more money or prove to be able to make more money". So how can we achieve these two. My first thought will be "innovation" and "keeping focus on both product and customer".  "Flat" supports better for innovation. "keeping focus on both product and customer" is strictly fine with any organization structure. But communication failure is mostly associated with "bureaucratic hierarchy". It will be hard to keep company focused with "bureaucratic hierarchy" because of communication failures. From my point of view, costs for communication will be much higher for "Bureaucratic Hierarchy" organizations too.

So I believe "flat" organization structure is much better at this time for start-ups. 

Problems with bindec 64bits

I have experience some problem using bindec and sprintf for 64bits binary string. The only cause that I can think of that is float number issues in php. I haven't tried any solution. But I do think bcmath may help in the case.

<?php

for($j = 0; $j < 15; $j++) {

   $binary_string = str_repeat('1', 64);

   for ($i = 0; $i < $j; $i++) {

      $binary_string[$i] = '0';

   }

   $ret = bindec($binary_string);

   $dec = sprintf("%.0f", $ret);

   echo "\n{$binary_string}\t{$ret}\t{$dec}";

   for ($i=63; $i>$j; $i--) {

      $binary_string[$i] = '0';

      $ret = bindec($binary_string);

      $dec = sprintf("%.0f", $ret);

      echo "\n{$binary_string}\t{$ret}\t{$dec}";

   }

   echo "\n";

}

This piece of code will generate some problematic outcome. If you look at some set of outcome at top of each set for first 10 sets. Although binary string are different. their bindec and sprintf outcome are different. Now we are temporary changed to use 32 bits. But is there anyone out there who can explain what has happened; The outcome is as following:

1111111111111111111111111111111111111111111111111111111111111111 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111111111110 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111111111100 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111111111000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111111110000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111111100000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111111000000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111110000000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111100000000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111111000000000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111110000000000 1.844674407371E+19 18446744073709551616

1111111111111111111111111111111111111111111111111111100000000000 1.844674407371E+19 18446744073709549568

1111111111111111111111111111111111111111111111111111000000000000 1.844674407371E+19 18446744073709547520

......

Tips for handling changes in one sprint

I have been working across several development houses. Changes have been forever topics for both SCRUM teams and no-SCRUM  teams. As a SCRUM Master, I have tried to stopped any changes into sprint. But attempts failed and it caused business loss too.Hopefully, my experience will help you guys and in the same time, please post comment if you have any suggestion or think my ideas are shit. So how have I been handling change in one sprint?

First of all, as SCRUM masters and product owners who works in startups and small development house,  we should "expect change" in all manners instead of resisting it as most of customers are very important to business growth.

Now we are expecting change, what we should do next? Should we just let changes in? Of course not,  team can't finish anything if you let all changes into team directly.
1. Let Product Owner know,  change will slow down projects and team productivity will be lower comparing with teams without changes.  But team will expect changes if it impact business values. Some items need to be exchanged from sprint backlog to backlog.
2. Working with team to teach them about proper feeling of urgency in fixing bugs
3. Have a tool and even specific tags/categories to track changes in one sprint
My personal feeling is most of bugs should be fixed straight away if it impacts customer and bugs which don't impact customers can be fixed in 48 hours. And tracking changes in one sprint will help you finding proper

I have found that for small startups, there are often requests of features or changes directly to development team. It must be stopped. And they should send requests to product owner instead of team members.

Now changes come and product owner agree to put that in current sprint for business reason. I have few tricks, you may want to use:

• Ask requestor and product owner standing in front of backlog and move other stories out before move "changes" in
• Ask requestor and product owner send email to business about there are some stories to be postponed because of new "changes"

After two or three sprints, normally I will have a feel how big of "changes" will be with tracking tools (even just with Excel). I have tried three ways after that:

1. Leaving space for bugs and feature requests
2. Leaving space for bugs and bug exchange feature request for stores in sprint backlog
3. Still as before must exchange stories in sprint backlog for changes like bugs and feature requests

For different environment, the outcome are different. 2 & 3 are better than 1 from my personal feeling because with 1, some team may not have enough stories to work with some time because that there is no bugs and much fewer bugs.

 Now, we are here. Will we let every change in? Definitly not. Good product owners will block most of change request doesn't fit product strategy and business value. From my personal experience as project manager, I am intending to block most of changes which are not related with current sprint stories if they are in a big size. How big is big? Every team will have its own definition.

But finally, tracking tools will help you team decide what to do and what to try. Some companies with legacy software and experiences lots of bugs, first important to solve "changes" in one sprint is to increasing test coverage and focus on best engineering practice like code review, pair programming and others. Rewriting part of systems may be good option. Every team will have their own way to handle that because problems are mostly different from one team to another. And team with lots of changes as feature requests, shorting sprint length and splitting stories are good options. And of course. SCRUM Master needs to discuss with team members and product owner to find solution for that.

SCRUM/Agile/Lean Team Building for Fast Growing Startups

Introduction

Fast growing startups face unique difficulties for building development and operation teams. Although I haven't had fully successful experience, I will share my experience across few different startups. There are successes and failures.

Difficulties

There are unique difficulties for building agile/SCRUM/Lean team for fast growing startup. The ones are on top of my list are:

(1) Uncertain about how big team will be in 3 months, 6 months, 12 months
(2) Lots of pressure from business to catch opportunities
(3) There are not enough senior team members which faciliate growth of business
(4) It is hard for business to find right candidates
(5) What is strategy for team growth, splitting team or build a new team
(6) System developed is a "Legacy" system although it just has been developed from 1-2 years ago

Of course, there are lots of others. If you want to share your experience, feel free to post here.

My Solutions

I will post my solutions and ideas here for broad discussion. And hopefully, with that, we can work together to find few best solution fits for several scenarios. Feel free to give critic and feedback. I love to hear them.

Agile Way to Find right candidate

For whatever reason it is, my personal experience with few startups are that team member growth and budget are uncertain. Most of time, recruitment will be managed by top level management who has technology exposures like General Management of Technology, CTO or others.

We have personally tried or experienced following ways:

- Start with contract and then moved as full time
- Have a graduate who gona graduate in next 6 months, and have him onboard fulltime after his/her graduation
- Interview someone and have he/she onboard if he/she is right candidate
- Buy a company for its development team members

Every way has its own advantage and dis-advantage and it fits for different purpose:

(1) Start with a contractor give business an opportunity having a look at candidate's skill set for a high pressure. It fits that there are spike for one or two major projects which have hard due date and current resource can't handle the project anymore.
(2) Have a fresh graduate or someone who gona graduate is big fun and interesting. Most of them love technology and work very hard with my experience. Most of them are willing to stay if company can offer reasonable salary package. It fits for projects which require unique skills. The advantage of this way will be low cost and outcomes are normally very good. The dis-advantage is that there are risks that developers may leave in short time and it takes time educate. If there are enough senior team members, this way will be best going forward. I have successfully mentor most fresh in past 3-4 years.
(3) Have someone full time after interview. I always am keen to recruit right employee instead an employee. For fast growing company, it should never stop looking for a best candidate.
(4) Buy a company which has unique skills that we require. There are many companies who have big budgets are doing these. The advantage is that new team members can pickup skills very quickly. Disadvantage is that costs are too much. Google even just throw product away and have team members to do other things.

The "Agile" way that I want to prompt is to try different choices and add team members one/two at once. My personal preference is (2) because most of time, the outcome is fantastic.

Cope with uncertain of team size

Uncertain of team size has been a big problem for me all the time. But it has been natural for most of startups.

There is a rule that I always follow "always solve current resource shortage" instead of "trying too hard to solve future problem".

I personally find out the "cumulative" diagram are extremely helpful to decide whether current resources can cope with workload that we have.

Senior VS Junior Candidate

For any company, it is good to have as many senior members as possible. But the reality, it is hard to rightful senior candidate for a startup working in niche market. Most of time, senior team members are familiar with other technologies which are different from the technologies you gona to use. At the moment, I am more towards to team members with 0-2 years experience which can be trained to use technologies which you are familiar with.

With my experience, a candidate without experience and lots of passion on technologies and business are far better than a candidate with more than 3 years experience and has not progressed well in career.

But the skills that you need to look out for are:

(1) Communication
(2) Over-Engineering (lots of candidate are intended to do over-engineering. It is hard to change the habit.)
(3) Be able and willing to talk with users directly
(4) Strong skills and knowledge background

Team Growth, Splitting or Build a new team

There are arguments about splitting or build a new team while team grows very quickly. I am more towards splitting team, the following are my points:

(1) The most important part for development to spread is "domain knowledge". Without splitting team, it is very hard to do so esp for startups
(2) The most concerning part for splitting team is bond and connection. If we can keep most of one team unchanged, the impact is small.

My solution for agile team growth are:
(1) Let a team grow to 8-10;
(2) Split 2-3 team members out as base for new team
(3) Try to train new team leader / SCRUM Master while team growth period

Legacy System

The startups that I have worked with have the same problem, a legacy system. There are general two ways to get test coverage up:
(1) Get test written for new jobs
(2) Write a new system from ground up and make sure that at least 95% test coverage

Difficulty of first way is the difficulty in calculate test coverage. But I am strongly towards first way as risk on re-write a system from ground up. But an alternate solution would be enforcing test cases on new jobs on old systems and write test cases on new systems to make sure test coverage if there are architecture problem in old system.

I am personally find that it is very easy to use the first method if you are able to get test coverage for existing system. Ironically, sometimes, I am able to easily calculate test coverage for one system but it is very hard to calculate for another. I just find it is very hard for most PHP systems.

If you are starting new projects and want to choose a framework, you need to choose a framework with test written.