Automation ROI

Today we hold a team postmortem about the last release, all the team members including lead/manager sit in a room, sharing the thoughts on the good-bad-todo review list. In the course of the discussion, I raised a point which has to do with the automation ROI, but looks like it's somehow controversial to others.

The problem is about the automation ROI (Return of Invest). Sometimes I feel its value is not very ideal as we expected. Take a look at the bug number which's found by autmation and by manual, and then calculate the cost we put into the automation. Do you find it has a high cost-performance? I suspect it and my guess is some person also has the same feeling =(

 

The reason I said so is not that because I don't like the autmaiton, instead I like the test framework in my team very much (it's a very powerful dev code-base for me to learn..), but it's depends on the specific test strategy - why I need to automate everything in my testing? And it's also impossible task,right? AWAK,80% feature coverage is ideal which I know from my team's original plan, but actaully we never have that =(

 

 

A detailed example is some guys in my team need to automate a feature which is related to site-publishing functionality. We target to automate as many scenarios as possible with our test framework, such as publshing site via differnet protocl (http, https, ftp, ftps, sfpt, etc) on different server, with different size of file, and then verify the file persistence, and UI side, and the publshing time if needed. I see they spent a lot of time on this work item, and recently they are working on adding many regression to the test code base, but they actully never have a 100% test pass in the daily run, so they never stop fix the code in the last 8 months (make the code test pass like a robert =! ) The final feature is reaiablt but we actually put too much on it, don't we? Sometime they also wonder why test lead wants to make it automated instead of manual?

 

 

Anyway, I DO think we need automtion to do the dupe-work for the regression, do the E2E scenario like a customer side and other types of test cases, but what' more important think is a good tester should not only know what/how to atuomate the testing work via the available tech, but also should be aware of when to use the automation - a better test strategy in the early time may bring a lots of working in building the test infrasture, but in the next, next version, you will see its potential ROI.

-----------------------------------------------

 

I often get thoughs from BJ's blog, which is a good place to me to learn and learn.