I don’t believe in Acceptance Tests
There – I said it. Heresy. But, I am yet to be sold on their value.
Before I continue, I should describe my observations of acceptance testing. From what I’ve seen, acceptance tests are created as assurance that a story is complete. They are a layer above unit, integration and persistence tests. Their assertions are based around acceptance criteria, and verify the functionality for the story. And, from what I have seen (as with all other tests) they are not constantly maintained, once written forever forgotten, and quite long to run.
Somewhere down the road of the project, the build takes too long to run, or you spend too much time fixing broken acceptance tests or you have flaky acceptance tests, and your mind starts to filter out a red build, thinking the problem must be caused by the said flaky test. A common workaround to these problems is to stop writing as many acceptance tests.
I guess my biggest problem with acceptance testing (in this form) is that I can push down any assertion that is made in the acceptance tests onto integration, persistence or unit tests and have a higher level of confidence that the acceptance criteria has been met. In fact, when I finish developing my story, the integration/persistence/unit tests already cover all acceptance criteria.
Now, you might argue that acceptance tests make good reference material, as there is one place you can go do to find out what the story was meant to do at the time of sign-off. If this is true, why do I see developers constantly referring to the code or unit tests for this information, rather than the acceptance tests?
I don’t see the value of these story-based acceptance tests.
What I can see real value in is scenario-based testing. What I would like are a handful of scenario tests which describe the system as a whole. After all, the stories we develop have all been derived from some scenario which delivers business value. That way, as I develop a new story, I just extend the scenario tests to capture the essence of the story, but have all acceptance criteria still tested on the unit/integration tests.
Testing is a form of art; I don’t think we have mastered acceptance level testing. Perhaps scenario testing is the answer, but it requires constant improvement to them, maintaining them, refactoring were necessary; generally looking out for the well being of the tests: one thing I don’t think we are good at. Remember, refactoring your code base is just as important as refactoring your test code.
Posted: September 29th, 2008 under Testing.
Tags: Testing
Comments
Comment from Jason Yip
Time September 29, 2008 at 12:00 pm
The way I think about it is like this. The Story acceptance test was valuable at the point when the story mattered. After that, we’re interested in something that efficiently describes the system, not something that reflects an obsolete mechanism that was once used for project management.
Pingback from Where do Acceptance Tests go to Die? at Fragmental.tw
Time September 29, 2008 at 1:05 pm
[...] My colleague Sarah Taraporewalla posted an interesting text on acceptance testing. She doesn’t believe in this technique. I’ve been thinking of acceptance tests for some months now and think that she has a valid –a bit too radical but still valid- point. [...]
Comment from Mark Burnett
Time September 29, 2008 at 3:28 pm
I have also been on a project where the acceptance tests were not maintained well.
I think they could have been much more useful if everyone took a bit more care of them.
I would be interested to find out what you see as the difference between integration and acceptance tests.
Comment from Daniel Wildt
Time September 29, 2008 at 4:33 pm
Good one.
I believe acceptance testing is a good way to keep track in what is important with one story for the whole team.
I focus on proving it works with integration/persistence/unit tests.
But… we cannot forget that we must have useful documentation… and maintainability is an important attribute in that way. So, if it is not working, the team need to figure out some other way.
In my current environment, not every test is approved to become automated, and even if approved, it’s based on a scenario, not a story.
Comment from Jay Padinjaredath
Time September 29, 2008 at 7:05 pm
I agree with you. If you read Deming, he indicates that testing a product’s quality once its manufactured is wasteful. You’re better off improving the quality in the first place.
I was thinking about unit testing and acceptance testing, the former helps bring quality into the product (and more) and the latter tests the product after its finished.
I have posted Deming’s 14 points here:
http://blog.jayanthan.com/2008/09/skip-to-content-w-edward-deming%E2%80%99s-out-of-crisis-ii/
Pingback from It’s not all about the acceptance tests at Mark Needham
Time October 2, 2008 at 3:30 pm
[...] few of my colleagues recently posted their opinions about acceptance tests which tied in nicely with a [...]
Pingback from Sarah Taraporewalla’s Technical Ramblings » Are types of testing important?
Time October 2, 2008 at 8:45 pm
[...] comments on my last post about acceptance tests have made me think a little more about testing, particularly the value in specifically declaring [...]
Comment from toni
Time October 3, 2008 at 6:33 am
I’ll quote Eric Gamma in XP Embrace change he writes: “If testing is good, everybody will test all the time (unit testing), even the customers
(functional testing).”
Dev writing tests is good, but not enough, making the customer writing tests we’ll ensure the system works as he really wants, not as we understood it should work.
Comment from Sarah
Time October 3, 2008 at 6:58 am
@kerry – I am glad that you refactor your tests – I assume that you align your functional tests with acceptance tests when you first do your story ie don’t have a separate test just as one point of contact to show that the story is complete, but rather dispersed throughout your tests.
@mark – the difference that I have seen in acceptance vs integration tests is that while integration tests are run in the same test suite as unit tests (ie nunit/junit/test runner) acceptance tests often use a different tool (eg JBehave, RSpec, Selenium, Watir/Watin). Because you are writing the test in a different tool set, for one you constantly need to switch between the different styles of testing. Also you tend to need to repeat your self in acceptance tests to bring the environment into a state in which you can start testing (ie implementing all the Given conditions first).
@toni – Have we really got sufficient tools now to get the customers to write functional tests? You may argue that JBehave2 aims to put this in the hands of the customer, but at the end of the day, the developers need to implement those tests. I have found that letting the business people specify the tests (ie they write given when then) and you take that interface and write the implementation underneath leads to inefficient tests, which means long build time which has a spiraling effect.
@all – thanks for the comments.
Pingback from thekua.com@work » Automated story-based acceptance tests lead to unmaintainable systems
Time October 12, 2008 at 8:24 pm
[...] teams seem to know what to do. It sounds exactly like the scenarios that my colleagues, Phillip and Sarah are experiencing or experienced [...]
Comment from PM Hut
Time November 4, 2008 at 8:14 pm
Acceptance Testing can be viewed from 2 perspectives:
- The development company (A)
- The client (B)
I want to focus on the financial aspect of the acceptance testing. (A) and (B) usually do the acceptance testing together (I know I’m saying the obvious), and at the end of the test, (B) can either accept or reject the product, in the latter case (A) as to work more in order to deliver the product to meet (B)’s criteria. Acceptance testing is usually loathed by development companies and sometimes (I’ve seen it happening) (B) uses acceptance to its benefit, such not releasing payments, etc… When the acceptance is signed, it means that everything is OK with the product, and all outstanding funds have to be paid, and the product will then move from development phase to maintenance phase (where the monetary part will be handled by a different, ongoing contract mainly turning the customer into a cash cow).
I think acceptance testing are unavoidable, but usually both parties (especially the customer) use them for their advantages.
PM Hut – http://www.pmhut.com
Comment from Kerry Buckley
Time September 29, 2008 at 7:54 am
Described that way, I can see why acceptance tests might not serve much purpose. I treat them a bit differently though.
We don’t have separate integration and acceptance tests, but use the accumulated acceptance tests for integration/regression testing in the CI build. Because we don’t worry too much about traceability of stories and acceptance criteria once they’ve been signed off, we are free to refactor the acceptance tests, organising them around functional areas rather than individual stories. This keeps them maintainable, and allows us to hold onto the discipline of not allowing the build to stay red.