- This topic has 26 replies, 10 voices, and was last updated 8 years, 10 months ago by
.
-
Posts
-
As most of you might be aware, there is currently work going on to develop standards for software testing. This process has been ongoing for a number of years but has got attention recently due to James Christie’s presentation at CAST 2014 on the standards. From this presentation a number of attendees at the conference established a professional testers manifesto in response to the upcoming development of the standards and then a petition to suspend the publication of the standards.
For arguments for the benefit of the standards, see this presentation by Stuart Reid. A balanced view on the pros and cons of the standards is nicely rounded up by Philip Howard here.
So seeing as this is a software testing forum, where better to discuss it? What’s your opinion on the standards. Are they good for the software testing community? Should testing be standardised at all? Is there just too much to standardise?
Also if you haven’t seen it Fiona Charles hosted a Google Hangout on Test Huddle recently on the topic of certification. The ISO standards came up a lot in those conversations. You can view that here
After a few discussions on various sites and in Fiona’s hangout: I have come to the conclusion that the only real problem with 29119 is the word standard. The connotations that that brings is what has upset so many people. The proponents of the ‘standard’ are happy because they see it as a guideline that you can or cannot use and can apply in full or in part. That doesn’t sound like a standard to me.
When it all boils down, it’s no different to any of the text books published by many test professionals telling you how to be agile, or rapid, or any other methodology you like. Except the title of standard. A new tester, or anyone not experienced in test can look to 29119 as a foundation and starting point for testing. As they grow and learn more, they will disregard it as a ‘standard’; the same as the rest of us experienced test professionals.
In general I don’t think a ‘standard’ can be applied to testing. Testing as a craft is ever evolving and learning new and better ways to test new and better solutions. The problem changes (if it didn’t, an existing product would do), and so the solution changes, and the way to validate the solution changes. How can you standardise that? Even the high level approach varies between projects, industry and companies. This does not need standardisation, each project must be assessed in isolation to other projects to identify the best current, or ‘traditional’ approach that fits with the goals, limitations, resources, product format, development approach/style, stakeholder expectations etc. There are to many variables and an every growing pool of keys to unlock the potential of each project.
In a draw full of keys, only one will fit the lock and open the door. There is no such thing as the mythical one key to open all doors. We don’t live in Mordor with one ring to rule them all
@stephen, I might play the devil’s advocate here. Is there not any common issues that testers come across that a standard remedy could be applied?
Also, as others have suggested, maybe the standards are a method of establishing a baseline of procedures and building upon that.@Ronan. It’s a valid question.
As an example we can take one common problem I’ve faced, and a believe everyone faces: The Test Plan.
There are common elements to a Test Plan that should be included, and you might say these can be found in a standard. However; the common elements are also common for any ‘project’ plan. So this is not really necessary to be kept in a testing standard, it should be referenced to a common project standard, or basic document type, in fact I’ve seen different companies have different templates for all corporate documents that define the basics.The test specific elements change depending on the method used, waterfall, agile, spiral, rapid, etc.. Some will require you to include test environment used, others will be run in production, some will discuss the tests; but I believe the tests should be left out of the plan and kept separate: It’s to limited for the dynamic world we live in. Agile will no have a specific Test Plan, rather a charter. Even when you settle on a style of testing that requires a test plan, one plan may work and be acceptable in one company, but not in another. I’ve even seen the same happen with teams in a company. You might argue “But isn’t that why we have standards?”.
If every project I work on in my company had to have the same test plan template/boilerplate with style and definitions dictated, not all projects would get past the Test Entry gate. Whilst I have a documents style template to maintain a consistent look, I have to tailor each test plan content and sections to the project type. A Service Desk test plan does not fit an asset management test plan, or a Business to Business communications tool. The content and sections for each plan is very different.
Note there is already a test standard for test plans IEEE829, I have tried to use this many times, in fact for each new company I start at. Not once has it really served me. Once I have hacked it, removed the in-appropriate sections, added required sections and made it work, the resulting test plan does not resemble the standard.
In short if my stakeholders were to get wind of a ‘Globally accepted Testing Standard’ that defined the style and content of a Test Plan, it could add weeks of time and money trying massage it to fit each project and gain the test plan signed of by all stakeholders. If we every manage to make each project fit and manage to explain the document to all stakeholders.
I think I’ve said before, a standard is good for someone new to the business. i.e someone who doesn’t have a base to start from. But once they learn more, I’m convinced they will stray from the standards and do a better job.
“I think I’ve said before, a standard is good for someone new to the business. i.e someone who doesn’t have a base to start from. But once they learn more, I’m convinced they will stray from the standards and do a better job”.
I think you are right there. Seeing that so many seem to come into the industry from different backgrounds (sometimes not even related to computers), a set of standards might be very helpful for new comers to the industry?
Just out of curiosity and to give another example, if there are, as you say, so many diverse ways to complete a project, how might agreement be decided on a project? (I suppose that is what the manager is for!)
Could standards potentially be sub-divided for project type?
The Test Manager needs to assess the best fit method for the solution, the project, the company, the team and the stakeholders.
However a company often has a preference for one or two methodologies and will recruit the team to support that. One of the hardest jobs of a Test Manager is to continually evaluate new methods and approaches and where appropriate introduce them to the company and its projects: Change is not easy to introduce; People like “If it ain’t broke don’t fix it!”. Where as a tester says “We don’t know it isn’t faulty, let’s look for a better way to do it.”For a standard to cover all methods available it would have to be the size of the Encyclopedia Britania; Which would make it unweildy. And with the ever maturing testing industry a truely representative standard would have to be a living document, with a update lifecyle in weeks not years.
I think the main problem I and many others have is the word ‘Standard’. It doesn’t matter if the official definition refers to it being a guideline, the moment you use the word ‘Standard’ the conotations are that it is mandatory. And a lot of us cannot see ‘standards’ like IEEE 829 or ISO 29119 being all encompasing efficient processes, that can be applied liberally, in it’s natural form, so that the uninitated can enforce it without compromising quality and cost; or alternatively use it to leverage liability against the professional tester who is not ‘compliant’ with an ‘internationally agreed standard’.
Personally, I think ISO 29119 will go the same way IEEE 829. It will become an accedemic document that is taught for certifications and then abused and mutated beyond recognition when adapted and applied to a real world project. I’m not overly worried, just not concerned.
If ISO 29119 was labeled a guide to how testing can be managed, or a foundation for test management; then it would be easier to accept without opposition. How can you oppose a guide, unless it is dangerous. The current danger in inherent in the word ‘Standard’.
As I remember from my university study where, I studied bit deeper on ISO 29119 and IEEE 829 standard in the subject called “Software Engineering”. But, when I started working with industry, I personally feel there is no need of take standard very aggressively (using forceful methods to succeed) with the “software testing” due to the “complex technology”.
ISO 29119 and IEEE 829 standard not bring constant value to software testing industry, due to the rapid growth of the technology.
But standard help to understand the basic principles to anyone about “software testing” or you can says 29119 and IEEE 829 standards are pillars to your building, but it’s also depend on you also how you build the building on your pillars.
@padmaraj I would agree. Standards would seem to be a good guide for new entrants to the industry but might have less relevance the more experience one develops in the industry?
I haven’t seen many people in favour of the standards. This blog post on the Test Eagle blog at least offers reasons why people might be for it. It also echoes a lot of what you said Padmaraj about the standards and is a good read on the subject too.
I mean, I like to adopt the standards, but with complex technology, standards become more time consuming.
So most of companies, derived many things from ISO 29119 and IEEE 829 standards and used according to their internal requirement with the project.I strongly agree that, standards help a lot in term of documentation and process management with QA.
@padmaraj I guess we will have to wait and see what the standards look like, if and when they are published.
I found another interesting article on this topic. it is by Freddy Vega: An Open Letter to the Testing Community with some interesting responses from James Christie and Michael Bolton. It suggests that the petition #stop29119 might be polarising the testing community.
@Ronan, “polarizing” is an understatement! What good ever comes from zealotry? What I witnessed minutes ago was disturbing. I have no interest in engaging with bullies and inciters of mob violence. What beneficial purpose would be served by engaging? The mob isn’t interested in discourse, they are simply in a frenzied blood thirst. Has this form of behavior served human societies well?
Greetings, I find my self in the same boat with those that believe that, as @Geoff puts it (paraphrasing) “what do we gain from inciting mobs”? and the answer, for me, is that there is no purpose in engaging. Maybe I’m weird like that, but standards don’t scare me.
And Geoff, “polarizing is an understatement” — I concur!
What if we were to take a value perception view on this? What value will the standards bring in the eyes of a customer? A customer being the real end user of the system under test. If a standard provides a test professional with structure and this leads to a customer wanting to high five the person next to them because they’ve just had a great experience using the thing that said test professional had tested, then the standard has indeed been a success. The opposite is of course true. The standard is used and it adds no real value to the end user then the standard has been unsuccessful.
With my A/B test head in place I’d be intrigued to run an experiment. Take two sets of test professionals who work for the same company where both sets are serving a similar customer base. One group uses ISO #29119 and the other doesn’t. Chuck in a customer NPS survey and let’s see which one comes out tops.
@simon Interesting thoughts. That sort of reminds me of those psychology experiments done with identical twins to see how environment influences personality. I guess people’s view on it would depend on how they came into the industry and where and from whom they learned their testing skills.
Also probably the most comprehensive article on the ISO debate I have found so far.
Link is here.Another very comprehensive article (it has a lot of links) on the ISO debate. It’s by Eric Proegler. You can read it hear. Eric is opposed to the standards but I found a quote in his blog post that I thought was quite interesting and relevant to all testers:
“The risks of autonomous software issues in medicine, transportation, economics, energy, and “defense” need to be met by engaged, expert testers. We need thinking, exploring testers with time and space to do their best in order for our future to be safe“.
What do you think?
I like that quote Ronan. When you’re talking about those five areas then of course you need the best of the best. Certainly puts my experience of working on a major classifieds website and major real-estate website into perspective. The worst that could possibly happen with those would be that a tonne of customers got annoyed and revenues took a hit. No big deal relative to the importance of the five areas mentioned in that quote.
I am just curious when should I start carrying about some standards in my job as a tester, since I’ve never had, nor I see in the near future an interaction with them. I think that a company that I might work for some day, might want to try and apply one of these standards, ok. But, won’t they realize it’s useless after 1 or 2 months of trying to implement them? I wouldn’t mind, cause it would just be an experience of lets say the dark side of testing that I might not like. But then again this is a good thing cause I could form a good opinion for the next company I might work for that would want to implement standards.
In conclusion my current stand is that I don’t care, until they prove themselves useful for a company, or for a testers team, or for anything in general.
And all this discussions about whether they are useful or not, might just convince some people to go ahead and buy the standards and try them.
You know that saying – the more you are said no, don’t do that, don’t try that, more you want to do it…I have tried to write a comment regarding the standard and was hoping to keep it as neutral as possible. Link
And as mentioned by Stefan that most likely this will not really be applicable to most companies and organizations, and will then be an expensive experience and hopefully this will be spread to others and they can then make a more informed decision on implementing the standard or not.
The talk that the ISO29119 should be used by medicine as an example to further improve quality and control seems to me a bit like paying twice for something, dont most medicine companies already follow the rules set by FDA ?My hope is that: the more we know about the standard, as more and more information is coming, the better the discussion regarding it will be.
The focus must be on the facts and kept on a good level in order to let people respond and actually be heard.
Most likely the majority of the testers today just don’t believe in standards and feel that the discussion is a tempest in a teapot.Sure ISO29119 is getting a lot of free “advertisment” in this discussion, but the more one knows about things the better one are to make the right decision when the time comes.
@bjorn Going on a lot of comments here, it does seem that many in the testing community do not have strong opinions and have yet to make up their minds about them.
@stefan I see what you are saying about the negativity towards the standards. Do you think if some testers are saying “Don’t use these standards” that it might drive others to look at them?
Another interesting comment on the debate. I really like the sub-heading on the article: “Can’t We All Just Get Along?”. It’s here if you want to read it.
Some of you folks might find this interesting. It’s a well chaired and structured debate on the ISO 29119 standards. It features arguments from Rex Black, Jon Hagar, Jean Ann Harrison and Griffin Jones
A statement on ISO29119 from eBay shared online recently:
https://docs.google.com/document/d/1WJPO9zoPY7X4hRCGXimx89soNti5-vG5pbQDOO5Tto0/edit
Not sure if it’s an official statement!? Maybe someone can confirm?
It’s a one page statement that finishes with the following paragraph:
“For that reason eBay will not adopt or implement the ISO29119 standard, nor do we support the standardisation of testing in any form. It does not make sense to do so. We make this statement so that others can help themselves to make an informed decision about whether or not they think standardisation of testing is useful for them.”
Is this likely to be a recurring theme from test teams?
- You must be logged in to reply to this topic.