I have problems with the first axiom, mostly on political or pragmatic grounds.

I think you have to identify the people who will approve or disapprove your process and temporal or monetary budget. These may be very different from the people who will use and benefit from the test evidence.

You need to frame your tests at least as much for the guy who will decide whether to fix a bug, as for the guy who will eventually use the software and have to put up with the unfixed bug. (The assumption here is that it's the same guy, but experience suggests it often isn't.)

Yeah, that's pretty cynical. But you need to be aware that you're not testing in a vacuum, and that your testing may shape the final product fully as much as the initial design did. It's not just what you test, but how you go about it and how you present the results.

All too often, authority is in the hands of those who don't have any direct responsibility for the outcome. What you present and how you present it to, e.g., the vice-prez who has to authorize the bugfixing budget (on a project that's already overdue) may have a critical impact on what gets fixed and how well - particularly if what you're pointing at is really a basic design issue.

Paul Replies...

I don't think this is an objection really. In my mind, I think of stakeholders as the people who use the information I provide as tester. Notwthis very definitely includes the developers who fix bugs of course. And if I'm a user-tester, I might be my own stakeholder. But as often as not, user management, product and project management, the product 'buyer' all use and/or benefit from the evidence provided by testers. Essentially, Axiom 1 says "know your customer". We conduct tests to learn how software behaves and we need to interpret/analyse/communicate that knowledge to our various stakeholders. So stakeholders is a fairly inclusive term - we need to know who will use out test evidence and construct our approach and method of comunication to suit differing needs perhaps.

Developers need to know exactly how to reproduce a behaviour so they can diagnose it. A user manager needs to understand the patterns of failure so he can make a call - "can we run this business using this versin of the software or not?" These are different information needs and we need to understand exactly what information these folk need from us.

Your stakeholder might be the guy who is paying. Most of these stakeholders like to see some form of substantive plan against which they can track progress and the spend of their money. Over-planning is a common response to over-cautious stakeholders. Without a plan, or any process which describes the activity and deliverables being funded it's hard to get the funding. Other stakeholders might be less risk averse and be content to be involved at every stage, to 'see for themselves' rather than rely on heavywieght documents and periodic reviews. It is surely a good thing to know who you are dealing with - your approach to getting funding and conducting your tests is critically affected by this.

Imagine getting that wrong?

Okay, I'll start. :)

a) Axioms? Or heuristics?

b) Is this intended to be descriptive of what testing is like everywhere, or what it's like in a certain context? Or is it intended to be prescriptive of what testing should be like, presumably everywhere? (This may answer question (a), too.)

c) When you say things like, "Define the sources of knowledge whether documented, physical, experience or hearsay-based to be used to determine expected behaviour," do you mean in advance? In an ongoing way? Retrospectively?

Cheers,

---Michael B.

Paul Replies...

Thanks for the comments michael.

a) Axioms. I'm asking the community - are there some absolute fundamental rules which goven testing? To meet the needs or follow the lead of an axiom, one might choose to use heuristics to plan a test or you might use (different) heuristics to decide exactly what you do next in an exploration. Heuristics imply choice. Axioms don't.

b) Descriptive isn't the right word. An axiom is like a law of physics. It doesn't describe what you do but if you ignore it, well you'll get in trouble in some way. An axiom just IS. For example, every tester in some preplanned or immediate way should have a notion of coverage in their mind (or documented somewhere possibly - take your choice). I'm not suggesting how or when they do this - but coverage is one way we are able to assure throughness or some measure ofcompleteness or significance in the evidence we give to stakeolders. I'm not suggesting "Thou shalt define a coverage target and pursue it relentlessly". To answer questions like "have we enough test evidence to stop testing?", "how confident can we be in our decision?", "are there parts of our software we haven't tested yet?" - the notion of coverage is essential to answer those questions. Suppose you can't answer those questions? "Don't know" isn't a comfortable option. Whether you have a pre-planned notion or one in your mind, the decision to plan one more test or click one more button on a screen is based partly on the notion of coverage real or percieved.

c) Yes. All of them and more. I'm suggesting if you don't know what software should do you can't judge whether it works or does not. (Maybe an axiom 'define "works"' is needed here :-) ). But that's an unsatisfactoryanswer.

Somehow, you need to compare in some way, the behaviour of software with an expectation. Sometimes its black/white - pass/fail - sure. But if the behaviour is "interesting", it is interesting because in some way, it didn't behave the way you expected. The oracle axiom simply says, understand what you will use to make that call. A document, an existing system, your experience, your imagination. Whatever floats your boat. The important thing is to recognise what you are using as a source. It's only fair to let your stakeholders know that. If you don't qualifiy your comments by stating your oracle - you are in BIG trouble. For example, when discussing behaviour with programmers, surely it would help to say, "The spec says...", "my experience tells me...", "I had imagined...". Simple really. The essential thing is to recognise the basis upon which you make comments on behaviour. Universal - isn't it?

Can you please clarify your meaning or interpretation of the word "Axiom" (meaning there are many meanings for this word)

a) A self-evident truth
b )An irrefutable principle
c) A universal truth that does not require a proof
d )This is truth, do not dispute it. This is unquestionable.

I sense that "irrefutability" or "no need for proof" is fundamental to your axioms. Am I right?

If yes... here is my first question. Why do you think that unquestionable truth and do not require proof?

I am questioning the basis for calling these 12 items as axioms.

Paul replies...

I prefer the notion of self-evidence. That is, I'm not worried or threatened by trusting an axiom to be true so I spend my whole life trying to disprove it. I would use an analogy.

It seems self-evident that it is a good idea to drive on the right side of the road. Not the left side or the right side. The CORRECT side. Now I don't need mathematical proof to tell me this is a good idea. What is the axiom? "Check which side people drive on before you start the car".

Is this unquestionable? Certainly not. Am I worred that this axiom will ruin my life? No. I can live with it without getting neurotic or paranoid.

What might a testing axiom look like? "Find out what my stakeholders want before I waste a lot of my time and their money".

>>>Heuristics imply choice. Axioms don't.

Would not that be a very "narrow" and "static" view? If I accept your 12 axioms of testing - I have no choice but to believe them and follow them. Can a dyanmic world of testing be that "fixed", "understood" and "defined" ?

It creates more confusion and scare in me as these axioms do not tell me what will happen if I ignore them accept that something bad will happen and I must just follow them. Do I have a safe exit path in case my world of testing, things I value for, results that I expect are not in consistent with the model of testing as characterstised by your axioms.

So, don't you think, heuristic is more flexible as it provides choices in a world of software testing where there are as many as practices as many as stars in the sky?

Why move towards a world where there no differences and no thinking required - just follow what axioms say. They have defined "a good testing" world for us ? right?

Shrini

Paul Replies...

Great points, which I think I am 24 hours ahead of you on.

The execution sequence axiom suggests that it is a good idea to put your most valuable/important/interesting tests early in execution. Whether you are doing heavily planned and scripted or exploratory, it always makse sense to do this within the context of your allowed timescales.

You ARE limited by time (every project has a budget limit somewhere) So it makes sense to use that time/money wisely, if, for whatever reason you start late,or have to finish early or are squeezed because the quality is poor etc.

How do you prioritise? The prioritisation axiom suggests you agree a mechanism with stakeholders rather than guess or take responsibility (when it is not yours to take).

What happens if you ignore an axiom? Well, this is a natural next step and might help explain them better. If I ignore the coverage axiom I cannot articulate how much testing i might plan, or have done or have left. I cannot say when I will finish. Not good.

If I do not establish a need for an environment, I will get one too late or I will get the wrong environment. I cannot test or I cannot trust my tests. Not good.

And so on.

Heuristics are one way (not the only way) of deciding what to do next. I can use heuristics to design tests (the design axiom) I can use heuristics to prioritise (the prioritisation axiom). I can choose to use something other than heuristics if I want. Your freedom is complete. but isn't it a good idea to use your brain before selecting a test and maybe use an heuristic to be systematic? The design axiom says make a choice.

The axioms do absolutely the opposite of 'not thinking'. They are the antidote to planned thinking - i.e. not thinking at all (e.g. the IEEE 829 as a strategic approch - yuk).

An axiom challenges you to think: "you had better think this through and choose your solution".

Hi Paul

There can't be axioms since software testing deals with people, and for people's behaviors and expectations; nothing is irrefutable.

But there can be guidelines, principles, practices; and as such I think your axioms seem context-dependent on larger projects and commercial situations (where cost and time is important.)

My counter-example is from last week:
I was looking at an art project (emotionalcities.com) and found some strange behaviors. As a tester, I couldn't help looking more carefully, and it ended with an e-mail to the artist with some major and some minor software issues on their web site.
The input was well received, at least one bug has been fixed; and therefore I consider this activity to be good testing (not complete, not the best, but better than nothing.)

I did not adhere to any of the 12 axioms.
I implicitly used parts of a few of them, e.g. I sent one-liner bugs in e-mail (axiom 6); but this was not defined (it was just done) as is stated in the axiom.

In this situation (and in many more); many actions implied by the axioms would take too much time from the actual testing effort.

Regards,
Rikard

Paul Replies...

the axioms as I've stated them are not about people - they are about thinking and decision making. Yes people do the thinking and decision making and there's a million ways their thinking might be biased or muddled or their prejudices mislead them.

But let's look at what I think you might have done in your generous feedback to the website owner. I expect you saw some anomlaies and then decided to investigate and then let the site owner know what you found. My expectation with regards to the axioms would be the tester is employed to do some testing. But I think most of the axioms are appropriate even so.

Axiom numbers in brackets.

You decided that the site owner might want to know aobut some anomalies (1). The test basis selected itself as you noticed some anomalies and felt they were worth investigating (2). Sound like the oracle was your own experience (3). Scope and coverage - clearly you spent a limited amount fo time in your investigation and needed to limit it (4, 5). You used email to deliver the comments (6). Your environment was pre-defined (7) - you chose not to use some other environment. This was an ad-hoc activity. The only event was your email notification so I don't think us thought about 8. There must have come a point aw which you thought - I've spentenough time on this - and the tests you had run must, in some way have been deemed more important/interesting than those you could have tried but didn't. Implicitly you had an idea of priorisation and sequence. (9, 10). Sounds like you used your experience and intuition to try things out (11). If the website designer tells you about the fixes - will you retest? would you regression test? (12).

I know this isn't a 'real' test as we would both recognise - but the thought processes - I think you probably consciously used 11 of the thought processes associated with the axioms. It's a very loose connection perhaps - but it's a very loose test activity.

Nice list, but I wonder why only two axioms include “managing changes”. Everything may change – stakeholders, their goals, and oracles as a result. Let me give you and example from recent project – I’m doing back-end test automation and a project manager comes to my place and say: I know front end was not priority, but I need you to (Ad Hoc) test it now and report as much bugs as possible by end of tomorrow. And I have no problems with that.
Besides as a mathematician I had problems accepting how you abuse term axiom (original idea is to list assumptions as basis for further theory). It seems you have listed checklist to define assumptions for each project, which is great of course.
In this case I would extend/rework some items. For example I would extend Axioms 8 and 6 with and say that depend on:
“Project change”: how much you are willing and able to accept unplanned events to influence testing (e.g. started test cycle can’t be influenced – only the next).
“Project Impact”: how much unexpected informational findings during testing (does not include expected problems such as some test failures or high defect rates) could influence project

Paul replies

I know - as several folk have already reminded me - that axioms have to be failry watertight to be called axioms. I'm an engineer by training so on the one hand I'm very uncomfortable abusines the laws of physics, but to make life easier we compromise all the time - perhaps that's part of the way I'm thinking.

Also - this is a thought experiment. It's uncomfortable for me to be criticised - but who would think seriously and comment back to me if I had used the term 'musings' or 'thoughts'. Not many. So it's a deliberate challenge to people to think outside the dogma preached by some folk.

It's possible the 'axioms' could be called 'thinking tools'. I have no problem with that - but the criteria for inclusion is they must be used. Heavens! We've used checklists forever. But checklists STOP people thinking too often and that's what I want to avoid at all costs.

Now, I take your point about change and the impact it has on us. But I'm going to suggest these aren't testing issues. these are project management, change management, maybe configuration management issues. I have paid a small amount of attention to them by referencing change in axiom 4 and I suppose there's a hint in axiom 8. Your comments are definitely on my list of things to think aobut. Maybe there's a 13th axiom :-)

Thanks for your post.

Paul:
Further to my replies against your other two current entries (“Does a set of irrefutable test axioms exist?” and “Schools Out!”) – here is my response to your suggested content of your proposed 12 axioms. As you might expect from someone of my disposition:
• I’ve cross-related your 12 to my 33 “always-good-principles” (or maybe now merely heuristics) – I’ve also thought about two other cross-checks:
o Kipling’s what-why-who-where-when-how; and
o the James Bach – Michael Bolton “Rapid Testing” course, http://www.satisfice.com/rst.pdf

There’s loads of material in James & Michael’s course – and I’m not just grabbing the slides, I have attended it, most enjoyably, the exercises are key – it’s all presented as things you might consider using in context using and developing your skills, not axiomatic – (and by the way, I suppose some contexts could require non-rapid testing!). One slide in particular (19) looks quite close to your list, so it could be claimed on this evidence we may be arguing more about philosophy, semantics and the amount of detailed prescriptiveness, than essential content. (But if course, that is only one of the slides in the course, it represents the test design & execution part; input to test design includes project environment, quality criteria & product elements – slide 17).

Here’s a rough tabulation (my opinion, done very quickly so not bulletproof, comments please – and I’ve numbered and expanded my principles here subsequently to the STAREast paper):

Gerrard..........................Bach & Bolton...........................(Kipling) Thompson

1. Stakeholders..............1 Model test space?....................(Who) 5
2. Test Bases.................1 Model test space?....................(Why) 4, 5, 20, 21, 24
3. Test Oracles...............3 Determine oracles....................(What) 13
4. Scope Management...1, 2, 3 & 4 Procedures?..............(What) 28
5. Coverage....................2 Determine coverage................(What) 25, 26
6. Delivery.......................9 Report test results?.................(How) 9, 10, 31, 32
7. Environment................5 Configure test system?............(Where) 18
8. Events.........................4, 7, 8 & 9?.................................(When) 26
9. Prioritisation................2 Determine coverage?..............(Why) 22
10. Execution Sequencing 6 Operate test system?...........(When) 23
11. Design......................2 & 4?.........................................(How) 23
12.Repeat-Tests.............6, 7 & 8 Evaluate test results?....(How) 30
13. (suggestions?)..............................................................SEE BELOW

So... all Kipling’s honest serving men are represented, but is the balance right? I don’t know, what do you think?.

Of my 33, the following don’t appear represented explicitly in your description:
1. Must seek effectiveness (doing the right thing in terms of verification & validation, Boehm’s definition);
2. Should also seek efficiency (doing the right thing without wasting time or resources, making best use of opportunities therein);
3. Should decide process effectiveness/efficiency targets (and improve over time if appropriate);
6. Should be pragmatic with quality targets (not too ivory-tower, not too laissez-faire);
7. Should define & use metrics (of some appropriate kind);
8 Should have some awareness of ROI principle – testing as “insurance”;
11. Should plan acceptance ahead, to minimise nasty surprises, eg “rehearse” acceptance tests;
12. Should use some kind of handover criteria (between test levels/stages), and acceptance criteria;
14. Should use some kind of W-model (even in iterative lifecycles) – consider reviews/static testing, and early test chartering/planning;
15. Should use some higher-level testing independent of developers (even if this is just the users themselves);
16. Should use an appropriate skills mix in test team (best you can get within constraints);
17. Must define & agree roles & responsibilities;
19. Should use appropriate tools (including automation) appropriately;
27. Document (or at least ensure known) overall project/product test execution & management procedures;
29. Must prioritise (respectively) urgency & importance of each bug, and handle accordingly;
33. Should assess causes of bugs (as far back as practical from failure to fault, defect, error, bad working environment/peopleware context, tie-wearing managers...).

So, candidates for “axiom” 13 are as follows (though I also prefer a different term). And also you might proceed, in the manner of the Spanish Inquisition – no-one expected that? – to at least 16 of them, to include something about:
• reviews/static testing (my 14),
• tools/automation (my 19),
• bug management (my 29), and
• error source analysis/process improvement (my 1,2,3,6,7,8 & 33).

My #11 might be considered to be part of the stakeholder axiom; #27 might be part of execution sequencing, though I’d put it wider than that;. #s12, 15, 16 & 17 might be part of test bases?

Regards,
Neil

Paul replies...
I'm really busy right now, but will reply asap

>>>>>>Heuristics imply choice. Axioms don't.

>>> Great points, which I think I am 24 hours ahead of you on.

I am not (could not figure out - how you are 24 hours ahead). As reply to my queries about axioms/heuristics - you went on explanining one your axiom - sequencing axiom. Note I am not commenting on any of the axioms yet. I am having troubles in accepting your line of resoning that the items you have listed are axioms (universally applicable) but not heuristic.

You seem to use the words like "good practice", "good idea (in some context)" at times and suddenly shift to "must follow", "no alternative" and so on. What I am missing here?

Paul replies: I posted the blog as an experiment. The test I posted as 'axionms' is not well thought out and too vague. I WILL post a revision to them in the next 48 hours. (In the meantime, I have to earn a living ;-) )

So the words you've picked out are inconsistent. I agree with your point and I'm working on better definitions. The new text, I hope, will be much clearer.

>>>Heuristics are one way (not the only way) of deciding what to do next. I can use heuristics to design tests (the design axiom) I can use heuristics to prioritise (the prioritisation axiom). I can choose to use something other than heuristics if I want. Your freedom is complete. but isn't it a good idea to use your brain before selecting a test and maybe use an heuristic to be systematic? The design axiom says make a choice.

Your explaination of Heuristic is line with my understanding. So your axioms are atually "heuristics" - fallible methods for solving a problem under *some* circumstances. Why call them as axioms?

Paul replies: No - they will be a set of proposed axioms. They will not be fallible. I've added a new axiom, "Your sources of knowledge are fallible and incomplete". Is that true? I think so, and I expect you do to. It's not meant to force you to do anything other than think what that means.

If requirements are fallible, what does that mean to us? It depends... One ‘school’ takes the view that every effort should be made to get requirements right (highly detailed requirements, copious reviews, inspections and through coverage-based test planning and execution etc.). Another school says, go with it and the testing becomes both a requirements elicitation process and a validation of the delivered functionality.

Requirements are never perfect, complete etc. so the axiom I'm proposing simplty states that, and demands that you select an approach to deal with it. One context might be to try and perfect requirements as far as possible and plan a test ahead of time, another might be to start exploring. Each is a different heuristic to accommodate the particular context.

>> An axiom challenges you to think: "you had better think this through and choose your solution".

That is heuristic to me - not an axiom.

With specific reference to the comment made by Neil Thompson comparing your axioms, Bach/Bolton RST course material and Kippling --

Note that your axioms are "universally" true items where as Bach/Bolton's RST methodology is based on heuristics. That is key difference.

Shrini

If I take a homey definition of heuristic - a rule of thumb that might be appropriate in some contexts but not others. I'm not in the context-driven school, but I believe all testingis context-driven.

I pick and choose my heuristics depending on the context. "Your sources of knowledge are fallible and incomplete" this is a fact of life. "Acceptance is always a compromise" is another fact of life. I believe that is true for ALL projects and contexts.

So, in the context of a billion dollar defence software project or a 20 page web site for a small startup company, these are still true I suggest. It's the approach a project takes to these facts of life that vary. The defence contractor will use one set of heuristics; the part-time web site tester can take another. That's what I mean by your choice of heuristics.

Unfortunately, the poor wording of the proposed axioms has been a distraction.

Paul,

Thank you for the kind almost humbling reference.

I have taken a look through the axioms that you have defined, or is it discovered, and have observed that there has been a bit of energy in some blog posts around the word Axiom. I recall that at the UK Software Testing Retreat earlier in the month, in a soap-box slot, I asked the question of whether there were any immutable laws which governed software testing. As a thought experiment I had tried to define some but every time I thought I got near something, I could always make an opposing case. I think I ended the soap-box slot by using the line form the Pirates of the Caribbean, with reference to the Pirate Code, that these laws might be ‘more like guidelines’.

My thoughts on the Axioms are listed below:

1. The Stakeholder Axiom
There is an interesting distinction between the people who will use and benefit from the information and the people who are paying the bills and have control. I think some of the blog comments have missed this distinction. This is actually quite a subversive point. I used to say that I took information to the people it would worry the most. Very much like the environmental campaigners who have, as I write, climbed onto the roof of the House of Commons to make a protest about ever increasing CO2 emissions form aeroplanes.

2. The Test Basis Axiom
The ‘purist’ tester (and I understand all of the connotations of this term) would argue that without defining your expected results in advance of performing the test activity, that the activities you were performing were no testing, but a form of development. The world is not as clear cut as that. Particle physicists sometimes take a different approach. Because time in particle accelerators is so expensive and limited, and it takes so long to evaluate the results (thousands of years of processing time), the physicists can take a different approach. They run the accelerator, gather the data, then theorise about what interactions there may be, and only then go back and examine the output from the run. This is still testing. And in some ways analogous to regression testing and parallel runs.
I am very much a fan of the ‘Architecture as a Metaphor’ and wonder if there is scope for the Test Basis as a Metaphor?

3. The Test Oracle Axiom
I do have a problem with the word Oracle, as adopted by the software testing industry. The usage does not fit my understanding from the dictionary definition. I think this is another term that IT gleefully misuses.
The problem I see with the Oracle concept is that we don’t necessarily have all of the information to hand, however the term Oracle as used would imply that we do. Information builds up over time. We do not have very much to begin with, some of it may be wrong, etc. etc.
Your definition is good, in that information comes from many sources, but maybe the notion of completeness or confidence in the level of information in the Oracle may be useful.

4. The Scope Management Axiom
I think scope definition is important, and I think is linked to 9 The Prioritisation Axiom. Without accurate and comprehensive scope it will be difficult to prioritise.

5. The Coverage Axiom
I think this is also linked to the Scope Management Axiom, in that for each element of the scope, you need (should / could) to define some coverage targets. Now to some that is going to sound restrictive, but the benefits are that the early compromise in terms of acceptance criteria and identification of when to stop testing are greatly aided.

6. The Delivery Axiom
I think that this is the very least that you could say about evidencing the testing. In some areas there are actually legal, regulatory, compliance, or at the very least audit requirements for doing so. It is a good thing. The rest of the world also thinks it is a good thing.

7. The Environment Axiom
Would you consider that tools also be included in this axiom?

8. The Event Axiom
I am not quite clear what this axiom covers. Is this just communication, which is vital, or is this more?

9. The Prioritisation Axiom
See comments from above in terms of Scope Management. Misquoting the line form the book 1984 (Orwell), ‘If some Axioms are more important than others, then this would be one of them’.

10. The Execution Sequencing Axiom
Again to misquote Orwell, ‘some of our test cases are more important than others’. Is there anything wrong with the word planning, or has that fallen out of favour?

11. The Design Axiom
Would this model cover more than just test cases design and the techniques used. Does this also cover which levels of testing to carry out? What lifecycle to follow?

12. The Repeat Test Axiom
I concur, although re-testing and regression testing are different. Is there any subtlety here? I used to think that the purpose of testing was to deliver a regression test pack for future releases, because inherent in the nature of its production, would have been the testing of system or application.

I have lots of questions which don’t quite seem to fit into the Axiom mould. Here are some:

• We all believe that trained people do better testing than untrained people. Did you give any thought to capability in putting together the Axioms?

• You haven’t implied any sequencing for the Axioms. Was that a consideration?

• You are well known for your work on Risk (with Neil Thompson) and I notice that you didn’t mention Risk in the Axioms. What was the thinking behind that?

I hope these comments help with what I consider an excellent initial start at thinking around this contentious topic. I am proud to think that I may have influenced you on this journey.

Thanks
Graham

Paul Replies

• We all believe that trained people do better testing than untrained people. Did you give any thought to capability in putting together the Axioms?

no. :-)

• You haven’t implied any sequencing for the Axioms. Was that a consideration?

none - I would like them to be context-independent, sequence independent

• You are well known for your work on Risk (with Neil Thompson) and I notice that you didn’t mention Risk in the Axioms. What was the thinking behind that?

Using risk as a mechanism for scoping, prioritising tests is a choice - so no - it's not an axiom.

I'll try and discuss your comments more fully when I post a revised set of axiom defnitioins.

Paul