Hi Paul,

Good initiative! Here is some constructive feedback on the 3rd axiom (Test Oracle) of which I had some thoughts recently:

Doesn't this depend on the goal of your test activities. Most testing is indeed done to judge if certain behaviour is good or bad. But what if the goal is just to measure (e.g. performance). Not to see if it at least can handle a certain processing rate or does not exceed a pre-defined memory usage, but more to see what the value currently is. Some people (e.g. system architects) may be interested in these numbers, just to see how much room for development there is for next releases.

Paul Replies...
Yes, you have a point. It's common for non-functional requirements to be a bit 'thin on the ground' and in fact maybe half the time performance testing isn't requirements driven, as you say.

Let me think this through...

If there are no requirements or performance test objectives at all, the first stage is for the tester to create at least some load profiles. So work begins on getting some scenarios, transaction rates, database volumes etc. together. Let's call these load profiles. These need to be agreed by ones stakeholders as a reasonable basis for load simulations.

Then one needs to agree how one might vary loads around the 'design' average or peaks... Typically one creates a range of loads building up to design loads and beyond to provide a set of response time measurements for selected transactions against varying load. These can be presented as load-response time graphs.

Now, what is the required response time? Well, if they aren't defined ahead of time, I would propose some maximum acceptable response times. Finding this time on the graphs, we can then suggest maximum acceptable loads.

So now, we can discuss load profiles and anticipated response times with stakeholders. They might be lucky and it looks great - or unlucky. But at least now there's some hard data available for discussion.

Is there an 'oracle' - I think I would say that the forum convened to discuss the results are 'the oracle'. Ive worded the oracle axiom as, "Define the sources of knowledge whether documented, physical, experience or hearsay-based to be used to determine expected behaviour." With performance testing maybe it should read "... to be used to compare actual behaviour against"?

So, the oracle isn't necessarily used to predict the behaviour but could be if you have one in time ;-).

Now, as you say, architects might want to see how their beloved infrastructure performs. Presumably, they have some performance objectives to test against - otherwise what parameters did they use to create the architecture in the first place? (Did they just guess?) Anyway, if they want to simply find out what their architecture is capable of, I would call that a benchmarking or profiling exercise - so I would finesse that situation by saying, it's not really a test :-)

What do you think?

Hello Paul,
It feels like I volunteer on your journey to define Test Axioms. And I'm curious were it will end.

To understand your Axioms I tried to figure out some kind of structure like an organizational structure or test method structure. Hoping this helped me to identify some other Axioms.

I think all kinds of models are in these Axioms defined. Perhaps that is mandatory to create a list like this to prevent this list is guided by one model and input from others is forgotten.

Related to you 15th Axiom: You have also the opposite: Axiom Name: Never Exists Axiom: Testing never exists; it starts. Action: Acknowledge the moment you start with testing. This is the moment you have to question about other Axioms. If you don’t: starts testing you just do something. And what is not started cannot be stopped.

Somehow I’m missing a relation towards the team. As if no team is there no testing can be performed. If no skilled team is there axiom-names like: coverage and good-enough are answered differently.

I hope this give you some further input for your attempt.

With regards,
Jeroen

Paul replies...
The axioms aren't settled yet of course - but I do believe they are 'above' pre-defined approaches. That is, they are worth considering very early to help you understand the best way to approach testing. If the axioms are real, then they are true for any context - that's what I'm suggesting.

With regards you your suggested axiom... I'm not sure I understand. I think you are saying that the axioms need to be considered before you start (to even plan?) testing. I agree with that. I think this might be the value of the axioms as a set of challenges to your thinking about test. Maybe there's a way of wording it very precisely. I'll have to think about that one!