Manual vs. Automated Testing

While reading about Manual vs Automated Testing, I came across a good article by "James Whittaker". The title of post is "manual v. automated testing again". The source and article is mentioned below:

Source:
http://blogs.msdn.com/b/james_whittaker/archive/2008/10/29/manual-v-automated-testing-again.aspx


In my Future series I was accused of supporting both sides of the manual v. automated debate and flip-flopping like an American politician who can’t decide whether to kiss the babies or their moms. Clearly this is not an either-or proposition. But I wanted to supply some clarity in how I think about this.

This is a debate about when to choose one over the other and which scenarios one can expect manual testing to outperform automated testing and vice versa. I think the simplistic view that automation is better at regression and API testing and manual is better for acceptance and GUI testing diverts us from the real issues.

I think the reality of the problem has nothing to do with APIs or GUIs, regression or functional. We have to start thinking about our code in terms of business logic code or infrastructure code. Because that is the same divide that separates manual and automated testing.

Business logic code is the code that produces the results that stakeholders/users buy the product for. It’s the code that gets the job done. Infrastructure code is the code that makes the business logic work in its intended environment. Infrastructure code makes the business logic multiuser, secure, localized and so forth. It’s the platform goo that makes the business logic into a real application.

Obviously, both types of code need to be tested. Intuitively, manual testing should be better at testing business logic because the business logic rules are easier for a human to learn than they are to teach to a piece of automation. I think intuition is bang-on correct in this situation.

Manual testers excel at becoming domain experts and they can store very complex business logic in the most powerful testing tool around, their brains. Because manual testing is slow, testers have the time to watch for and analyze subtle business logic errors. Low speed but also low drag.

Automation, on the other hand, excels at low-level details. Automation can detect crashes, hangs, incorrect return values, error codes, tripped exceptions, memory usage and so forth. High speed but also high drag. Tuning automation to test business logic is very difficult and risky. In my humble hindsight I think that Vista got bit by this exact issue, depending so much on automation whereas a few more good manual testers would have been worth their weight in gold.

So whether you have an API or a GUI, regress or test fresh, the type of testing you choose depends on what type of bug you want to find. There may be special cases, but the majority of the time manual testing beats automated testing in finding business logic bugs and automated testing beats manual testing in finding infrastructure bugs.

There I go again, fishing in both sides of the pond.

Exploratory Testing

According to Wikipedia, "Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution". It is further stated that, "Cem Kaner, who coined the term in 1983, now defines exploratory testing as a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project".

Exploratory testing has got an important role in the industry which has grown much mature in the field of software testing. Although all testing to an extent is exploratory but there are various reasons under which exploratory testing becomes a requirement.

In my view, time constraint or short deadlines are important factors giving rise to the need of performing exploratory testing. It is due to short deadline that testers are asked by the management to perform the exploratory testing, as there is not much time to learn the application.

Hiring a testing resource at a critical time may also end up with the resource performing exploratory testing. If hire a new testing resource at a critical time in project, he is usually asked to start with testing the application as soon as he can. So, the tester starts with exploratory testing while also learning the application (by using the application or by reading the learning material).

Exploratory testing is also performed when there is an incomplete or no documentation available. Documentation may include the requirement specifications, user guides or any other learning material.

These are few common reasons and all these make exploratory testing a challenge for the software testers. The challenge for the software testing companies is to select the right person for this job.

Different testers have different skill set and are strong in different domains, platforms and tools. In order to select the right person for exploratory testing, any company should focus on various factors.

These factors may include the previous experience of a software tester. For example, it may not turn out to be a success story with one hires a fresh resource and asks him to perform exploratory testing of a large financial application, as the fresh resource may have difficulty in learning the application along with testing of it at same time. In comparison, if a company hires a resource with previous experience of testing financial applications then this resource can perform much better. Similarly, platforms and tools required to test should also be considered. Selecting the tester who has best match to the required domain, platform and tools, does improve the quality of exploratory testing.

This article does not cover all aspects of exploratory testing.

Reference:

http://en.wikipedia.org/wiki/Exploratory_testing

Defect Triage

The term "Triage" comes from the French verb trier, meaning to separate, sort, sift or select. Triage" is a medical term describing the process of determining the priority of patient’s treatment based on the severity of their condition. It refers to dividing wounded or sick people into three categories: those who will die no matter what you do, those who will recover even if unaided, and those who will recover only if aided. In a situation where there's too much to do, you must concentrate on the third group (Ex: Watch Pearl Harbor scene, showing situation in hospital after Japanese have bombed the harbor).

This term has been introduced to software quality assurance to describe the function of assigning a priority and severity to new defects, usually by addressing several new defects at once every few days or as necessary.

If your project is very small, or isolated from other business units, triage may not be very useful at all. Similar to an emergency room that receives one patient per hour. This is usually not the case with large business.

Triage in software development requires three parties be present. Project manager, development lead, and test lead must be present at this discussion. Test lead facilitates a bug triage with a bug tracking system. Test lead reads the description of the bug to everyone, product generally asks some clarifying questions, and testing and development go back and forth discussion the importance and impact of their point of view. All three parties come to agreement and understanding on the severity and priority, or “actionable priority”, of the bug or defect, and the defect is marked as Triaged. Consistent triage meetings are important to ensuring a clear product backlog and reducing the need for meetings to talk about priorities.

Triaging a bug involves making sure the bug has enough information for the developers and makes sense, bug is filed in the correct place, bug has sensible "Severity" and "Priority" fields.

Test to Break Attitude - Poor Concept

If you ask question that what makes a good software tester then most people reply that a good tester should have "Test to Break Attitude". Ex: “softwareqatest” website has mentioned "Test to Break Attitude" as first quality that software test engineer should have.

My point of view is different. For me, it is not the responsibility of software tester to break anything and neither does he break. It is the software developer that has left broken things for software testers to find it. If a tester finds a bug then it does not mean that he has broken anything, it was already broken. A tester just find outs the points and areas which are already broken and then digs deep in it.

Detecting a breakage is not a job all done for a tester? It’s a job half done and maybe it’s just the beginning. Further responsibilities of tester are the find out the conditions under which this break occur and under which does it not occur? And to figure out the exact and simple steps to reproduce this break, which are supported by images, presentations or clips etc. All this will help developers and other stake holders to reproduce the break at their end. Other questions that tester have to find out are: Scope: all areas effected by the break; after finding this break do we need to re-execute some of other test cases or what test cases to re-execute after getting the fix for break; what risks will arise due to break; was it possible to detect this break earlier, if yes then what factors avoided its earlier detection; should the testing process or strategy be modified to detect breaks earlier?

As a software tester, your job may include planning, conducting, and analyzing results in controlled conditions, which are designed to find any kind of breakages.
However, if you like to break then you may try to take up any demolition job.

Testing Glossary

A
Acceptance Testing
Testing conducted to enable a user/customer to determine whether to accept a software product. Normally performed to validate the software meets a set of agreed acceptance criteria.

Acceptance Criteria
The exit criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity. [IEEE]

Accessibility Testing
Verifying a product is accessible to the people having disabilities (deaf, blind, mentally disabled etc.).

Ad Hoc Testing
A testing phase where the tester tries to 'break' the system by randomly trying the system's functionality. Can include negative testing as well. See also Monkey Testing.

Agile Testing
Testing practice for projects using agile methodologies, treating development as the customer of testing and emphasizing a test-first design paradigm. See also Test Driven Development.

Alpha Testing
Testing conducted internally by the manufacturer, alpha testing takes a new product through a protocol of testing procedures to verify product functionality and capability. In-house testing. This is the period before Beta Testing. In-house testing performed by the test team.

Anomaly
Any condition that deviates from expectation based on requirements specifications, design documents, user documents, standards, et, or from someone's perception or experience. Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or use of software products or applicable documentation. [IEEE]

Challenges of Software Testing

According to the book, Surviving the Top Ten Challenges of Software Testing, A People-Oriented Approach by William Perry and Randall Rice, the top ten people challenges have been identified as:
1)  Training in testing
2)  Relationship building with developers
3)  Using tools
4)  Getting managers to understand testing
5)  Communicating with users about testing
6)  Making the necessary time for testing
7)  Testing “over the wall” software
8)  Trying to hit a moving target
9)  Fighting a lose-lose situation
10) Having to say “no”

Self-assessment can be performed to identify our own strengths & weaknesses and then build a plan for improvement.

Essential testing skills include test planning, using test tools (automated and manual), executing tests, managing defects, risk analysis, test measurement, designing a test environment, and designing effective test cases. Additionally, a solid vocabulary of  testing is essential. A tester needs to understand what to test, who performs what type of test, when testing should be performed, how to actually perform the test,
and when to stop testing.

Testing at Ad Hoc Level Organizations

Ad hoc means unstructured, inconsistent levels of performance. If any company is at the ad hoc level, tasks are not performed the same way by different people or different groups. For example, one system development group may use part of the system development methodology, but improvise other parts; another group may select different parts of the same system development methodology to use, and decide not to perform tasks done by a previous group.
At this level, management manages people and jobs. Management will establish goals or objectives for individuals and teams, and manage to those objectives and goals with minimal concern about the means used to achieve the goals. This level is normally heavily schedule driven, and those that meet the schedules are rewarded. Since there are not any standards against which to measure deliverables, people’s performance is often dependent upon their ability to convince management that the job they have done is excellent. This causes the environment to be very political. Both management and staff become more concerned with their personal agenda than with meeting their organization’s mission.
In such case, emphasis needs to be on discipline and control. The emphasis should be on getting the work processes defined, training the people in the work processes, implementing sufficient controls to assure compliance to the work processes, and producing products that meet predefined standards.

Software testers face a much greater challenge testing software developed by low maturity level, than they do by testing software developed by higher maturity levels. At low maturity level, testing and rework will consume more than 50% of the total software development effort. As software development processes mature, two things happen: more testing occurs during the building of software and the amount of testing required is reduced.