Testing Types
Functionality Testing:
During unit testing each component or object is tested for completeness and correctness of implementation of a specific feature of functionality. In integration testing we focus on functionality that requires the correct operation of two or more components or a flow of data between them.
During system testing we should consider functionality in terms of whole sequence of end user operations or an entire area of functionality. In functionality testing we should consider "right action", "wrong side effect" risks and also test with both valid and invalid data sets.
User Interface Testing
User interface of a project is the first thing a user encounters in a product. While the product may perform its intended function correctly, if the UI behaves incorrectly, does not refresh properly or over writes meaningful text on the screen inadvertently the user can be seriously annoyed.
Testing the user interface of a product usually consist of the operation or simulation of keyboard, typing and mouse clicking and movement to do menu selection, field entry, field motion, control operation, request confirmation and so on.
User interface testing during the unit testing phase often involves only a select few UI objects or specific screen or page. During integration testing we need to focus on navigation/page flow across several screens/pages. In system testing we need to test the complete navigation required to meet client requirements. In addition to testing the normal navigation with reference to various types of users, it is necessary also to test for negative behavior.
Concurrent Execution Testing
Although a product may appear to function properly when tested by a single user, when multiple instances of a product run concurrently undesirable results may occur: they may show up in the application or in the operating system. A product that can have many instances of it running at the same time in parallel should undergo these concurrent execution tests:
• Simple usage (Two users)
• Standard usage (many users)
• Boundary situations (maximum number of users plus or minus one) if this limit exists
Multiple Configuration Testing
A product may be usually tested under a default configuration, it is important to test the non-default configurations also. Sometimes new tests may have to be written to test under a different configuration. It is possible that there could be infinite possible configurations that can be set up. It is necessary to identify key configurations that are important for the customer to operate the product and test the product for these configurations
Dependency Testing
Dependency testing is recommended to test any API calls made to other products and ensure that these calls act as promised. Such tests also provide a good regression mechanism when the product being developed is integrated with the new version of products it depends on. We need to test all interactions between products, including error cases. In general, any where data or control is transferred from one component to another component (or components), either immediately or in delayed fashion, an interface exists that can cause trouble.
Forward/Backward Compatibility Testing:
The product must be compatible (to the published extent) with any previous release and supported dependencies. Backward compatibility testing ensures that no problem will arise from a user trying to use the new version of a product along with an old version and verifies integration between products.
Installation and Upgrade Testing
If a user cannot install or upgrade the product whether it is through an inability to understand documentation, a failure of the installation/upgrade program, or lack of resource, the product has failed the user.
We need to check the installation/upgrade documentation for accuracy and follow all instructions form the point of view of a naïve user. Test the entire installation and upgrade procedure, both with correct and incorrect response to all prompts. If verification scripts are provided to test the installation/upgrade, run the script as an additional check. Generate possible system error to check the response of the installation/upgrade program.
User Documentation Testing
The user documentation is part of the product. If a new user is unable to use the software because of poor documentation, the product as a whole is unusable. All sections of the user documentation should be tested for accuracy against the reality of the product. Ideally user documentation should be held under source control and made part of regression testing of the product. Verify as corrected and complete all demos, tutorials and exercise described in the user documentation.
Standards Conformance Testing
Specific set of industry standards has to be tested. Tests need to be planned and conducted to detect bugs in standards conformance that could seriously affect the prospects of the product in the market.
Capacity and Volume Testing
During unit testing, from a structural test perspective, every buffer, occurrence, storage resource, processor, bus, and I/O channel in the system has to be tested. In integration testing aspects related to capacity of network to handle traffic and its behavior under various volumes are tested. Similarly the data storage capability is tested. During system testing the capacity and volume limitations are tested from a user's point of view.
Localization
Localization typically refers to problems associated with different languages. Languages other than English may require different character sets in which data can be represented. Testing character set conversion for data conversion between different localization is a very important consideration for planning localization testing. Another important item is the provision of different message files in various languages for error messages and other system information. Testing for localization may require setting up special hardware and software.
Security Testing
Security testing attempts to verify that protection mechanism built into a system will, in fact protect it from improper penetration. During security testing the tester plays the role of the individual who desires to penetrate the system. The tester would attempt to acquire passwords, access the system through some indirect/direct means, overwhelm the system so that it is not available to others, cause system errors, etc.
Performance Testing
Any product, no matter how well constructed and reliable, is worthless if its performance does not meet the demands of its users. Testing must be done in realistic setting to ensure adequate performance. However, we have to be cost effective.
We need to test how the product performs during operations normally executed by the users regularly. If performance goals are set for various operations of the product, it is necessary to verify that these goals are met. Performance is not only "How many per second" but also" how long".
Smoke Testing
Smoke testing is an integration testing approach that is commonly used when "Shrink wrapped" software products are being developed. It is designed as a pacing mechanism for time-critical projects, allowing the software team to assess its projects on a frequent basis.
Software components produced are integrated into a "build". A series of tests are designed to expose errors that will keep the build from properly performing its function. The intent is to uncover "Show stopper" errors that have the highest likelihood of throwing the software project behind schedule. Sometimes daily builds are made and subjected to smoke testing.
The smoke test should exercise the entire system from end to end. It does not have to be exhaustive, but it should be capable of exposing major problems. The smoke test should be thorough enough that if the build passes you can assume that it is stable enough to be tested more thoroughly.
Regression Testing
Regression testing is the re-execution of some subset or all tests that have been previously conducted to ensure that changes have not propagated unintended side effects. Regression testing is the activity that helps to ensure that changes do not introduce unintended behavior or additional errors. When sub sets of test are selected for regression testing, they should contain:
• A representative sample of test that will exercise all software functions
• Additional tests that focus on software functions that are likely to be affected by the change
• Tests that focus on the software components that have been changed.
Networks and Distributed Environments
If a product works in a network or distributed environment we need to test for
• Date and time zone handling
• Communication and telephony standards
Environment factors (temperature, pressure, etc) and their impact on the product should also be tested if required.
Error / Disaster Handling and Recovery
A software system must recover from faults and resume processing within a prescribed time. In some cases, a system must be fault tolerant: that is, processing faults must not cause overall system function to cease. In other cases, a system failure must be corrected within a specific period of time.
Recovery testing forces the software to fail in a variety of ways and verifies that recovery is properly performed. If recovery is automatic then re-initialization, check pointing mechanisms, data recovery and restart are evaluated for correctness. If recovery requires human intervention, the mean-time -to -repair (MTTR) is evaluated to determine whether it is with in acceptable limits.
Quality risks in this area include unacceptable failure rates, unacceptable recovery times and the inability of the system to function under legitimate conditions without failure.
Alpha and Beta Testing
If software is developed as a product to be used by many customers, it is impractical to perform formal acceptance tests with each one. Most software product builders use a process called alpha and beta testing to uncover errors that only the end user seem to be able to find.
A customer conducts the alpha test at the developer site. The software is used in a natural setting with the developer recording the errors and usage problems. Alpha tests are conducted in a controlled environment.
The beta test is conducted at one or more customer sites by the end-user of the software unlike alpha testing the developer is not present. Beta test is conducted at live sites not controlled by the developer. The customer records all problems encountered during beta and reports them to the developer
Stress Testing
Stress testing is the search for load dependent bugs that occur only when the system is stressed and the specification of stress tests is a key activity. Stress Testing increases the overall quality and reliability of software. Stress-related bugs are the most difficult to find and exhume.
In stress testing, the system is subjected to unrealistically harsh inputs or loads with inadequate resources, with the intention of breaking it. We test to find out weather the system can operate reliably at the limit of its resources or not.
Stress testing with high background load, to a point where one or more or all resources are simultaneously saturated. It is to system testing what destructive testing is to physical objects.
Stress testing often finds bugs for the following reasons:
• It forces race conditions
(Race conditions reflect temporal boundaries. Normally event A occurs before event B; By chance if event B occurs before A, then the system may fail, giving a race condition bug)
• It totally distorts the normal order of processing, especially processing that occurs at different priority levels.
• It forces the exercise of all system limits, thresholds, or other controls designed to deal with overload situations.
• It greatly increases the number of simultaneous actions.
• It depletes resource pools in extraordinary and unexpected sequences.
Some of the reasons why stress-related bugs are encountered in system are:
• Assumptions that there will be no interrupts.
• Failure to block or unblock interrupts.
• Assumptions that code is reentrant or not reentrant.
• By passing data interlock.
• Failure to close or open interlock.
• Assumptions that a called routine is resident or non-resident.
• Assumption that the calling program is resident or non-resident.
• Assumption that registers or memory were initialized or not initialized.
• Assumptions that the registers or memory location content did not change.
• Local setting of global parameter.
• Global setting of local parameter.
Thursday, August 16, 2007
PERFORMANCE TESTING
Scripting Language used for Performance Testing is VU Scripting
Cookies
Cookie is a temporary file stores the session details
set_cookie Emulation Function
Adds a cookie to the cookie cache.
SYNTAX: set_cookie(name, value, domain, path [, secure])
Syntax Element Description
name A string expression that specifies the name of the cookie.
value A string expression that specifies the value for the cookie.
domain A string expression that specifies the domain for which this cookie is valid.
path A string expression that specifies the path for which this cookie is valid.
secure An optional string expression that, if given, provides the secure modifier for the cookie. The value of this parameter should be "secure".
COMMENTS
The set_cookie function creates the named cookie with the given value. If a cookie already exists with this name for the given domain and path then set_cookie() sets the value of that cookie to value.
The expiration date of the cookie is set sufficiently in the future that it will not expire during the run.
Dynamic Data Correlation
Dynamic data correlation is a technique to supply variable data values to a script when the transactions in a script depend on values supplied from the server.
For example, when you record an http script, the Web server may send back a unique string, or session ID, to your browser. The next time your browser makes a request, it must send back the same session ID to authenticate itself with the server.
The session ID can be stored in three places:
• In the Cookie field of the HTTP header.
• In an arbitrarily named field of the HTTP header.
• In an arbitrary hidden field in an actual HTML page.
Rational Suite TestStudio finds the session IDs (and other correlated variables) and, when you run the suite, automatically generates the proper script commands to extract their actual values.
Before you record a script, you can choose whether TestStudio correlates all possible values (the default), does not correlate any values, or correlates only a specific list of variables that you provide. For Information about correlating your data, see:
Identifying data that needs to be correlated
Coding the Script to Correlate the Data
Although varying test values may work for those transactions that depend on the result of an earlier transaction, other transactions may depend on values received from the server. If a script contains these transactions, you must manually edit the script to replace some of the missing client logic so that the values correlate dynamically. This is called dynamic data correlation.
Cookies
Cookie is a temporary file stores the session details
set_cookie Emulation Function
Adds a cookie to the cookie cache.
SYNTAX: set_cookie(name, value, domain, path [, secure])
Syntax Element Description
name A string expression that specifies the name of the cookie.
value A string expression that specifies the value for the cookie.
domain A string expression that specifies the domain for which this cookie is valid.
path A string expression that specifies the path for which this cookie is valid.
secure An optional string expression that, if given, provides the secure modifier for the cookie. The value of this parameter should be "secure".
COMMENTS
The set_cookie function creates the named cookie with the given value. If a cookie already exists with this name for the given domain and path then set_cookie() sets the value of that cookie to value.
The expiration date of the cookie is set sufficiently in the future that it will not expire during the run.
Dynamic Data Correlation
Dynamic data correlation is a technique to supply variable data values to a script when the transactions in a script depend on values supplied from the server.
For example, when you record an http script, the Web server may send back a unique string, or session ID, to your browser. The next time your browser makes a request, it must send back the same session ID to authenticate itself with the server.
The session ID can be stored in three places:
• In the Cookie field of the HTTP header.
• In an arbitrarily named field of the HTTP header.
• In an arbitrary hidden field in an actual HTML page.
Rational Suite TestStudio finds the session IDs (and other correlated variables) and, when you run the suite, automatically generates the proper script commands to extract their actual values.
Before you record a script, you can choose whether TestStudio correlates all possible values (the default), does not correlate any values, or correlates only a specific list of variables that you provide. For Information about correlating your data, see:
Identifying data that needs to be correlated
Coding the Script to Correlate the Data
Although varying test values may work for those transactions that depend on the result of an earlier transaction, other transactions may depend on values received from the server. If a script contains these transactions, you must manually edit the script to replace some of the missing client logic so that the values correlate dynamically. This is called dynamic data correlation.
Monday, May 21, 2007
Guidelines on Testing
Different types of testing :
Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.
Unit testing: the most 'micro' scale of testing; to test particular functions or code modules.
Integration testing - Testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server
Functional testing - Black-box type testing geared to functional requirements of an application; This type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)
Regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing. Load Testing
Acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.
Load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails.
Recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
Security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques
Alpha testing - testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.
Beta testing - testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
What's a 'test plan”?
A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The following are some of the items that might be included in a test plan, depending on the particular project:
• Title
• Objective of testing effort
• Software product overview
• Relevant related document list, other test plans, etc.
• Relevant standards or legal requirements
• Overall software project organization and personnel/contact-info/responsibilties
• Testing priorities and focus
• Scope and limitations of testing
• Test outline - a test approach by test type, feature, functionality, module, etc. as applicable
• Test environment validity analysis - differences between the test and production systems and their impact on test validity.
• Test tools to be used, including versions, patches, etc.
• Test script/test code maintenance processes and version control
• Problem tracking and resolution - tools and processes
• Reporting requirements and testing deliverables
• Software entrance and exit criteria
• Test site/location
What's a 'test case'?
• A test case is a document that describes an input, action, or event and an expected response, to determine if a feature of an application is working correctly. A test case should contain particulars such as test case identifier, test case name, objective, test conditions/setup, input data requirements, steps, and expected results.
What should be done after a bug is found?
The bug needs to be communicated and assigned to developers that can fix it. After the problem is resolved, fixes should be re-tested, and determinations made regarding requirements for regression testing to check that fixes didn't create problems
What steps are needed to develop and run software tests?
• Obtain requirements, functional design, and internal design specifications and other necessary documents.
• Determine project-related personnel and their responsibilities, reporting requirements, required standards and processes
• Identify application's higher-risk aspects, set priorities, and determine scope and limitations of tests
• Determine test approaches and methods - unit, integration, functional, system, load, usability tests, etc.
• Determine testware requirements(bug tracking, etc.)
• Determine test input data requirements
• Identify tasks, those responsible for tasks, and labor requirements
• Set schedule estimates, timelines, milestones
• Prepare test plan document and have needed reviews/approvals
• Write test cases
• Have needed reviews/inspections/approvals of test cases
• obtain needed user manuals/reference documents/configuration guides/installation guides, set up test tracking processes, Obtain and install software releases
• Perform tests
• Evaluate and report results
• Track problems/bugs and fixes
• Retest as needed
• Maintain and update test plans, test cases, test environment, and testware through life cycle
How to Enter Bug – Registry
• Complete information such that developers can understand the bug, get an idea of it’s severity, and reproduce it if necessary.
• Current bug status
• The application name
• The function, module, feature, object, screen, etc. where the bug occurred
• Full bug description
• Description of steps needed to reproduce the bug if not covered by a test case or if the developer doesn't have easy access to the test case
• File or the data that would be helpful in finding the cause of the problem
• Tester name
• Test date
• Bug reporting date
• Name of developer/group/organization the problem is assigned to
• Description of problem cause
• Description of fix
• Date of fix
• Retest date
• Retest results
• Regression testing results
A reporting or tracking process should enable notification of appropriate personnel at various stages. For instance, testers need to know when retesting is needed, developers need to know when bugs are found and how to get the needed information, and reporting/summary capabilities are needed for managers.
Note: Tester has to check-in the Bug Registry at the end of the day and must see to that all the bugs recorded in the registry is debugged
Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.
Unit testing: the most 'micro' scale of testing; to test particular functions or code modules.
Integration testing - Testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server
Functional testing - Black-box type testing geared to functional requirements of an application; This type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)
Regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing. Load Testing
Acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.
Load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails.
Recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
Security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques
Alpha testing - testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.
Beta testing - testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
What's a 'test plan”?
A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The following are some of the items that might be included in a test plan, depending on the particular project:
• Title
• Objective of testing effort
• Software product overview
• Relevant related document list, other test plans, etc.
• Relevant standards or legal requirements
• Overall software project organization and personnel/contact-info/responsibilties
• Testing priorities and focus
• Scope and limitations of testing
• Test outline - a test approach by test type, feature, functionality, module, etc. as applicable
• Test environment validity analysis - differences between the test and production systems and their impact on test validity.
• Test tools to be used, including versions, patches, etc.
• Test script/test code maintenance processes and version control
• Problem tracking and resolution - tools and processes
• Reporting requirements and testing deliverables
• Software entrance and exit criteria
• Test site/location
What's a 'test case'?
• A test case is a document that describes an input, action, or event and an expected response, to determine if a feature of an application is working correctly. A test case should contain particulars such as test case identifier, test case name, objective, test conditions/setup, input data requirements, steps, and expected results.
What should be done after a bug is found?
The bug needs to be communicated and assigned to developers that can fix it. After the problem is resolved, fixes should be re-tested, and determinations made regarding requirements for regression testing to check that fixes didn't create problems
What steps are needed to develop and run software tests?
• Obtain requirements, functional design, and internal design specifications and other necessary documents.
• Determine project-related personnel and their responsibilities, reporting requirements, required standards and processes
• Identify application's higher-risk aspects, set priorities, and determine scope and limitations of tests
• Determine test approaches and methods - unit, integration, functional, system, load, usability tests, etc.
• Determine testware requirements(bug tracking, etc.)
• Determine test input data requirements
• Identify tasks, those responsible for tasks, and labor requirements
• Set schedule estimates, timelines, milestones
• Prepare test plan document and have needed reviews/approvals
• Write test cases
• Have needed reviews/inspections/approvals of test cases
• obtain needed user manuals/reference documents/configuration guides/installation guides, set up test tracking processes, Obtain and install software releases
• Perform tests
• Evaluate and report results
• Track problems/bugs and fixes
• Retest as needed
• Maintain and update test plans, test cases, test environment, and testware through life cycle
How to Enter Bug – Registry
• Complete information such that developers can understand the bug, get an idea of it’s severity, and reproduce it if necessary.
• Current bug status
• The application name
• The function, module, feature, object, screen, etc. where the bug occurred
• Full bug description
• Description of steps needed to reproduce the bug if not covered by a test case or if the developer doesn't have easy access to the test case
• File or the data that would be helpful in finding the cause of the problem
• Tester name
• Test date
• Bug reporting date
• Name of developer/group/organization the problem is assigned to
• Description of problem cause
• Description of fix
• Date of fix
• Retest date
• Retest results
• Regression testing results
A reporting or tracking process should enable notification of appropriate personnel at various stages. For instance, testers need to know when retesting is needed, developers need to know when bugs are found and how to get the needed information, and reporting/summary capabilities are needed for managers.
Note: Tester has to check-in the Bug Registry at the end of the day and must see to that all the bugs recorded in the registry is debugged
Important Testing Checklist
Software Testingis the Process of
Systematically Running a Software System Or Program
To uncover errors or defects
Software Testing in Organizations aiming for SEI Level 3 +:
•An acknowledged competitive advantage contributing to business success
•Moved from ad hoc to an engineering discipline
•Carrier for faster product delivery, with version controlled test plans and test cases
•Led to reliability prediction through defect metrics
•Focussed on defect elimination at the stage of defect induction itself
Testing Phases:
–Unit Testing
–Integration Testing
–System Testing
–Acceptance Testing
All Test Plans and Test Cases shall be Reviewed and Approved before Testing starts.
Unit Testing
–Lowest-level component test–Key foundation for later levels of testing
–Detects 65% - 75% of all bugs
–Stand-alone test
–Ensures the conformance to each unit in DDD
Integration Testing
–An incremental series of tests of combinations or subassemblies of selected components in an overall system
–Integration testing is incremental in that successively larger and more complex combinations of components are tested in sequence, proceeding from the unit level to eventually the full-integrated system
–Ensures the conformance to each module in HLDD
System Testing
–Highest level of application functionality testing performed by the systems group
–Ensures the conformance to the Functional Requirements as specified in the SRS
Acceptance Testing
–Independent test performed by the users or QA prior to accepting the delivered system
–Ensures the conformance to the Functional Requirements as specified in the URD
Testing Activities
•Test Case Design
•Review of Test Case Design
•Testing
•Recording Testing Results
•Review and Sign-Off of Testing Results
•Defect Reporting
Testing Documents and Records
Documents (SCIs)
Test Plan for the Project
Test Cases Documents
Records
Test Report
Defect Log
Review Reports
Unit Test Plan (UTP)
•Plan for the entire Unit Testing in the project
•Identifies the Units/Sub-units covered during Unit Testing
•Each feature in the DDD mapped to a Unit Test Case ID
•To be prepared at DDD phase itself
•The test cases are designed and documented •References to the Test Case document(s) shall be given in UTP
•Deliverables at Unit Testing Phase shall be identified in UTP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
Testing coverage shall be identified in the UTP:
•Path Coverage
•Statement Coverage
•Decision (Logic/Branch) Coverage
•Condition Coverage
•Decision/Condition Coverage
•Multiple-Condition Coverage
•Functionality testing
•User interface testing
•Regression testing
Unit Testing CoveragePath Coverage
-Test cases will be written to cover all the possible paths of control flow through the program.
Statement Coverage
–Test cases will be written such that every statement in the program is executed at least once
Decision (Logic/Branch) Coverage
–Test Cases will be written such that each decision has a true or false outcome at least once.
Condition Coverage
–Test Cases will be written such that each condition in a decision takes on all possible outcomes at least once.
Decision/Condition Coverage
–Test Cases will be written such that each condition in a decision takes on all possible outcomes at least once, each decision takes on all possible outcomes at least once, and each point of entry is invoked at least once.
Multiple-Condition Coverage
–Test Cases will be written such that all possible combinations of condition outcomes in each decision, and all points of entry, are invoked at least once.
Integration Test Plan (ITP)
•Plan for the entire Integration Testing in the project
•Identifies the Modules/Sub-Modules covered during Integration Testing
•Each component of HLDD mapped to a Integration Test Case ID
•Recommended to prepare at HLD phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in ITP
•Deliverables at Integration Testing Phase shall be identified in ITP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
Testing coverage shall be identified in the ITP:
•Functionality Testing•User interface testing
•Dependency (API) testing
•Smoke testing
•Capacity and volume testing
•Error / disaster handling and recovery
•Concurrent execution testing
•Equivalence partitioning
•Boundary-value analysis
•Cause-effect graphing
•Error guessing
•Equivalence Partitioning
•The test cases are partitioned into equivalence classes such that
–each test case should invoke as many different input conditions as possible in order to minimize the total number of test cases necessary
–if one test case in an equivalence class detects an error, all other test cases in the equivalence class would be expected to find the same error
Boundary-value analysis
–Test cases explore boundary conditions
–Test cases consider input conditions as well as output conditions
Cause-effect graphing
–Technique to identify test cases by translating the specifications into Boolean logic network.
–This is a systematic method of generating test cases representing combinations of conditions
Error guessing
–The test cases are written both by intuition and experience to uncover certain probable type of errors
–The test case design is by the knack of “smelling out” errors
System Test Plan (STP)
•Plan for the entire System Testing in the project
•Identifies the features covered during System Testing
•Each feature in the SRS mapped to a System Test Case ID
•Recommended to prepare at SRS phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in STP
•Deliverables at System Testing Phase shall be identified in STP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
•Functionality Testing
•User interface testing
•Usability testing
•Volume Testing
•Stress Testing
•Security Testing
•Performance Testing
•Installation and upgrade testing
•Standards conformance testing
•Configuration testing
•Network and distributed environment testing
•Forward / backward compatibility testing
•Reliability Testing
•Error / disaster handling and recovery testing
•Serviceability Testing
•Documentation testing
•Procedure testing
•Localization testing
Acceptance Test Plan (ATP)
•Plan for the entire Acceptance Testing in the project
•Identifies the features covered during Acceptance Testing
•Each feature in the URD mapped to a Acceptance Test Case ID
•Recommended to prepare at URD phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in ATP
•Deliverables at Acceptance Testing Phase shall be identified in ATP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
•Alpha Test•Beta Test
•All the tests covered under system testing can be repeated as per user requirements
Terminology
Functionality Testing
–Functionality testing is the determination of whether each functionality mentioned in SRS is actually implemented. The objective is to ensure that all the functional requirements as documented in the SRS are accomplished.
User Interface Testing
–Focus is on testing the user interface, navigation and negative user behavior
Concurrent Execution Testing
–Focus is on testing with simple usage, standard usage and boundary situations
Volume Testing
–Volume Testing is to ensure that the software
•can handle the volume of data as specified in the SRS
•does not crash with heavy volumes of data, but gives an appropriate message and/or makes a clean exit.
Example:
•A compiler would be fed an absurdly large source program to compile
Stress Testing
–Executes the software in a manner that demands resources in abnormal quantity, frequency and volume and verifies that the system either performs normally, or displays message regarding the limitations of the system
Usability Testing
–Usability testing is an attempt to uncover the software usability problems involving the human-factor
Example:
•Are the error messages meaningful, easy to understand?
•Security Testing
–Attempts to verify that protection mechanism built into a system will in fact protect it from improper penetration
Example:
•a database management system's data security mechanisms
•Performance Testing
–Tests the run-time performance of software within the context of an integrated system
Example:
•the response times under certain configuration conditions.Installation and Upgrade Testing
–Focus is on testing whether the user would be install and upgrade the system
Configuration Testing
–Configuration testing includes either or both of the following:
•testing the software with the different possible hardware configurations
•testing each possible configuration of the software
Network and Distributed Environment Testing
–Focus is on testing the product in the required network and distributed environment
Dependency (API) Testing
–Focus is on testing the API calls made by the system to other systems
Localization Testing
–Focus is on testing problems associated with multiple languages support, conversion and also hardware aspects
Reliability Testing
–The various software testing processes have the goal to test the software reliability. The "Reliability Testing" which is a part of System Testing encompasses the testing of any specific reliability factors that are stated explicitly in the SRS
Error / Disaster Handling and Recovery Testing
–Forces the software to fail in a variety of ways and verifies that the system recovers and resumes processing
Serviceability Testing
–Serviceability testing covers the serviceability or maintainability characteristics of the software
Example:
•service aids to be provided with the system, e.g., storage-dump programs, diagnostic programs
•the maintenance procedures for the system
Documentation Testing
–Documentation testing is concerned with the accuracy of the user documentation. This involves
•Review of the user documentation for accuracy and clarity
•Testing the examples illustrated in the user documentation by preparing test cases on the basis of these examples and testing the system
Smoke Testing
–Focus is on testing the system releases as it is being built to uncover critical and showstopper errors
Configuration Testing
–Configuration testing includes either or both of the following:
•testing the software with the different possible hardware configurations
•testing each possible configuration of the software
Procedure Testing
–If the software forms a part of a large and not completely automated system, the interfaces of the developed software with the other components in the larger system shall be tested. These may include procedures to be followed by
•The human operator
•Database administrator
•Terminal user
–These procedures are to be tested as part of System testing
•Standards Conformance Testing–Focus is on testing whether the product conforms to prescribed and published standards
•Alpha Test–Within a vendor, the last level of internal test prior to a limited beta release
•Beta Test
–The vendor’s equivalent of a pilot test (usually with a greater number of participating sites than for a pilot)
•Alpha Test–Within a vendor, the last level of internal test prior to a limited beta release
•Beta Test
–The vendor’s equivalent of a pilot test (usually with a greater number of participating sites than for a pilot)
•Equivalence Test
–A test using only a small sample of all possible test conditions, but ones which are chosen to uncover almost as many defects as an exhaustive test would have uncovered
– The key question is :
•What subset of all possible test cases has the highest probability of detecting the most errors ?
Regression Testing:
Comprehensive re-test of an entire system :
•after system delivery, when a modification has been made, or
•before delivery and at the end of system test , after all test cases have been passed, but not passed together against the final version of the system product
Regression Testing:
•Regression testing requires that a regression test bed (comprehensive set of re-usable system test cases) be available throughout the useful life of the delivered system
•Functional acceptance tests form the core of this regression test bed.
Regression Testing:
•The regression test must be maintained, to keep it aligned with the system as the system itself evolves . This maintenance may not be a trivial effort.
•If a careful determination is made that only portions or subsystems will be affected by a particular change (i.e., the subsystems are decoupled and insulated), then only a partial regression re-test of the affected portions is absolutely necessary
Regression Testing:
•If errors are detected during on-going system operation, then test case(s) should be added to the existing regression test bed to detect any possible further re-occurrences of the error or related errors
•The additional effort to build a regression test facility is relatively minor if it is done during system development : the attitude should be that test cases are designed and organized to have an on-going life after system delivery, not one-time throw-aways
•The decision to perform any particular regression test is based on an analysis of the specific risks and costs
Regression Testing:
•Regression tests are more manageable and cost-effective they are coordinated with scheduled releases, versus a piece-meal approach to system modification
•Full regression testing should always be performed when the overall system architecture has been affected
•A guideline for setting the boundaries of regression testing: include any interdependent, integrated applications; exclude tangential applications.
•Assessing Test Effectiveness•Techniques that provide means to assess the effectiveness, coverage and robustness of a set of test cases:
–Error Seeding
–Mutation Analysis
Note :
These techniques are talked about more than they are practiced. Many people see them as somewhat impractical because they can be difficult to apply well.
Mutation Analysis Approach:
Create numerous (minor) variations of the baseline system or program. Then test each variation with the original, unchanged set of test cases, and determine how many of the tests behave as if the product was not changed
Purpose:
Mutations which do not test differently from the baseline product are carefully examined, to determine whether in fact the tests are inadequate
Systematically Running a Software System Or Program
To uncover errors or defects
Software Testing in Organizations aiming for SEI Level 3 +:
•An acknowledged competitive advantage contributing to business success
•Moved from ad hoc to an engineering discipline
•Carrier for faster product delivery, with version controlled test plans and test cases
•Led to reliability prediction through defect metrics
•Focussed on defect elimination at the stage of defect induction itself
Testing Phases:
–Unit Testing
–Integration Testing
–System Testing
–Acceptance Testing
All Test Plans and Test Cases shall be Reviewed and Approved before Testing starts.
Unit Testing
–Lowest-level component test–Key foundation for later levels of testing
–Detects 65% - 75% of all bugs
–Stand-alone test
–Ensures the conformance to each unit in DDD
Integration Testing
–An incremental series of tests of combinations or subassemblies of selected components in an overall system
–Integration testing is incremental in that successively larger and more complex combinations of components are tested in sequence, proceeding from the unit level to eventually the full-integrated system
–Ensures the conformance to each module in HLDD
System Testing
–Highest level of application functionality testing performed by the systems group
–Ensures the conformance to the Functional Requirements as specified in the SRS
Acceptance Testing
–Independent test performed by the users or QA prior to accepting the delivered system
–Ensures the conformance to the Functional Requirements as specified in the URD
Testing Activities
•Test Case Design
•Review of Test Case Design
•Testing
•Recording Testing Results
•Review and Sign-Off of Testing Results
•Defect Reporting
Testing Documents and Records
Documents (SCIs)
Test Plan for the Project
Test Cases Documents
Records
Test Report
Defect Log
Review Reports
Unit Test Plan (UTP)
•Plan for the entire Unit Testing in the project
•Identifies the Units/Sub-units covered during Unit Testing
•Each feature in the DDD mapped to a Unit Test Case ID
•To be prepared at DDD phase itself
•The test cases are designed and documented •References to the Test Case document(s) shall be given in UTP
•Deliverables at Unit Testing Phase shall be identified in UTP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
Testing coverage shall be identified in the UTP:
•Path Coverage
•Statement Coverage
•Decision (Logic/Branch) Coverage
•Condition Coverage
•Decision/Condition Coverage
•Multiple-Condition Coverage
•Functionality testing
•User interface testing
•Regression testing
Unit Testing CoveragePath Coverage
-Test cases will be written to cover all the possible paths of control flow through the program.
Statement Coverage
–Test cases will be written such that every statement in the program is executed at least once
Decision (Logic/Branch) Coverage
–Test Cases will be written such that each decision has a true or false outcome at least once.
Condition Coverage
–Test Cases will be written such that each condition in a decision takes on all possible outcomes at least once.
Decision/Condition Coverage
–Test Cases will be written such that each condition in a decision takes on all possible outcomes at least once, each decision takes on all possible outcomes at least once, and each point of entry is invoked at least once.
Multiple-Condition Coverage
–Test Cases will be written such that all possible combinations of condition outcomes in each decision, and all points of entry, are invoked at least once.
Integration Test Plan (ITP)
•Plan for the entire Integration Testing in the project
•Identifies the Modules/Sub-Modules covered during Integration Testing
•Each component of HLDD mapped to a Integration Test Case ID
•Recommended to prepare at HLD phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in ITP
•Deliverables at Integration Testing Phase shall be identified in ITP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
Testing coverage shall be identified in the ITP:
•Functionality Testing•User interface testing
•Dependency (API) testing
•Smoke testing
•Capacity and volume testing
•Error / disaster handling and recovery
•Concurrent execution testing
•Equivalence partitioning
•Boundary-value analysis
•Cause-effect graphing
•Error guessing
•Equivalence Partitioning
•The test cases are partitioned into equivalence classes such that
–each test case should invoke as many different input conditions as possible in order to minimize the total number of test cases necessary
–if one test case in an equivalence class detects an error, all other test cases in the equivalence class would be expected to find the same error
Boundary-value analysis
–Test cases explore boundary conditions
–Test cases consider input conditions as well as output conditions
Cause-effect graphing
–Technique to identify test cases by translating the specifications into Boolean logic network.
–This is a systematic method of generating test cases representing combinations of conditions
Error guessing
–The test cases are written both by intuition and experience to uncover certain probable type of errors
–The test case design is by the knack of “smelling out” errors
System Test Plan (STP)
•Plan for the entire System Testing in the project
•Identifies the features covered during System Testing
•Each feature in the SRS mapped to a System Test Case ID
•Recommended to prepare at SRS phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in STP
•Deliverables at System Testing Phase shall be identified in STP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
•Functionality Testing
•User interface testing
•Usability testing
•Volume Testing
•Stress Testing
•Security Testing
•Performance Testing
•Installation and upgrade testing
•Standards conformance testing
•Configuration testing
•Network and distributed environment testing
•Forward / backward compatibility testing
•Reliability Testing
•Error / disaster handling and recovery testing
•Serviceability Testing
•Documentation testing
•Procedure testing
•Localization testing
Acceptance Test Plan (ATP)
•Plan for the entire Acceptance Testing in the project
•Identifies the features covered during Acceptance Testing
•Each feature in the URD mapped to a Acceptance Test Case ID
•Recommended to prepare at URD phase itself
•The test cases are designed and documented
•References to the Test Case document(s) shall be given in ATP
•Deliverables at Acceptance Testing Phase shall be identified in ATP
•Resources required shall be mentioned in Project Plan
•Schedules shall be planned
•Alpha Test•Beta Test
•All the tests covered under system testing can be repeated as per user requirements
Terminology
Functionality Testing
–Functionality testing is the determination of whether each functionality mentioned in SRS is actually implemented. The objective is to ensure that all the functional requirements as documented in the SRS are accomplished.
User Interface Testing
–Focus is on testing the user interface, navigation and negative user behavior
Concurrent Execution Testing
–Focus is on testing with simple usage, standard usage and boundary situations
Volume Testing
–Volume Testing is to ensure that the software
•can handle the volume of data as specified in the SRS
•does not crash with heavy volumes of data, but gives an appropriate message and/or makes a clean exit.
Example:
•A compiler would be fed an absurdly large source program to compile
Stress Testing
–Executes the software in a manner that demands resources in abnormal quantity, frequency and volume and verifies that the system either performs normally, or displays message regarding the limitations of the system
Usability Testing
–Usability testing is an attempt to uncover the software usability problems involving the human-factor
Example:
•Are the error messages meaningful, easy to understand?
•Security Testing
–Attempts to verify that protection mechanism built into a system will in fact protect it from improper penetration
Example:
•a database management system's data security mechanisms
•Performance Testing
–Tests the run-time performance of software within the context of an integrated system
Example:
•the response times under certain configuration conditions.Installation and Upgrade Testing
–Focus is on testing whether the user would be install and upgrade the system
Configuration Testing
–Configuration testing includes either or both of the following:
•testing the software with the different possible hardware configurations
•testing each possible configuration of the software
Network and Distributed Environment Testing
–Focus is on testing the product in the required network and distributed environment
Dependency (API) Testing
–Focus is on testing the API calls made by the system to other systems
Localization Testing
–Focus is on testing problems associated with multiple languages support, conversion and also hardware aspects
Reliability Testing
–The various software testing processes have the goal to test the software reliability. The "Reliability Testing" which is a part of System Testing encompasses the testing of any specific reliability factors that are stated explicitly in the SRS
Error / Disaster Handling and Recovery Testing
–Forces the software to fail in a variety of ways and verifies that the system recovers and resumes processing
Serviceability Testing
–Serviceability testing covers the serviceability or maintainability characteristics of the software
Example:
•service aids to be provided with the system, e.g., storage-dump programs, diagnostic programs
•the maintenance procedures for the system
Documentation Testing
–Documentation testing is concerned with the accuracy of the user documentation. This involves
•Review of the user documentation for accuracy and clarity
•Testing the examples illustrated in the user documentation by preparing test cases on the basis of these examples and testing the system
Smoke Testing
–Focus is on testing the system releases as it is being built to uncover critical and showstopper errors
Configuration Testing
–Configuration testing includes either or both of the following:
•testing the software with the different possible hardware configurations
•testing each possible configuration of the software
Procedure Testing
–If the software forms a part of a large and not completely automated system, the interfaces of the developed software with the other components in the larger system shall be tested. These may include procedures to be followed by
•The human operator
•Database administrator
•Terminal user
–These procedures are to be tested as part of System testing
•Standards Conformance Testing–Focus is on testing whether the product conforms to prescribed and published standards
•Alpha Test–Within a vendor, the last level of internal test prior to a limited beta release
•Beta Test
–The vendor’s equivalent of a pilot test (usually with a greater number of participating sites than for a pilot)
•Alpha Test–Within a vendor, the last level of internal test prior to a limited beta release
•Beta Test
–The vendor’s equivalent of a pilot test (usually with a greater number of participating sites than for a pilot)
•Equivalence Test
–A test using only a small sample of all possible test conditions, but ones which are chosen to uncover almost as many defects as an exhaustive test would have uncovered
– The key question is :
•What subset of all possible test cases has the highest probability of detecting the most errors ?
Regression Testing:
Comprehensive re-test of an entire system :
•after system delivery, when a modification has been made, or
•before delivery and at the end of system test , after all test cases have been passed, but not passed together against the final version of the system product
Regression Testing:
•Regression testing requires that a regression test bed (comprehensive set of re-usable system test cases) be available throughout the useful life of the delivered system
•Functional acceptance tests form the core of this regression test bed.
Regression Testing:
•The regression test must be maintained, to keep it aligned with the system as the system itself evolves . This maintenance may not be a trivial effort.
•If a careful determination is made that only portions or subsystems will be affected by a particular change (i.e., the subsystems are decoupled and insulated), then only a partial regression re-test of the affected portions is absolutely necessary
Regression Testing:
•If errors are detected during on-going system operation, then test case(s) should be added to the existing regression test bed to detect any possible further re-occurrences of the error or related errors
•The additional effort to build a regression test facility is relatively minor if it is done during system development : the attitude should be that test cases are designed and organized to have an on-going life after system delivery, not one-time throw-aways
•The decision to perform any particular regression test is based on an analysis of the specific risks and costs
Regression Testing:
•Regression tests are more manageable and cost-effective they are coordinated with scheduled releases, versus a piece-meal approach to system modification
•Full regression testing should always be performed when the overall system architecture has been affected
•A guideline for setting the boundaries of regression testing: include any interdependent, integrated applications; exclude tangential applications.
•Assessing Test Effectiveness•Techniques that provide means to assess the effectiveness, coverage and robustness of a set of test cases:
–Error Seeding
–Mutation Analysis
Note :
These techniques are talked about more than they are practiced. Many people see them as somewhat impractical because they can be difficult to apply well.
Mutation Analysis Approach:
Create numerous (minor) variations of the baseline system or program. Then test each variation with the original, unchanged set of test cases, and determine how many of the tests behave as if the product was not changed
Purpose:
Mutations which do not test differently from the baseline product are carefully examined, to determine whether in fact the tests are inadequate
Sunday, May 13, 2007
Manual Web Testing Points to be Consider
Module Testing
GUI (Graphical User Interface) Testing
Database testing
Performance testing
Browser compatibility
Integration testing
System testing
Subscribe to:
Posts (Atom)