General
a) All Programs are Compiled successfully
b) All Programs are commented properly.
c) All Programs do not have spelling mistakes and grammar errors in comments.
d) All program names in Build are same as in Release Notes.
e) Online Help exists for each program unit.(for each page if applicable)
f) User Manual does contain all program names and associated text.
GUI Interface
a) Background and Foreground color combination suits the eyes
b) Program works in all screen resolution modes
c) Program does not contain orphan/broken web links
d) Application does not raise any critical error message ( It should be returned to developer if continuously severe errors encountered)
Tuesday, January 27, 2009
what are the acceptance requirements that a system must meet in acceptance testing?
Acceptance requirements that a system must meet can be divided in 4 categories:
a) Functionality requirements, which relate to the business rules that the system must execute.
b) Performance requirements, which relate to operational requirements such as time or resource constraints.
c) Interface quality requirements, which relate to a connection to another component of processing.
d) Overall software quality requirements are those that specify limits for factors or attributes such as reliability, testability, correctness, and usability.
a) Functionality requirements, which relate to the business rules that the system must execute.
b) Performance requirements, which relate to operational requirements such as time or resource constraints.
c) Interface quality requirements, which relate to a connection to another component of processing.
d) Overall software quality requirements are those that specify limits for factors or attributes such as reliability, testability, correctness, and usability.
List the concerns that testers face in assuring that the test plan will be complete.
a) Not Enough Training
b) Us versus Them mentality
c) Lack of Test Tools
d) Lack of Management Understanding/Support of testing
e) Lack of Customer and User Involvement
f) Not Enough Time for testing
g) Over Reliance On Independent Testers
h) Rapid Changes
i) Testers Are in a Lose-Lose Situations
j) Having to Say ‘No’
b) Us versus Them mentality
c) Lack of Test Tools
d) Lack of Management Understanding/Support of testing
e) Lack of Customer and User Involvement
f) Not Enough Time for testing
g) Over Reliance On Independent Testers
h) Rapid Changes
i) Testers Are in a Lose-Lose Situations
j) Having to Say ‘No’
If you happen to be the customer/user of the software, what are (your) the responsibilities for software acceptance?
a) Creation (or Update/review) of Acceptance Test Plan
b) Resource and Schedule for Acceptance Testing
c) Special Data collection for Acceptance Testing
d) Training Logistics for Acceptance Testing
e) Following Standard Practices while carrying out any tasks.
b) Resource and Schedule for Acceptance Testing
c) Special Data collection for Acceptance Testing
d) Training Logistics for Acceptance Testing
e) Following Standard Practices while carrying out any tasks.
what is meant by error seeding and fault injection?
Error Seeding: Error Seeding is a statistical method used to assess the number and characteristics of the faults remaining in the program. First faults are seeded into the program. Then the program is tested and the number of faults discovered is used to estimate the number of faults yet undiscovered. A difficulty with this technique is this that the seeded errors should represent the yet undiscovered errors.
Fault Injection: An incorrect step, process or data definition in a program (code). The outgrowths of a mistake produce by a human action that produces the incorrect result (a failure e.g. a fault causing an OS to lock-up.) Fault injection has been extensively used in the last decade to evaluate fault tolerance mechanisms and to assess the impact of faults in the systems. A measure issue is to assure that the injected faults are representative of actual faults, as this is the necessary condition to obtain meaningful results. First faults are seeded into the program. Then the program is tested and the number of faults discovered is used to estimate the number of faults yet undiscovered.
Fault Injection: An incorrect step, process or data definition in a program (code). The outgrowths of a mistake produce by a human action that produces the incorrect result (a failure e.g. a fault causing an OS to lock-up.) Fault injection has been extensively used in the last decade to evaluate fault tolerance mechanisms and to assess the impact of faults in the systems. A measure issue is to assure that the injected faults are representative of actual faults, as this is the necessary condition to obtain meaningful results. First faults are seeded into the program. Then the program is tested and the number of faults discovered is used to estimate the number of faults yet undiscovered.
Write a Test transaction for a scenario where 6.2% of tax deduction for the first $62,000 of income has to be done?
For test transaction need to be consider Lower boundary, On the boundary and Upper boundary while creating scenario.
a) Check if 6.2% tax deduction is calculated for income less than $62,000.
b) Check if 6.2% tax deduction is calculated for income equal to $62,000
c) Check if 6.2% tax deduction is not calculated for income more than $62,000. (Because in the question he mentioned clearly that it’s for the first $62,000)
a) Check if 6.2% tax deduction is calculated for income less than $62,000.
b) Check if 6.2% tax deduction is calculated for income equal to $62,000
c) Check if 6.2% tax deduction is not calculated for income more than $62,000. (Because in the question he mentioned clearly that it’s for the first $62,000)
Explain any 5 Test Metrics
Test Coverage = Number of units (KLOC/FP) tested / total size of the system
Test cost (in %) = Cost of testing / total cost *100
Cost to locate defect = Cost of testing / the number of defects located
Defects detected in testing (in %) = Defects detected in testing / total system defects*100
Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria
Test cost (in %) = Cost of testing / total cost *100
Cost to locate defect = Cost of testing / the number of defects located
Defects detected in testing (in %) = Defects detected in testing / total system defects*100
Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria
Your customer doesn’t know how to prepare acceptance test plan. Write contents of the acceptance test plan for his guidance.
Acceptance criteria should include success criteria for:
a) Functional Requirements -which relate to the business rules that the system must execute
b) Performance Requirements which relate to operational requirements such as time or resource constraints
c) Interfacing quality Requirements-which relate to a connection to another component of processing(human/machine-machine/module)
d) Overall Software Quality requirements –are those that specify limits for factors or attributes such as reliability, testability, correctness and usability
- Project description
- Scope
- User roles and responsibilities
- Acceptance criteria
- Test cases to be executed
Acceptance Criterion
-----------------------------------------------------------------------------------------------
No Acceptance req Critical Result Remark
Yes No Accept Reject
-----------------------------------------------------------------------------------------------
1 Login time
- should not be Yes Yes Accept
more than 6 sec
2. Concurrent users (min 40 Users) Yes Reject Only 30 works
a) Functional Requirements -which relate to the business rules that the system must execute
b) Performance Requirements which relate to operational requirements such as time or resource constraints
c) Interfacing quality Requirements-which relate to a connection to another component of processing(human/machine-machine/module)
d) Overall Software Quality requirements –are those that specify limits for factors or attributes such as reliability, testability, correctness and usability
- Project description
- Scope
- User roles and responsibilities
- Acceptance criteria
- Test cases to be executed
Acceptance Criterion
-----------------------------------------------------------------------------------------------
No Acceptance req Critical Result Remark
Yes No Accept Reject
-----------------------------------------------------------------------------------------------
1 Login time
- should not be Yes Yes Accept
more than 6 sec
2. Concurrent users (min 40 Users) Yes Reject Only 30 works
Write standards for test plan, test script, test report.
Test Plan (Make this in table format)
a) Preface
b) Executive Summary
c) Introduction of the Project
d) Organization of Document
e) Test Organization
f) Test Techniques
g) Test Tools
h) Acceptance Criteria
i) Glossary
Test Script
a) Date of Test Conducted
b) Type of Test Conducted
c) Serial Number
d) Statement to Execute
e) Expected Result
f) Actual Result
g) Remark
Test Report
a) Name of the Project
b) Name of the Module/Subsystem
c) Type of the Testing
d) In the Heading Present the total number of Defects
e) Graphical Analysis of Test Result visa Histogram or so….
a) Preface
b) Executive Summary
c) Introduction of the Project
d) Organization of Document
e) Test Organization
f) Test Techniques
g) Test Tools
h) Acceptance Criteria
i) Glossary
Test Script
a) Date of Test Conducted
b) Type of Test Conducted
c) Serial Number
d) Statement to Execute
e) Expected Result
f) Actual Result
g) Remark
Test Report
a) Name of the Project
b) Name of the Module/Subsystem
c) Type of the Testing
d) In the Heading Present the total number of Defects
e) Graphical Analysis of Test Result visa Histogram or so….
Define and Give examples for a] Boundary Value Analysis b] Equivalence Partitioning c] Error Guessing?
a] Boundary Value Analysis:- It is a test data selection technique in which input data is selected from the input and output boundaries of a given functions. It is a technique that consists of developing test cases and data that focus on the input and output boundaries of a given function.
Example: - Salary should be between 10,000 and 15,000. so test data should be between
Low boundary plus or minus (9,999 & 10,001)
On the boundary (10,000 & 15,000)
Upper boundary plus or minus (14,999 & 15,001)
b] Equivalence Partitioning:-An equivalence class is a subset of data which representative of larger class. Equivalence partitioning is a technique for testing equivalence classes rather than under taking exhaustive testing of each volume of larger class.
Example: - A program which edits credit limits within a given range ($10,000 - $15,000) would have three equivalence classes
Less than 10,000 (Invalid)
Between 10,000 & 15,000 (Valid)
Greater than 15,000(Invalid)
c] Error Guessing
Error guessing is based on the theory that test cases can be developed based upon the intuition and experience of test engineer.
Example:-
Where one of the inputs is date a test engineer may try February 29, 2004 or 9/9/99
Example: - Salary should be between 10,000 and 15,000. so test data should be between
Low boundary plus or minus (9,999 & 10,001)
On the boundary (10,000 & 15,000)
Upper boundary plus or minus (14,999 & 15,001)
b] Equivalence Partitioning:-An equivalence class is a subset of data which representative of larger class. Equivalence partitioning is a technique for testing equivalence classes rather than under taking exhaustive testing of each volume of larger class.
Example: - A program which edits credit limits within a given range ($10,000 - $15,000) would have three equivalence classes
Less than 10,000 (Invalid)
Between 10,000 & 15,000 (Valid)
Greater than 15,000(Invalid)
c] Error Guessing
Error guessing is based on the theory that test cases can be developed based upon the intuition and experience of test engineer.
Example:-
Where one of the inputs is date a test engineer may try February 29, 2004 or 9/9/99
State 4 reasons: Why to test the software?
a) To uncover the bugs
b) Improve the quality prior to sales
c) To check the performance of application in
d) To ensure that application is as per user requirement
e) To reduce the failure cost
f) To reduce the risk
g) To build the confidence in to software
b) Improve the quality prior to sales
c) To check the performance of application in
d) To ensure that application is as per user requirement
e) To reduce the failure cost
f) To reduce the risk
g) To build the confidence in to software
When to stop testing?
When the test manager is confident that the application will perform as expected in the production environment. This confidence is derived by analysis conducted on:
a) Deadlines (release deadlines, testing deadlines, etc.) are met
b) Test cases completed with certain percentage passed
c) Test budget depleted
d) Coverage of code/functionality/requirements reaches a specified point
e) Meantime between failures
f) Risks associated with continuing to test
g) Percentage of coverage achieved
h) Number of open defects and their severity
i) Risk associated with move to production
a) Deadlines (release deadlines, testing deadlines, etc.) are met
b) Test cases completed with certain percentage passed
c) Test budget depleted
d) Coverage of code/functionality/requirements reaches a specified point
e) Meantime between failures
f) Risks associated with continuing to test
g) Percentage of coverage achieved
h) Number of open defects and their severity
i) Risk associated with move to production
Write Test plan considerations for ….new e-business solution
Understanding the ecommerce could have a huge financial impact on a company if not conducted properly. Also the fact that the infrastructure is not in place for the company also will require additional testing. The types of tests I would conduct include the following:
Structural Test
a) Stress - for the project volumes of transactions and number of users. It is key to know where the limits of the application exist.
b) Execution
c) Recovery - With financial transactions it is important to know a system can stop mid transaction and when it comes back up is able to return to processing when it comes back up.
d) Security - Internal and external security
Functional Test
a. Requirements
b. Error Handling
c. Intersystem
Risk:-
a) Security
b) Performance
c) Correctness
d) Compatibility
e) Reliability
f) Authentication
g) Privacy and Protection of Information
h) Fraud and Misrepresentation
i) Reliability of trading partners
j) Technology Risk
Personnel are involved:
a) Internal Auditor
b) Risk consultant
c) Data Processor
d) Security Officer
e) Computer operations manager
Structural Test
a) Stress - for the project volumes of transactions and number of users. It is key to know where the limits of the application exist.
b) Execution
c) Recovery - With financial transactions it is important to know a system can stop mid transaction and when it comes back up is able to return to processing when it comes back up.
d) Security - Internal and external security
Functional Test
a. Requirements
b. Error Handling
c. Intersystem
Risk:-
a) Security
b) Performance
c) Correctness
d) Compatibility
e) Reliability
f) Authentication
g) Privacy and Protection of Information
h) Fraud and Misrepresentation
i) Reliability of trading partners
j) Technology Risk
Personnel are involved:
a) Internal Auditor
b) Risk consultant
c) Data Processor
d) Security Officer
e) Computer operations manager
what are the points to be emphasized in testing of a system which has multiple workstations for inputting (as against a single workstation system)?
a) Seduconcurrancy testing :- When one user or console is updating /adding /deleting a record other user should not be allowed to process the same record - Other user should be in wait state
b) Load test / stress test should be done
c) Recovery test should be done
d) Performance testing: - Response time/Transaction completion time
e) Data validation: - Check if data is saved in correct order, data retrieval is correct
f) Access control
g) Database security
h) Concurrent user test
i) Update and Rollback
j) Response time and Server connectivity
b) Load test / stress test should be done
c) Recovery test should be done
d) Performance testing: - Response time/Transaction completion time
e) Data validation: - Check if data is saved in correct order, data retrieval is correct
f) Access control
g) Database security
h) Concurrent user test
i) Update and Rollback
j) Response time and Server connectivity
which 3 tools would you purchase for your company for use in testing and justify why you would want them?
# Test management tool - (Optional)
why because : Helps in :
- Saving time in the creation and duplication of test scripts
-Staffing decisions accurately
-Testing process and implementation
- Maintaining documentation and implementation to meet the
requirements
- Taking corrective steps before the system goes live
- Tracking specific information about individual test scripts
- Getting updates on daily testing activities
- Record test results and produce statistical reports on test results
# Unit test tool -
Why because:
unit test must be performed by the programmers and each class implemented must have programmer-developed unit tests, for everything that "could possibly break". These tests are to be written during coding of the class, preferably right before implementing a given feature. Tests are run as frequently as possible during development, and all unit tests in the entire system must be running at 100% before any developer releases his code.
# functional/regression test tool -
If I adopt this tool then i can reduce risks and make the most of limited resources by implementing automated functional and regression testing of critical applications, whereby development and QA teams create reusable test assets that provide in-depth defect analysis and expedite resolution of defects.
Why because:
-It will ensure that the quality at every step of delivery
-It will fill the gap between the development and testing teams
-it will quickly identify and correct problems and cost- effective
# Load and stress tool -
Why because
It will help us to interpret the data, and point at the most likely performance problems.
# Bug tracking tool -(Optional)
Why because:
It is quite hard to keep track of what bugs have been raised, and which of them have been fixed. As well as automating this process, bug tracking tools make it extremely easy to analyze projects using metrics and reports
why because : Helps in :
- Saving time in the creation and duplication of test scripts
-Staffing decisions accurately
-Testing process and implementation
- Maintaining documentation and implementation to meet the
requirements
- Taking corrective steps before the system goes live
- Tracking specific information about individual test scripts
- Getting updates on daily testing activities
- Record test results and produce statistical reports on test results
# Unit test tool -
Why because:
unit test must be performed by the programmers and each class implemented must have programmer-developed unit tests, for everything that "could possibly break". These tests are to be written during coding of the class, preferably right before implementing a given feature. Tests are run as frequently as possible during development, and all unit tests in the entire system must be running at 100% before any developer releases his code.
# functional/regression test tool -
If I adopt this tool then i can reduce risks and make the most of limited resources by implementing automated functional and regression testing of critical applications, whereby development and QA teams create reusable test assets that provide in-depth defect analysis and expedite resolution of defects.
Why because:
-It will ensure that the quality at every step of delivery
-It will fill the gap between the development and testing teams
-it will quickly identify and correct problems and cost- effective
# Load and stress tool -
Why because
It will help us to interpret the data, and point at the most likely performance problems.
# Bug tracking tool -(Optional)
Why because:
It is quite hard to keep track of what bugs have been raised, and which of them have been fixed. As well as automating this process, bug tracking tools make it extremely easy to analyze projects using metrics and reports
What fields would you include in creating a new defect tracking program (used by QA, developers, etc)?
It should content following fields:-
Defect Id
Status:- New, Active, Fixed, Resolve, Duplicate, By Design, Closed, Reopen
Assigned to
Source of Defect (Test case or other source)
Stage of origination
Project
Subject
Summary
Description
R&D comments
Reproducible Yes/No
Severity: - Low, Medium, High, Very High, Urgent
Priority: - Low, Medium, High, Very High, Urgent
Detected By
Detected date
Detected in version
Closed in version
Screen print & Logs etc.
Defect Id
Status:- New, Active, Fixed, Resolve, Duplicate, By Design, Closed, Reopen
Assigned to
Source of Defect (Test case or other source)
Stage of origination
Project
Subject
Summary
Description
R&D comments
Reproducible Yes/No
Severity: - Low, Medium, High, Very High, Urgent
Priority: - Low, Medium, High, Very High, Urgent
Detected By
Detected date
Detected in version
Closed in version
Screen print & Logs etc.
Test plan update
"The top management was feeling that when there are any changes in the technology being used, development schedules etc, it is a waste of time to update the Test Plan also from time to time. Instead, they are emphasizing that you should put your time into testing than working on the test plan. Your Project Manager asked for your opinion. You have argued that Test Plan is very important and you need to update your test plan from time to time. It is not at all a waste of time and testing activities would be more effective when you have your plan clear. Explain with some metrics; how you would support your argument to have the test plan in place all the time."?
Answer
Testing is a process and needs planning; too. The test plan serves as a contract between QA and the project team. It serves as the foundation for the QA team. If you don't plan, plan to fail. QA will not be able to verify the test coverage of the new technology if it is not addressed in the test plan. Further, the test plan will address how the technology will be tested in regards to test factors and if the technology will negatively impact other features.
Without a test plan, we don’t know
a) What we are going to do
b) How we are going to do it
c) How long will it take
d) Resources required
e) Cost
Without a documented test plan
a) The test can not be verified
b) Coverage can not be analyzed
c) The test is not repeatable
d) Test can not be controlled
e) Test plan is a tool to verify or confirm requirement
f) Review of test plan can identify gaps and missing or vague requirement
g) Test plan ensures that each requirement is testable
h) Planning helps in selection of test cases
i) Test plan is a dynamic document like any other plan
j) Test plan defines overall testing objectives and approach
k) Test design defines what is selected to test and describes expected results
Or
If we keep the Test Plan up-to-date, it will enable:
a) Keep a track of all deliverables as defined in test plan with any slippages in their schedules. If any slippages are occurring, take preventive actions. Also enable to estimate h/w and manpower resources/efforts, based on technological changes.
b) Test progress S curve (plan, attempted, actual test cases) shall give a fair indication of the status of testing activity.
c) Technological changes might lead to changes in test strategies. An updated test plan would enable effective testing coz strategy would be well defined and updated
d) Ascertain when the testing activity should stop
e) Estimate correct value of Code coverage (Percentage of lines of code executed per total lines of code) and take necessary action to test components that cover less code
f) Serve as a basis for testing a similar kind of application in future
Answer
Testing is a process and needs planning; too. The test plan serves as a contract between QA and the project team. It serves as the foundation for the QA team. If you don't plan, plan to fail. QA will not be able to verify the test coverage of the new technology if it is not addressed in the test plan. Further, the test plan will address how the technology will be tested in regards to test factors and if the technology will negatively impact other features.
Without a test plan, we don’t know
a) What we are going to do
b) How we are going to do it
c) How long will it take
d) Resources required
e) Cost
Without a documented test plan
a) The test can not be verified
b) Coverage can not be analyzed
c) The test is not repeatable
d) Test can not be controlled
e) Test plan is a tool to verify or confirm requirement
f) Review of test plan can identify gaps and missing or vague requirement
g) Test plan ensures that each requirement is testable
h) Planning helps in selection of test cases
i) Test plan is a dynamic document like any other plan
j) Test plan defines overall testing objectives and approach
k) Test design defines what is selected to test and describes expected results
Or
If we keep the Test Plan up-to-date, it will enable:
a) Keep a track of all deliverables as defined in test plan with any slippages in their schedules. If any slippages are occurring, take preventive actions. Also enable to estimate h/w and manpower resources/efforts, based on technological changes.
b) Test progress S curve (plan, attempted, actual test cases) shall give a fair indication of the status of testing activity.
c) Technological changes might lead to changes in test strategies. An updated test plan would enable effective testing coz strategy would be well defined and updated
d) Ascertain when the testing activity should stop
e) Estimate correct value of Code coverage (Percentage of lines of code executed per total lines of code) and take necessary action to test components that cover less code
f) Serve as a basis for testing a similar kind of application in future
what activity is done in Acceptance Testing, which is not done in System testing to ensure the Customer requirements?
Main objective of both system and acceptance testing is ensure all the functionalities and requirement are implemented correctly or not. In Acceptance testing. real time test data is used, usually acceptance testing is done by client or clients representative. Here client will check the all functionalities. But in System testing. - project team and testing team should do testing. Here we have to do all system testing like load, stress, performance etc. - Its not mandatory that we have to use real time data again its depends on the type of application.
a) Acceptance test is executed by users where as system test is executed by the testers.
b) Acceptance test is the responsibility of the client and is a final check point to accept or reject the software.
c)Acceptance testing involves the actual production environment where as system testing simulates the production environment
d) Acceptance testing involves testing the acceptance test criteria mentioned in the Acceptance test plan and verifying that the product meets those criteria where as system test involves testing the entire system and verifying the performance as a whole.
e) System test is carried out when the entire system is ready where as acceptance tests can be carried out at pre - defined intervals.
a) Acceptance test is executed by users where as system test is executed by the testers.
b) Acceptance test is the responsibility of the client and is a final check point to accept or reject the software.
c)Acceptance testing involves the actual production environment where as system testing simulates the production environment
d) Acceptance testing involves testing the acceptance test criteria mentioned in the Acceptance test plan and verifying that the product meets those criteria where as system test involves testing the entire system and verifying the performance as a whole.
e) System test is carried out when the entire system is ready where as acceptance tests can be carried out at pre - defined intervals.
How would u prepare the Test plan for the Project?
Assumptions:-
The requirements are given clearly to u, stating that u have to prepare
Plan for CSTE certification Online by
1. Recording candidate information
2. Scheduling candidates for exams
3. Keeping track of results and
4. Sending out certificates
Test Plan covering the details
1. Who is going to prepare the plan?
2. Which is your reference document for preparing the Plan?
3. What SDLC the project is going to attain in that when testing comes.
4. When Testing starts for each module of the application.
5. What type of testing (Manual or Automation?)
6. Who is going to test it?
7. When he is going to test it
8. Which tools you’re going to use?
9. Plan or give date if training is need for Tools
10. What are the Deliverables?
11. What is the knowledge of the application by the Tester, check if
Training is needed.
12. What level of testing is going to be done?
13. When Test Cases is going to prepared
14. Who is going to review it?
15. Who the Defect Report is reported?
The requirements are given clearly to u, stating that u have to prepare
Plan for CSTE certification Online by
1. Recording candidate information
2. Scheduling candidates for exams
3. Keeping track of results and
4. Sending out certificates
Test Plan covering the details
1. Who is going to prepare the plan?
2. Which is your reference document for preparing the Plan?
3. What SDLC the project is going to attain in that when testing comes.
4. When Testing starts for each module of the application.
5. What type of testing (Manual or Automation?)
6. Who is going to test it?
7. When he is going to test it
8. Which tools you’re going to use?
9. Plan or give date if training is need for Tools
10. What are the Deliverables?
11. What is the knowledge of the application by the Tester, check if
Training is needed.
12. What level of testing is going to be done?
13. When Test Cases is going to prepared
14. Who is going to review it?
15. Who the Defect Report is reported?
The five dimensions of the Risks are?
a) Technology Integration - What kind of technology are we using a new hardware/software technology. We may not have enough expertise on that. The
technology might not be time proven yet.
b) Size and Complexity - The Size and the Complexity of the application. e.g Banking applications are huge and more complex
c) System Environment and Stability - Env (e.g web based might not be stable)
d) Criticality/Mission Impact - Life critical applications like medical equipment simulation, flight simulation are highly critical. Financial, e-banking are critical as per the financial aspect
e) Reliability and Integrity - Finally, the application must be reliable and must do what it is intended for...
Q3) Explain testing life cycle?
Ans3:
Test Analysis---Test Plan---Test case Design---Test Execution---Test
Log--------Test Report.
|
^
|
|
Tracking Defect
technology might not be time proven yet.
b) Size and Complexity - The Size and the Complexity of the application. e.g Banking applications are huge and more complex
c) System Environment and Stability - Env (e.g web based might not be stable)
d) Criticality/Mission Impact - Life critical applications like medical equipment simulation, flight simulation are highly critical. Financial, e-banking are critical as per the financial aspect
e) Reliability and Integrity - Finally, the application must be reliable and must do what it is intended for...
Q3) Explain testing life cycle?
Ans3:
Test Analysis---Test Plan---Test case Design---Test Execution---Test
Log--------Test Report.
|
^
|
|
Tracking Defect
Write a brief test plan for this new project?
Test plan of the above will consist of:
a) Objective: To automate the CSTE certification online
b) Identifier: CSTE-AUT/1/India
c) References: Records of earlier CSTE Exams held Names of the people involved in earlier exams held CBOK, exam material, application forms, certification
d) Test Items:
The forms of the site for application, re-scheduling, postponing exam etc.
On line tutorials.
Help support provided
CSTE main exam links
Connectivity.
Databases
e) Risks:
Recovery
Correctness
Security
Unauthorized access of the application
Databases Integrity
Redundancy, Configuration management,
portability, Maintainability
Ease of use, ease of operation.
f) Testing Strategy:
White box testing
Black box testing
Stress testing,
Performance testing
Configuration Management,
System testing, Security Testing,
(can add some more techniques for testing)
g) Acceptance Criteria:
Criteria for exit like the requirements specified by user are met fully, the site is fault free, and the application for the main exam is working fine. The functionality is fully satisfied.
h) Suspension or Reject Criteria:
Testing cannot progress if any of the test items do not satisfy all the input data is provided. Enough time is provided.
i) Schedules:
Schedules given here for each type of testing and reporting results.
j) Environmental Needs:
Tools required for testing. Connectivity provided. Recovery data provided.
k) Training Needs:
Training of the testers, training of the personnel
l) Approval:
Approval of the management, users.
m) Glossary:
All the vocabulary support, data dictionaries etc.
a) Objective: To automate the CSTE certification online
b) Identifier: CSTE-AUT/1/India
c) References: Records of earlier CSTE Exams held Names of the people involved in earlier exams held CBOK, exam material, application forms, certification
d) Test Items:
The forms of the site for application, re-scheduling, postponing exam etc.
On line tutorials.
Help support provided
CSTE main exam links
Connectivity.
Databases
e) Risks:
Recovery
Correctness
Security
Unauthorized access of the application
Databases Integrity
Redundancy, Configuration management,
portability, Maintainability
Ease of use, ease of operation.
f) Testing Strategy:
White box testing
Black box testing
Stress testing,
Performance testing
Configuration Management,
System testing, Security Testing,
(can add some more techniques for testing)
g) Acceptance Criteria:
Criteria for exit like the requirements specified by user are met fully, the site is fault free, and the application for the main exam is working fine. The functionality is fully satisfied.
h) Suspension or Reject Criteria:
Testing cannot progress if any of the test items do not satisfy all the input data is provided. Enough time is provided.
i) Schedules:
Schedules given here for each type of testing and reporting results.
j) Environmental Needs:
Tools required for testing. Connectivity provided. Recovery data provided.
k) Training Needs:
Training of the testers, training of the personnel
l) Approval:
Approval of the management, users.
m) Glossary:
All the vocabulary support, data dictionaries etc.
what can be done if software requirements are changing continuously?
1. Work with the project's stakeholders early on to understand how requirements might change so that alternate test plans and strategies can be worked out in advance, if possible.
2. It's helpful if the application's initial design allows for some adaptability so that later changes do not require redoing the application from scratch.
3. If the code is well-commented and well-documented this makes changes easier for the developers.
4. Use rapid prototyping whenever possible to help customers feel sure of their requirements and minimize changes.
5. The project's initial schedule should allow for some extra time commensurate with the possibility of changes.
6. Try to move new requirements to a 'Phase 2' version of an application, while using the original requirements for the 'Phase 1' version.
7. Negotiate to allow only easily-implemented new requirements into the project, while moving more difficult new requirements into future versions of the application.
8. Be sure that customers and management understand the scheduling impacts, inherent risks, and costs of significant requirements changes. Then let management or the customers (not the developers or testers) decide if the changes are warranted - after all, that's their job.
9. Balance the effort put into setting up automated testing with the expected effort required to re-do them to deal with changes.
10. Try to design some flexibility into automated test scripts.
11. Focus initial automated testing on application aspects that are most likely to remain unchanged.
12. Devote appropriate effort to risk analysis of changes to minimize regression testing needs.
13. Design some flexibility into test cases (this is not easily done; the best bet might be to minimize the detail in the test cases, or set up only higher-level generic-type test plans)
14. Focus less on detailed test plans and test cases and more on ad hoc testing (with an understanding of the added risk that this entails).
2. It's helpful if the application's initial design allows for some adaptability so that later changes do not require redoing the application from scratch.
3. If the code is well-commented and well-documented this makes changes easier for the developers.
4. Use rapid prototyping whenever possible to help customers feel sure of their requirements and minimize changes.
5. The project's initial schedule should allow for some extra time commensurate with the possibility of changes.
6. Try to move new requirements to a 'Phase 2' version of an application, while using the original requirements for the 'Phase 1' version.
7. Negotiate to allow only easily-implemented new requirements into the project, while moving more difficult new requirements into future versions of the application.
8. Be sure that customers and management understand the scheduling impacts, inherent risks, and costs of significant requirements changes. Then let management or the customers (not the developers or testers) decide if the changes are warranted - after all, that's their job.
9. Balance the effort put into setting up automated testing with the expected effort required to re-do them to deal with changes.
10. Try to design some flexibility into automated test scripts.
11. Focus initial automated testing on application aspects that are most likely to remain unchanged.
12. Devote appropriate effort to risk analysis of changes to minimize regression testing needs.
13. Design some flexibility into test cases (this is not easily done; the best bet might be to minimize the detail in the test cases, or set up only higher-level generic-type test plans)
14. Focus less on detailed test plans and test cases and more on ad hoc testing (with an understanding of the added risk that this entails).
Difference between Load testing and Stress testing?
A. Stress testing is subjecting a system to an unreasonable load
while denying it the resources (e.g., RAM, disc, mips, interrupts,etc.) needed to process that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in a
decent manner (e.g., not corrupting or losing data). Bugs and failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load (incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.
B. Load testing is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the system can sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.
C. Performance testing: - Performance testing is basically the process of understanding how the application and its operating environment respond at various user load levels. In general, we want to measure the latency, throughput, and utilization of resources.
while denying it the resources (e.g., RAM, disc, mips, interrupts,etc.) needed to process that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in a
decent manner (e.g., not corrupting or losing data). Bugs and failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load (incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.
B. Load testing is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the system can sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.
C. Performance testing: - Performance testing is basically the process of understanding how the application and its operating environment respond at various user load levels. In general, we want to measure the latency, throughput, and utilization of resources.
SOAP (Simple Object Access Protocol)
SOAP was originally developed by Microsoft, IBM, DevelopMentor and Userland Software and was then submitted to the Internet Engineering Task Force (IETF), who eventually made it an official recommendation. Though Microsoft camp seem to have played a major role in defining the specification of SOAP.
The motivation of developing this protocol was the widely used inter-application communication protocols like Microsoft’s DCOM protocol or Java’s RMI or OMG’s IIOP protocols which have till recent times fulfilled the needs of giving component services over an interconnected network .The richness of these protocols helped them gain a strong hold over distributed computing over controlled environment. Almost all these protocols need a smart client able to run the proxies of the server components; moreover, these protocols reveal their limitations when it comes to INTERNET. Internet cannot guarantee what kind of client and server will be operating at either end of the connection — it can only guarantee that they are both communicating via HTTP.
SOAP (Simple Object Access Protocol) is a simple solution for interaction of different applications built in different languages and running on different platforms as it uses HTTP as its transport and XML as its payload for sending and receiving messages. It’s a lightweight and a loosely coupled protocol for exchange of information in a decentralized and a distributed environment.
What’s so light in SOAP?
Soap relies on HTTP as a transport mechanism to send XML based messages, the messages are packed in what is called a SOAP envelop and send to the server to process in a Request/Response fashion. SOAP unlike proprietary protocols like DCOM or RMI does not require strong connection between client and the server and the SOAP messages are sting based messages passed from the Client to Server and vice versa in the form of SOAP envelops.
What makes SOAP loosely coupled?
Most proprietary protocols require the applications of the same breed to be running on both the ends, what if the server is implemented in a different programming language. The ability to access service of a component in a language/location and platform transparent manner reduces the tight coupling between the client and the server. SOAP enables “incompatible” systems to interoperate.
How does SOAP Message look like and How is it different from a method invoked on a Object?
To demonstrate how SOAP messages are different from method invocation on an component Let’s take a look at this ActiveX component, which accepts two integers and returns the total, as a return value would look something like this,
Server ActiveX Component (MyComponent.MathComponent)
Public Function Add (Num1 as integer, Num2 as integer)
Add = Num1 + Num2
End Function
The Client would create an instance of this component and would invoke the method to get the total.
Sub Button1_click()
Dim objAdd as new MyComponent.MathComponent
Dim intResult as integer
intResult = objAdd.Add(10,20)
end sub
This MyComponent.MathComponent would get instantiated and serve the request from the client application.
At a higher level a SOAP request to fulfill this request would look something like this
100
400
In order to ensure that server can correctly identify the method call it is wrapped up in a SOAP envelop
100
400
This is just an abstract example to give you a gist of the SOAP message.
The real SOAP message will contain more information like the component name and stuff.
This message is sent to a SOAP listener through HTTP in a Request/Response fashion.
The SOAP Listener accepts this SOAP message and does the processing on the component and returns the SOAP message back for the client.
In the Server you will need to implement a SOAP listener, wsdlgen.exe is a tool which comes with the SOAP Toolkit will create a SOAP listener and a WSDL (Web Service description Language) file for a given COM Component. For more information on WSDL visit http://www.msdn.microsoft.com/xml/general/wsdl.asp
OH GODDDDDDDD, do I need to Marshall and UN-Marshall the XML messages myself?
The answer is NO, The Microsoft SOAP Toolkit 2 will take care of the marshaling and UN-marshaling of the XML messages behind the scene for you.
The SOAP Toolkit also comes with a set of COM Component to access SOAP Web Services easily, Visual Basic SOAP client
Sub Button1_click()
Dim objSOAPClient As Object
Set objSOAPClient = CreateObject("MSSOAP.SoapClient")
objSOAPClient.mssoapinit "http://ServerName/VirtualDirectory/wsdlfile.WSDL"
MsgBox objSOAPClient.Add(100,400)
End sub
That’s it, the SOAP message marshaling and UN-marshaling will taken care by the SOAP client and what you will get is the result.
SOAP and Security
SOAP as it works on HTTP get the benefits of all the security that are currently available to HTTP, a SOAP message can pass through the firewall of the web server normally from port 80.
Limitations of SOAP
Though SOAP is evolving, and has a number of striking features, which are important in development of, distributed applications but still there are certain things which SOAP cannot support which I would like to address
SOAP is a simple protocol: As the name suggests it’s a simple protocol and works on HTTP, we cannot expect all the functionality’s offered by other protocols like DCOM or RMI.
SOAP may turn out to be slower compared to other proprietary protocols as it requires additional XML processing.
Still SOAP fares well in most areas compared to other wire protocol. As per the specifications, SOAP can also use other transport carrier like SMTP to transmit SOAP messages. Still lot of information on this topic in not available.
Summary
SOAP is a compelling and innovative use of available technologies that are not tied to a specific vendor. SOAP will play a major role in Microsoft’s “.NET” platform for building WebService. And Microsoft is expecting huge revenue to be generated from WebService
The motivation of developing this protocol was the widely used inter-application communication protocols like Microsoft’s DCOM protocol or Java’s RMI or OMG’s IIOP protocols which have till recent times fulfilled the needs of giving component services over an interconnected network .The richness of these protocols helped them gain a strong hold over distributed computing over controlled environment. Almost all these protocols need a smart client able to run the proxies of the server components; moreover, these protocols reveal their limitations when it comes to INTERNET. Internet cannot guarantee what kind of client and server will be operating at either end of the connection — it can only guarantee that they are both communicating via HTTP.
SOAP (Simple Object Access Protocol) is a simple solution for interaction of different applications built in different languages and running on different platforms as it uses HTTP as its transport and XML as its payload for sending and receiving messages. It’s a lightweight and a loosely coupled protocol for exchange of information in a decentralized and a distributed environment.
What’s so light in SOAP?
Soap relies on HTTP as a transport mechanism to send XML based messages, the messages are packed in what is called a SOAP envelop and send to the server to process in a Request/Response fashion. SOAP unlike proprietary protocols like DCOM or RMI does not require strong connection between client and the server and the SOAP messages are sting based messages passed from the Client to Server and vice versa in the form of SOAP envelops.
What makes SOAP loosely coupled?
Most proprietary protocols require the applications of the same breed to be running on both the ends, what if the server is implemented in a different programming language. The ability to access service of a component in a language/location and platform transparent manner reduces the tight coupling between the client and the server. SOAP enables “incompatible” systems to interoperate.
How does SOAP Message look like and How is it different from a method invoked on a Object?
To demonstrate how SOAP messages are different from method invocation on an component Let’s take a look at this ActiveX component, which accepts two integers and returns the total, as a return value would look something like this,
Server ActiveX Component (MyComponent.MathComponent)
Public Function Add (Num1 as integer, Num2 as integer)
Add = Num1 + Num2
End Function
The Client would create an instance of this component and would invoke the method to get the total.
Sub Button1_click()
Dim objAdd as new MyComponent.MathComponent
Dim intResult as integer
intResult = objAdd.Add(10,20)
end sub
This MyComponent.MathComponent would get instantiated and serve the request from the client application.
At a higher level a SOAP request to fulfill this request would look something like this
In order to ensure that server can correctly identify the method call it is wrapped up in a SOAP envelop
This is just an abstract example to give you a gist of the SOAP message.
The real SOAP message will contain more information like the component name and stuff.
This message is sent to a SOAP listener through HTTP in a Request/Response fashion.
The SOAP Listener accepts this SOAP message and does the processing on the component and returns the SOAP message back for the client.
In the Server you will need to implement a SOAP listener, wsdlgen.exe is a tool which comes with the SOAP Toolkit will create a SOAP listener and a WSDL (Web Service description Language) file for a given COM Component. For more information on WSDL visit http://www.msdn.microsoft.com/xml/general/wsdl.asp
OH GODDDDDDDD, do I need to Marshall and UN-Marshall the XML messages myself?
The answer is NO, The Microsoft SOAP Toolkit 2 will take care of the marshaling and UN-marshaling of the XML messages behind the scene for you.
The SOAP Toolkit also comes with a set of COM Component to access SOAP Web Services easily, Visual Basic SOAP client
Sub Button1_click()
Dim objSOAPClient As Object
Set objSOAPClient = CreateObject("MSSOAP.SoapClient")
objSOAPClient.mssoapinit "http://ServerName/VirtualDirectory/wsdlfile.WSDL"
MsgBox objSOAPClient.Add(100,400)
End sub
That’s it, the SOAP message marshaling and UN-marshaling will taken care by the SOAP client and what you will get is the result.
SOAP and Security
SOAP as it works on HTTP get the benefits of all the security that are currently available to HTTP, a SOAP message can pass through the firewall of the web server normally from port 80.
Limitations of SOAP
Though SOAP is evolving, and has a number of striking features, which are important in development of, distributed applications but still there are certain things which SOAP cannot support which I would like to address
SOAP is a simple protocol: As the name suggests it’s a simple protocol and works on HTTP, we cannot expect all the functionality’s offered by other protocols like DCOM or RMI.
SOAP may turn out to be slower compared to other proprietary protocols as it requires additional XML processing.
Still SOAP fares well in most areas compared to other wire protocol. As per the specifications, SOAP can also use other transport carrier like SMTP to transmit SOAP messages. Still lot of information on this topic in not available.
Summary
SOAP is a compelling and innovative use of available technologies that are not tied to a specific vendor. SOAP will play a major role in Microsoft’s “.NET” platform for building WebService. And Microsoft is expecting huge revenue to be generated from WebService
Black Box Testing
Strictly speaking, black box testing involves testing a system as a whole, not as a
system of connected parts. Realistically, however, black box testing still provides some differentiation between clients, the server, and the network connecting them. This
means that when performing black box testing, it is possible to determine whether time
is being spent in the client, the network, or the server, and hence determine where
bottlenecks lie.
The limitation of this approach is that knowing, for example, that the server is the
bottleneck of a system is not necessarily sufficient. Is the bottleneck caused by a lack of memory, a lack of CPU performance, poor algorithms, or one of many other causes? To answer these questions, diagnostics need to be performed on the server while
generating a load.
system of connected parts. Realistically, however, black box testing still provides some differentiation between clients, the server, and the network connecting them. This
means that when performing black box testing, it is possible to determine whether time
is being spent in the client, the network, or the server, and hence determine where
bottlenecks lie.
The limitation of this approach is that knowing, for example, that the server is the
bottleneck of a system is not necessarily sufficient. Is the bottleneck caused by a lack of memory, a lack of CPU performance, poor algorithms, or one of many other causes? To answer these questions, diagnostics need to be performed on the server while
generating a load.
Load/Stress Testing
Load and stress testing are two different, but related forms of testing. Stress testing involves testing a system under extreme cases, which will rarely occur under normalconditions. Such testing can involve such scenarios as running many users
concurrently for short times, short transactions repeated quickly, and extraordinarily long transactions.
Load testing places a more realistic load on the system. Scenarios are modelled on the demand that the system is expected to face under real conditions. Load testing starts by placing a low demand on the system, and gradually increasing the load. This process allows measuring the performance of the system under different loads, as well as determining the maximum load that under which the system can respond acceptably.
Load and stress testing are clearly two interrelated processes, generally performed
concurrently.It is important to note what load/stress testing isn’t. They are not functional tests. In order to perform performance testing, it must be assumed that a reasonably stable and functional system is in use, otherwise results are meaningless. With that said, it is still common to uncover functional problems in a system while performing load testing.
However, uncovering excessive functional problems can be a sign of inadequate
functional testing.
concurrently for short times, short transactions repeated quickly, and extraordinarily long transactions.
Load testing places a more realistic load on the system. Scenarios are modelled on the demand that the system is expected to face under real conditions. Load testing starts by placing a low demand on the system, and gradually increasing the load. This process allows measuring the performance of the system under different loads, as well as determining the maximum load that under which the system can respond acceptably.
Load and stress testing are clearly two interrelated processes, generally performed
concurrently.It is important to note what load/stress testing isn’t. They are not functional tests. In order to perform performance testing, it must be assumed that a reasonably stable and functional system is in use, otherwise results are meaningless. With that said, it is still common to uncover functional problems in a system while performing load testing.
However, uncovering excessive functional problems can be a sign of inadequate
functional testing.
Misc Loadrunner Questions
• Hits per second graph
The Hits per Second graph shows the number of HTTP requests made by Vusers to the Web server during each second of the scenario run. This graph helps you evaluate the amount of load Vusers generate, in terms of the number of hits.
• Pages download per second graph
The Pages Downloaded per Second graph shows the number of Web pages (y-axis) downloaded from the server during each second of the scenario run (x-axis). This graph helps you evaluate the amount of load Vusers generate, in terms of the number of pages downloaded.
• Transaction Response time (under load) graph
The Transaction Response Time (Under Load) graph is a combination of the Running Vusers and Average Transaction Response Time graphs and indicates transaction times relative to the number of Vusers running at any given point during the scenario. This graph helps you view the general impact of Vuser load on performance time and is most useful when analyzing a scenario with a gradual load.
• Transaction Response time (percentile) graph
The Transaction Response Time (Percentile) graph analyzes the percentage of transactions that were performed within a given time range. This graph helps you determine the percentage of transactions that met the performance criteria defined for your system.
• Network delay time graph
The Network Delay Time graph shows the delays for the complete path between the source and destination machines (for example, the database server and Vuser load generator).
The Hits per Second graph shows the number of HTTP requests made by Vusers to the Web server during each second of the scenario run. This graph helps you evaluate the amount of load Vusers generate, in terms of the number of hits.
• Pages download per second graph
The Pages Downloaded per Second graph shows the number of Web pages (y-axis) downloaded from the server during each second of the scenario run (x-axis). This graph helps you evaluate the amount of load Vusers generate, in terms of the number of pages downloaded.
• Transaction Response time (under load) graph
The Transaction Response Time (Under Load) graph is a combination of the Running Vusers and Average Transaction Response Time graphs and indicates transaction times relative to the number of Vusers running at any given point during the scenario. This graph helps you view the general impact of Vuser load on performance time and is most useful when analyzing a scenario with a gradual load.
• Transaction Response time (percentile) graph
The Transaction Response Time (Percentile) graph analyzes the percentage of transactions that were performed within a given time range. This graph helps you determine the percentage of transactions that met the performance criteria defined for your system.
• Network delay time graph
The Network Delay Time graph shows the delays for the complete path between the source and destination machines (for example, the database server and Vuser load generator).
Why wouldn’t you want to run virtual users on the same host as the Load- Runner Controller or Database Server?
Running virtual users on the same host as the LoadRunner Controller will skew the results so that they no longer emulate real life usage. By having both the Controller and the Vusers on the same machine, the tester will not be able to determine the effects of the network traffic.
What is difference between manual scenario and Goal oriented scenario? What Goal Oriented scenarios can be created?
Manual scenario:
–Main purpose is to learn how many Vusers can run concurrently
– Gives you manual control over how many Vusers run and at what times
Goal oriented scenario:
– Goal may be throughput, response time, or number of concurrent Vusers
– LoadRunner manages Vusers automatically
–
Different Goal Oriented Scenarios are:
• Virtual Users
• Hits per second
• Transaction per second
• Transaction Response time
• Pages per minute
–Main purpose is to learn how many Vusers can run concurrently
– Gives you manual control over how many Vusers run and at what times
Goal oriented scenario:
– Goal may be throughput, response time, or number of concurrent Vusers
– LoadRunner manages Vusers automatically
–
Different Goal Oriented Scenarios are:
• Virtual Users
• Hits per second
• Transaction per second
• Transaction Response time
• Pages per minute
Where should the rendezvous be placed in the script?
The rendezvous should be placed immediately before the transaction where you want to create peak load. In this case, the rendezvous should be placed right before starting the Update Order transaction.
What do you need to do to be able to view parameter substitution in the Execution Log?
Check extended log and Parameter substitution in the Run-Time Settings.
When does VuGen record a web_submit_data instead of a web_submit_form? Why? (Be as specific as possible)
A web_submit_data is recorded when VuGen cannot match the action, method, data fields, and/or hidden data values with the page that is stored in the record proxy cache. Comparison failures are typically caused by something other than HTML setting the properties of the HTTP request. Because VuGen can parse only HTML, it cannot find all the properties of the HTTP request in memory. This results in the hard-coding of all the request information in a web_submit_data statement.
How can caching negatively affect load testing results?
When data is cached in the server’s memory, the server does not need to fetch it from the database during playback. Then, test results do not reflect the same performance they would if real users were loading the system with different data.
How can reusing the same data during iterative execution of a business process negatively affect load testing results?
In reusing the same data for each iteration, the server recognizes the same data is requested and places it in its cache. The load test then gets performance results that are not based on real server activity but caching. This will not provide the correct results during the analysis of the load test.
What are the benefits of multiple Action files within a Vuser?
They allow you to perform different business processes in one Vuser to represent a real user who does the same thing. They let you build Vusers that emulate real users defined in the User Community Profile. They also allow you to record the login and logoff separately from the Action files and thus to avoid iteration.
What is a function to capture dynamic values in the web vuser script?
Web_reg_save_param function saves dynamic data information to a parameter.
1. What are the three sections of a Vuser script and what is the purpose of each one?
1) Vuser_init - used for recording the logon.
2) Actions - used for recording the business process.
3) Vuser_end - used for recording the logoff.
2. For what purpose are Vusers created?
Vusers are created to emulate real users acting on the server for the purpose of load testing.
1. What are the three sections of a Vuser script and what is the purpose of each one?
1) Vuser_init - used for recording the logon.
2) Actions - used for recording the business process.
3) Vuser_end - used for recording the logoff.
2. For what purpose are Vusers created?
Vusers are created to emulate real users acting on the server for the purpose of load testing.
Where do you set automatic correlation options?
Automatic correlation from web point of view, can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation.
Automatic correlation for database, can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
Automatic correlation for database, can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
What is correlation? Explain the difference between automatic correlation and manual correlation?
Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.
Types of Goals in Goal-Oriented Scenario
Load Runner provides you with five different types of goals in a goal oriented scenario:
• The number of concurrent Vusers
• The number of hits per second
• The number of transactions per second
• The number of pages per minute
• The transaction response time that you want your scenario
Analysis Scenario (Bottlenecks):
In Running Vuser graph correlated with the response time graph you can see that as the number of Vusers increases, the average response time of the check itinerary transaction very gradually increases. In other words, the average response time steadily increases as the load increases.
At 56 Vusers, there is a sudden, sharp increase in the average response time.
We say that the test broke the server. That is the mean time before failure (MTBF). The response time clearly began to degrade when there were more than 56 Vusers running simultaneously.
• The number of concurrent Vusers
• The number of hits per second
• The number of transactions per second
• The number of pages per minute
• The transaction response time that you want your scenario
Analysis Scenario (Bottlenecks):
In Running Vuser graph correlated with the response time graph you can see that as the number of Vusers increases, the average response time of the check itinerary transaction very gradually increases. In other words, the average response time steadily increases as the load increases.
At 56 Vusers, there is a sudden, sharp increase in the average response time.
We say that the test broke the server. That is the mean time before failure (MTBF). The response time clearly began to degrade when there were more than 56 Vusers running simultaneously.
What is the difference between standard log and extended log?
The standard log sends a subset of functions and messages sent during script execution to a log. The subset depends on the Vuser type
Extended log sends a detailed script execution messages to the output log. This is mainly used during debugging when we want information about:
a) Parameter substitution
b) Data returned by the server
c) Advanced trace
Extended log sends a detailed script execution messages to the output log. This is mainly used during debugging when we want information about:
a) Parameter substitution
b) Data returned by the server
c) Advanced trace
What is think time? How do you change the threshold?
Think time is the time that a real user waits between actions.
Example:
When a user receives data from a server, the user may wait several seconds to review the data before responding. This delay is known as the think time.
Changing the Threshold:
Threshold level is the level below which the recorded think time will be ignored. The default value is five (5) seconds. We can change the think time threshold in the Recording options of the Vugen.
Example:
When a user receives data from a server, the user may wait several seconds to review the data before responding. This delay is known as the think time.
Changing the Threshold:
Threshold level is the level below which the recorded think time will be ignored. The default value is five (5) seconds. We can change the think time threshold in the Recording options of the Vugen.
How did you plan the Load? What are the Criteria?
Load test is planned to decide the number of users, what kind of machines we are going to use and from where they are run. It is based on 2 important documents, Task Distribution Diagram and Transaction profile. Task Distribution Diagram gives us the information on number of users for a particular transaction and the time of the load. The peak usage and off-usage are decided from this Diagram. Transaction profile gives us the information about the transactions name and their priority levels with regard to the scenario we are deciding.
What is the difference between Overlay graph and Correlate graph?
Overlay Graph: It overlay the content of two graphs that shares a common x-axis. Left Y-axis on the merged graph show’s the current graph’s value & Right Y-axis show the value of Y-axis of the graph that was merged.
Correlate Graph: Plot the Y-axis of two graphs against each other. The active graph’s Y-axis becomes X-axis of merged graph. Y-axis of the graph that was merged becomes merged graph’s Y-axis.
Correlate Graph: Plot the Y-axis of two graphs against each other. The active graph’s Y-axis becomes X-axis of merged graph. Y-axis of the graph that was merged becomes merged graph’s Y-axis.
How did you find database related issues?
By running “Database” monitor and help of “Data Resource Graph” we can find database related issues. E.g. You can specify the resource you want to measure on before running the controller and than you can see database related issues
How did you find web server related issues?
Using Web resource monitors we can find the performance of web servers. Using these monitors we can analyze throughput on the webserver, number of hits per second that occured during scenario, the number of http responses per second, the number of downloaded pages per second.
If web server, database and Network are all fine where could be the problem?
The problem could be in the system itself or in the application server or in the code written for the application.
How do you identify the performance bottlenecks?
Performance Bottlenecks can be detected by using monitors. These monitors might be application server monitors, web server monitors, database server monitors and network monitors. They help in finding out the troubled area in our scenario which causes increased response time. The measurements made are usually performance response time, throughput, hits/sec, network delay graphs, etc.
Explain the Configuration of your systems?
The configuration of our systems refers to that of the client machines on which we run the Vusers. The configuration of any client machine includes its hardware settings, memory, operating system, software applications, development tools, etc. This system component configuration should match with the overall system configuration that would include the network infrastructure, the web server, the database server, and any other components that go with this larger system so as to achieve the load testing objectives.
What is the relation between Response Time and Throughput?
The Throughput graph shows the amount of data in bytes that the Vusers received from the server in a second. When we compare this with the transaction response time, we will notice that as throughput decreased, the response time also decreased. Similarly, the peak throughput and highest response time would occur approximately at the same time.
If you want to stop the execution of your script on error, how do you do that?
The lr_abort function aborts the execution of a Vuser script. It instructs the Vuser to stop executing the Actions section, execute the vuser_end section and end the execution. This function is useful when you need to manually abort a script execution as a result of a specific error condition. When you end a script using this function, the Vuser is assigned the status “Stopped”. For this to take effect, we have to first uncheck the “Continue on error” option in Run-Time Settings.
What is the advantage of running the Vuser as thread?
VuGen provides the facility to use multithreading. This enables more Vusers to be run per generator.
If the Vuser is run as a process, the same driver program is loaded into memory for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers that can be run on a single generator.
If the Vuser is run as a thread, only one instance of the driver program is loaded into memory for the given number of Vusers(say 100). Each thread shares the memory of the parent driver program, thus enabling more Vusers to be run per generator.
If the Vuser is run as a process, the same driver program is loaded into memory for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers that can be run on a single generator.
If the Vuser is run as a thread, only one instance of the driver program is loaded into memory for the given number of Vusers(say 100). Each thread shares the memory of the parent driver program, thus enabling more Vusers to be run per generator.
What is Ramp up? How do you set this?
This option is used to gradually increase the amount of Vusers/load on the server. An initial value is set and a value to wait between intervals can be specified.
To set Ramp Up, go to ‘Scenario Scheduling Options’
To set Ramp Up, go to ‘Scenario Scheduling Options’
Monday, January 26, 2009
How do you perform functional testing under load?
Functionality under load can be tested by running several Vusers concurrently. By increasing the amount of Vusers, we can determine how much load the server can sustain.
Where do you set Iteration for Vuser testing?
We set Iterations in the Run Time Settings of the VuGen. The navigation for this is Run time settings, Pacing tab, set number of iterations.
What are the changes you can make in run-time settings?
The Run Time Settings that we make are:
a) Pacing - It has iteration count.
b) Log - Under this we have Disable Logging Standard Log and Extended.
c) Think Time - In think time we have two options like Ignore think time and Replay think time.
d) General - Under general tab we can set the vusers as process or as multithreading and whether each step as a transaction.
a) Pacing - It has iteration count.
b) Log - Under this we have Disable Logging Standard Log and Extended.
c) Think Time - In think time we have two options like Ignore think time and Replay think time.
d) General - Under general tab we can set the vusers as process or as multithreading and whether each step as a transaction.
How do you write user defined functions in LR? Give me few functions you wrote in your previous project?
Before we create the User Defined functions we need to create the external library(DLL) with the function. We add this library to VuGen bin directory. Once the library is added then we assign user defined function as a parameter. The function should have the following format:
__declspec (dllexport) char* (char*, char*)
Examples of user defined functions are as follows:
GetVersion, GetCurrentTime, GetPltform are some of the user defined functions used in my earlier project.
__declspec (dllexport) char* (char*, char*)
Examples of user defined functions are as follows:
GetVersion, GetCurrentTime, GetPltform are some of the user defined functions used in my earlier project.
How do you debug a LoadRunner script?
VuGen contains two options to help debug Vuser scripts-the Run Step by Step command and breakpoints. The Debug settings in the Options dialog box allow us to determine the extent of the trace to be performed during scenario execution. The debug information is written to the Output window.
We can manually set the message class within your script using the lr_set_debug_message function. This is useful if we want to receive debug information about a small section of the script only.
We can manually set the message class within your script using the lr_set_debug_message function. This is useful if we want to receive debug information about a small section of the script only.
When do you disable log in Virtual User Generator, When do you choose standard and extended logs?
Once we debug our script and verify that it is functional, we can enable logging for errors only. When we add a script to a scenario, logging is automatically disabled.
Standard Log Option:
When you select Standard log, it creates a standard log of functions and messages sent during script execution to use for debugging. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled
Extended Log Option:
Select extended log to create an extended log, including warnings and other messages. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled. We can specify which additional information should be added to the extended log using the Extended log options.
Standard Log Option:
When you select Standard log, it creates a standard log of functions and messages sent during script execution to use for debugging. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled
Extended Log Option:
Select extended log to create an extended log, including warnings and other messages. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled. We can specify which additional information should be added to the extended log using the Extended log options.
What is a function to capture dynamic values in the web Vuser script?
Web_reg_save_param function saves dynamic data information to a parameter.
Where do you set automatic correlation options?
Automatic correlation from web point of view can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation.
Automatic correlation for database can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
Automatic correlation for database can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.
How do you find out where correlation is required? Give few examples from your projects?
Two ways:
First we can scan for correlations, and see the list of values which can be “` correlated. From this we can pick a value to be correlated. Secondly, we can record two scripts and compare them. We can look up the difference file to see for the values which needed to be correlated.
In my project, there was a unique id developed for each customer, it was nothing but Insurance Number, it was generated automatically and it was sequential and this value was unique. I had to correlate this value, in order to avoid errors while running my script. I did using scan for correlation.
First we can scan for correlations, and see the list of values which can be “` correlated. From this we can pick a value to be correlated. Secondly, we can record two scripts and compare them. We can look up the difference file to see for the values which needed to be correlated.
In my project, there was a unique id developed for each customer, it was nothing but Insurance Number, it was generated automatically and it was sequential and this value was unique. I had to correlate this value, in order to avoid errors while running my script. I did using scan for correlation.
What is correlation? Explain the difference between automatic correlation and manual correlation?
Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.
Why do you create parameters?
Parameters are like script variables. They are used to vary input to the server and to emulate real users.
a) Different sets of data are sent to the server each time the script is run.
b) Better simulate the usage model for more accurate testing from the Controller; one script can emulate many different users on the system.
a) Different sets of data are sent to the server each time the script is run.
b) Better simulate the usage model for more accurate testing from the Controller; one script can emulate many different users on the system.
Explain the recording mode for web Vuser script?
We use VuGen to develop a Vuser script by recording a user performing typical business processes on a client application. VuGen creates the script by recording the activity between the client and the server. For example, in web based applications, VuGen monitors the client end of the database and traces all the requests sent to, and received from, the database server.
We use VuGen to:
a) Monitor the communication between the application and the server;
b) Generate the required function calls; and
c) Insert the generated function calls into a Vuser script.
We use VuGen to:
a) Monitor the communication between the application and the server;
b) Generate the required function calls; and
c) Insert the generated function calls into a Vuser script.
What is a scenario?
A scenario defines the events that occur during each testing session. For example, a scenario defines and controls the number of users to emulate, the actions to be performed, and the machines on which the virtual users run their emulations
What is a rendezvous point?
You insert rendezvous points into Vuser scripts to emulate heavy user load on the server. Rendezvous points instruct Vusers to wait during test execution for multiple Vusers to arrive at a certain point, in order that they may simultaneously perform a task. For example, to emulate peak load on the bank server, you can insert a rendezvous point instructing 100 Vusers to deposit cash into their accounts at the same time.
What Component of LoadRunner would you use to play Back the script in multi user mode?
The Controller component is used to playback the script in multi-user mode. This is done during a scenario run where a vuser script is executed by a number of vusers in a group.
What Component of LoadRunner would you use to record a Script?
The Virtual User Generator (VuGen) component is used to record a script. It enables you to develop Vuser scripts for a variety of application types and communication protocols.
What are the components of LoadRunner?
The components of LoadRunner are The Virtual User Generator, Controller, and the Agent process, LoadRunner Analysis and Monitoring, LoadRunner Books Online.
When do you do load and performance Testing?
We perform load testing once we are done with interface (GUI) testing. Modern system architectures are large and complex. Whereas single user testing primarily on functionality and user interface of a system component, application testing focuses on performance and reliability of an entire system. For example, a typical application-testing scenario might depict 1000 users logging in simultaneously to a system. This gives rise to issues such as what is the response time of the system, does it crash, will it go with different software applications and platforms, can it hold so many hundreds and thousands of users, etc. This is when we set do load and performance testing.
Explain the Load testing process?
step1: Planning the test
Here, we develop a clearly defined test plan to ensure the test scenarios we develop will accomplish load-testing objectives.
step2: Creating Vusers
Here, we create Vuser scripts that contain tasks performed by each Vuser, tasks performed by Vusers as a whole, and tasks measured as transactions.
step3: Creating the scenario
A scenario describes the events that occur during a testing session. It includes a list of machines, scripts, and Vusers that run during the scenario. We create scenarios using LoadRunner Controller. We can create manual scenarios as well as goal-oriented scenarios. In manual scenarios, we define the number of Vusers, the load generator machines, and percentage of Vusers to be assigned to each script. For web tests, we may create a goal-oriented scenario where we define the goal that our test has to achieve. LoadRunner automatically builds a scenario for us.
step4: Running the scenario
We emulate load on the server by instructing multiple Vusres to perform tasks simultaneously. Before the testing, we set the scenario configuration and scheduling. We can run the entire scenario, Vuser groups, or individual Vusers.
step5: Monitoring the scenario
We monitor scenario execution using the LoadRunner online runtime, transaction, system resource, Web resource, Web server resource, Web application server resource, database server resource, network delay, streaming media resource, firewall server resource, ERP server resource, and Java performance monitors.
step6: Analyzing test results
During scenario execution, LoadRunner records the performance of the application under different loads. We use LoadRunner’s graphs and reports to analyze the application’s performance.
Here, we develop a clearly defined test plan to ensure the test scenarios we develop will accomplish load-testing objectives.
step2: Creating Vusers
Here, we create Vuser scripts that contain tasks performed by each Vuser, tasks performed by Vusers as a whole, and tasks measured as transactions.
step3: Creating the scenario
A scenario describes the events that occur during a testing session. It includes a list of machines, scripts, and Vusers that run during the scenario. We create scenarios using LoadRunner Controller. We can create manual scenarios as well as goal-oriented scenarios. In manual scenarios, we define the number of Vusers, the load generator machines, and percentage of Vusers to be assigned to each script. For web tests, we may create a goal-oriented scenario where we define the goal that our test has to achieve. LoadRunner automatically builds a scenario for us.
step4: Running the scenario
We emulate load on the server by instructing multiple Vusres to perform tasks simultaneously. Before the testing, we set the scenario configuration and scheduling. We can run the entire scenario, Vuser groups, or individual Vusers.
step5: Monitoring the scenario
We monitor scenario execution using the LoadRunner online runtime, transaction, system resource, Web resource, Web server resource, Web application server resource, database server resource, network delay, streaming media resource, firewall server resource, ERP server resource, and Java performance monitors.
step6: Analyzing test results
During scenario execution, LoadRunner records the performance of the application under different loads. We use LoadRunner’s graphs and reports to analyze the application’s performance.
What is Performance testing?
Timing for both read and update transactions should be gathered to determine whether system functions are being performed in an acceptable timeframe. This should be done standalone and then in a multi user environment to determine the effect of multiple transactions on the timing of a single transaction.
What is load testing?
Load testing is to test that if the application works fine with the loads that result from large number of simultaneous users, transactions and to determine weather it can handle peak usage periods.
Subscribe to:
Posts (Atom)