Panel Discussion: Real life testing challenges and how to overcome them

Wednesday, September 6, 2006

at Countrywide, 2900 Madera, Simi Valley, CA 93065

Join us for a panel discussion with experienced testers who will discuss real life testing challenges and how to overcome them. We will discuss manual and automated testing, from starting a testing program, to maintaining and improving an existing test program. Our panel will consist of members of the SCQAA board representing over 50 years of testing experience.

In order to make this meeting worthwhile we would like to encourage attendee interaction and participation. We are asking for your questions about testing before hand so that we are sure to address the issues that quality professionals are struggling with now in their areas of responsibility. We will wrap-up with a free form question and answer session to round out the night. This is the time to let us know what kind of programs should be planned for next year.

Suggestions for future topics are always welcomed. Come and be part of the lively and informative discussion!

The SCQAA-SF Chapter President, Judy Cheves, will facilitate this session. Judy began her career in IT as a Project Manager specializing in allocating shared resources. She transitioned to Quality Assurance to learn how to mitigate quality related issues on projects - rework, schedule slippage, budget overruns.

She began her consulting career in QA and QC with clients ranging from commercial software development, utilities, insurance, private security & investigation, automotive to entertainment. For the past 7 years, she has specialized in building efficient, cost effective QC test teams. She is currently 1st Vice President of QC at Countrywide Finance Corporation providing testing services to various business units within the company.

Judy is a founding member of the San Fernando Chapter of SCQAA and has been President of the chapter since it's inception. She is past president of the Los Angeles SCQAA Chapter, where she served on the board for 4 years.

Submitted Questions

1. This isn't a simple testing question, but it is a real-life testing challenge:
   We have been tasked with regression testing, automated where possible. The difficulty comes in the repeatability of the tests. The test cases available vary with time, as do the results. The test case we ran two months ago may not be valid now, due to changes in the data or just the fact that time has passed. Even if the test case still matches the scenario, the results may be different because of new data to be included, time-dependent business rules, or just that it's two months later.
   For our current automated testing of a replacement system, we run huge volumes of test data against both the new and old systems. The automated test scripts ignore the correctness of the results. The test input and output are captured in a standard format. After both sets of tests are run, the inputs and outputs of the test cases are compared and reconciled. Any differences are analyzed to determine their magnitude, significance, and causes. When the new system goes live, we will no longer have a legacy system to use as an oracle. The challenge is to find a way to do automated and scripted manual testing in these conditions. We're sure we're not the only testers to have encountered such a situation.
2. How do you accurately estimate testing effort? Similarly, how is capacity determined?
3. Who has the final say if a product is to go into production with serious defects?
4. Who sets the priority on competing products for QC resources?
5. What is the most common issue surrounding you QC efforts?
6. How can we deal with PM's who have limited information on testing processes and underestimate testing effort? (btw, they usually don't want to know, either)
7. Are you happy with your test case repository? (TD, Excel, Word, but none are satisfactory)
8. What is the best way to improve test coverage while still meeting project deadlines?
9. How do you measure test coverage?
10. Bonus question: How do you improve the test documentation while still meeting project deadlines?