CISSP_Candidate
08-29-2006, 11:58 AM
First off I would like to say that I have found this forum/site very useful.
So here is my question:
Question 653 on the Quiz:
Which of the following would provide the best stress testing environment?
Test environment using test data.
Test environment using live workloads.
Production environment using test data.
Production environment using live workloads.
According to the quiz the correct answer “ test environment using live workloads.”
Which I chose and agree with.
It goes on to say:
Details
Stress testing is carried out to ensure a system can cope with production workloads, but as it may be tested to destruction, a test environment should always be used to avoid damaging the production environment. Hence, testing should never take place in a production environment. If only test data is used, there is no certainty that the system was adequately stress tested.
THEN
“Extra information: The CISSP Prep guide offers a very different opinion of what types of data should go into the stress testing environment and highlights the fact that abnormal conditions should also be tested: "Testing of the software module or unit testing should be addressed when the module are being designed. Personnel separate from the programmers should conduct this testing. The test data is part of the specifications. Testing should not only check the modules using normal and valid input data, but it should also check for incorrect types, out of range values, and other bounds and/or conditions. Live or actual field is not recommended for use in the testing procedures because both data types may not cover out of range situations and the correct outputs of the test are unknown. Special test suites of data that exercise all paths of the software to the fullest extent possible and whose correct resulting outputs are known before hand should be used."
Source: Information Systems Audit and Control Association, Certified Information Systems Auditor 2002 review manual, chapter 6: Business Application System Development, Acquisition, Implementation and Maintenance (page 299).
And: KRUTZ, Ronald L. & VINES, Russel D., The CISSP Prep Guide: Mastering the Ten Domains of Computer Security, 2001, John Wiley & Sons, Page 251.
“
First I am not trying to complain in any way about the quiz. I am just honestly confused by this type of thing. Perhaps it is the nature of the beast (multiple choice type testing). But this question has been nagging me. My point is that I agree with the quiz answer BUT the blurb at the end confuses me because it implies that we should answer:
a “test environment using test data”.
I used to work for a company that made protocol analyzers and network management hardware and software. I didn’t work on the software development side but I know that we used captured live network data for testing (from customers, with their permission of course). We also tested beta equipment on customers live networks. This may have been unique case because much of the testing (protocol capture & post analysis) was passive in nature. In essence we were using "live" data.
And S. Harris 3rd Edition, Chapter 11, P 842 says the following: “ the product should be tested in various environments with different applications...”
I am in no way trying to complain or defame the site/quizzing system or anyone(contributors etc) for that matter. I am just find this type of thing confusing. I am not saying the answer is wrong either. I was just looking for input, comments etc. Did anyone else find this question troublesome?
Perhaps this is like the SSL question (which layer…) or just an issue about semantics( i.e. what does “live” really mean).
So here is my question:
Question 653 on the Quiz:
Which of the following would provide the best stress testing environment?
Test environment using test data.
Test environment using live workloads.
Production environment using test data.
Production environment using live workloads.
According to the quiz the correct answer “ test environment using live workloads.”
Which I chose and agree with.
It goes on to say:
Details
Stress testing is carried out to ensure a system can cope with production workloads, but as it may be tested to destruction, a test environment should always be used to avoid damaging the production environment. Hence, testing should never take place in a production environment. If only test data is used, there is no certainty that the system was adequately stress tested.
THEN
“Extra information: The CISSP Prep guide offers a very different opinion of what types of data should go into the stress testing environment and highlights the fact that abnormal conditions should also be tested: "Testing of the software module or unit testing should be addressed when the module are being designed. Personnel separate from the programmers should conduct this testing. The test data is part of the specifications. Testing should not only check the modules using normal and valid input data, but it should also check for incorrect types, out of range values, and other bounds and/or conditions. Live or actual field is not recommended for use in the testing procedures because both data types may not cover out of range situations and the correct outputs of the test are unknown. Special test suites of data that exercise all paths of the software to the fullest extent possible and whose correct resulting outputs are known before hand should be used."
Source: Information Systems Audit and Control Association, Certified Information Systems Auditor 2002 review manual, chapter 6: Business Application System Development, Acquisition, Implementation and Maintenance (page 299).
And: KRUTZ, Ronald L. & VINES, Russel D., The CISSP Prep Guide: Mastering the Ten Domains of Computer Security, 2001, John Wiley & Sons, Page 251.
“
First I am not trying to complain in any way about the quiz. I am just honestly confused by this type of thing. Perhaps it is the nature of the beast (multiple choice type testing). But this question has been nagging me. My point is that I agree with the quiz answer BUT the blurb at the end confuses me because it implies that we should answer:
a “test environment using test data”.
I used to work for a company that made protocol analyzers and network management hardware and software. I didn’t work on the software development side but I know that we used captured live network data for testing (from customers, with their permission of course). We also tested beta equipment on customers live networks. This may have been unique case because much of the testing (protocol capture & post analysis) was passive in nature. In essence we were using "live" data.
And S. Harris 3rd Edition, Chapter 11, P 842 says the following: “ the product should be tested in various environments with different applications...”
I am in no way trying to complain or defame the site/quizzing system or anyone(contributors etc) for that matter. I am just find this type of thing confusing. I am not saying the answer is wrong either. I was just looking for input, comments etc. Did anyone else find this question troublesome?
Perhaps this is like the SSL question (which layer…) or just an issue about semantics( i.e. what does “live” really mean).