EMIS 7307 T&E Part 5 Test Planning and Analysis Landing distance of an F-18 1 EMIS 7307 T&E Part 5 Test Planning and Analysis 2 EMIS 7307 T&E Part 5 Test Planning and Analysis 3 EMIS 7307 T&E Part 5 Test Planning and Analysis • What if 60% isn’t sufficient confidence? – Typically 95% is the desired confidence. • Need more data points! – How many? – Can’t say for sure. – Depends on s and x with the new data. 4 EMIS 7307 T&E Part 5 Test Planning and Analysis Let’s assume that if we collect more data the standard deviation and (x -V*) don’t change. This not a very good assumption, but possibly the best we can do. Then we can rewrite ts as ts= k *√(n-1). With this assumption k = .351. For: n =2, ts= .351, α=.39, (1-α) = .61 n =5, ts= .702, α=.27, (1-α) = .73 n =10, ts= 1.053, α=.16 (1-α) = .84 n = 17, ts = 1.404, α=.09 (1-α) = .91 n = 26, ts = 1.755, α=.04 (1-α) = .96 n = 37, ts = 2.106, α=.015 (1-α) = .985 5 EMIS 7307 T&E Part 5 Test Planning and Analysis •From the previous chart we get a sense of the “cost” of confidence. •For an increase of only three tests (from 2 to 5) we can boost our confidence by 12%. •But the change from 17 to 26 only further increased our confidence by 5%! •This illustrates why 90% confidence is often chosen as good enough. It’s a compromise of increased confidence versus cost. 6 EMIS 7307 T&E Part 5 Test Planning and Analysis ANOVA R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 7 EMIS 7307 T&E Part 5 Test Planning and Analysis R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 8 EMIS 7307 T&E Part 5 Test Planning and Analysis ANOVA R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 9 EMIS 7307 T&E Part 5 Test Planning and Analysis R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 10 EMIS 7307 T&E Part 5 Test Planning and Analysis R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 11 EMIS 7307 T&E Part 5 Test Planning and Analysis R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 12 EMIS 7307 T&E Part 5 Test Planning and Analysis R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 13 EMIS 7307 T&E Part 5 Test Planning and Analysis R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. 14 EMIS 7307 T&E Part 5 Test Planning and Analysis R. Walpole and R. Myers, [1993], Probability and Statistics for Engineers and Scientists, Prentice-Hall, Englewood Cliffs, New Jersey. •In this example the table f value is 2.78. Use 4 and 25 as the degrees of freedom. •The calculated f is 4.3. •Therefore, reject the null hypothesis. 15 EMIS 7307 T&E Part 5 Test Planning and Analysis 16 EMIS 7307 T&E Part 5 Test Planning and Analysis • ANOVA is a powerful data analysis tool – Using a few samples answers the question, does an effect on a subgroup of a hypothetical population cause that subgroup to not really be from the hypothetical population? • We hypothesized that the five different aggregates were from a population that absorbed moisture similarly. • The ANOVA showed that they were not! 17 EMIS 7307 T&E Part 5 Test Planning and Analysis • ANOVA is readily available in many spreadsheet packages. – Excel has it under tools then click on data analysis and select ANOVA. 18 EMIS 7307 T&E Part 5 Test Planning and Analysis • Bottom line. – Use student t or normal distribution (n>30) when comparing to a spec. – Use ANOVA when looking to see if two groups are statistically the same to some confidence level. 19 EMIS 7307 T&E Part 5 Test Planning and Analysis • Bottom line (cont’d) – For both test planning and data analysis at least one member of the team needs to be statistics savvy. – From a management perspective. • This facilitates planning for adequate data collection to meet the desired confidence. • Assures data collection ends when sufficient data exist. 20 EMIS 7307 T&E Part 5 Special Topic Software testing • In an earlier lecture I presented the results of a study showing the economic impact of poor or limited software testing. • I’ve also made the case that earlier problem finding and fixing is exponentially cheaper. • Here I want to introduce several software testing concepts. 21 EMIS 7307 T&E Part 5 Percent of Specification Requirements Involving Software 100 F-22 Percent 80 B-2 60 F-15 F-16 Series1 40 F-111 20 F-4 A-7 0 1960 1964 1970 1975 1982 1990 2000 Ref: Lockheed - Martin Corp. 22 EMIS 7307 T&E Part 5 Special Topic Software testing •Human testing is defined as informal, noncomputer-based methods of evaluating architectures, designs and interfaces. It can consist of: • Inspections – The programmer explains his/her work to a small group of peers with discussion and direct feedback on errors, inconsistencies and omissions. • Walk-through – A group of peers develop test cases to evaluate work to date and give direct feedback to the programmer. • Desk Checking – A self evaluation is made by the programmer of his/her own work. There is a low probability of identifying his/her errors of logic or coding.23 EMIS 7307 T&E Part 5 Special Topic Software testing • Peer Ratings – Mutually supportive, anonymous reviews are performed by groups of peers with collaborative evaluations and feedback. • Design Reviews – Preliminary design reviews (PDRs) and critical design reviews (CDRs) provide milestones in the development efforts that review development and evaluations to date. 24 EMIS 7307 T&E Part 5 Special Topic Software testing • Computerized software testing – Black box • Functional testing of a software unit without knowledge of how the internal structure or logic will process the input to obtain the specified output. Within-boundary and out-of-boundary stimulants test the software’s ability to handle abnormal events. Most likely cases are tested to provide a reasonable assurance that the software will demonstrate specified performance. Even the simplest software designs rapidly exceed capacity to test all alternatives. 25 EMIS 7307 T&E Part 5 Special Topic Software testing • Computerized software testing – White box • Structural testing of the internal logic and software structure provides an opportunity for more extensive identification and testing of critical paths. The process and objectives are otherwise very similar to black box testing. 26 EMIS 7307 T&E Part 5 Special Topic Software testing • IV&V – Independent verification and validation are riskreducing techniques that are applied to major software development efforts. The primary purpose of IV&V is to ensure that software meets requirements and is reliable and maintainable. The IV&V is effective only if implemented early in the software development schedule. Requirements analysis and risk assessment are the most critical activities performed by IV&V organizations. 27 EMIS 7307 T&E Part 5 Special Topic Software testing • IV&V – The desire to boast of a high SEI rating in proposals and to generally improve the corporations efficiency has moved many companies to significantly improved software development capabilities. – This in turn has reduced the perceived need by customers to require and pay for IV&V. 28