Thesis Proposal – for Master of Software Engineering



DESIGN OF A PARAMETRIC OUTLIER DETECTION SYSTEMBy Ronald J. EricksonB.S., Colorado Technical University, 2001 A project thesis submitted to the Graduate Faculty of the University of Colorado at Colorado SpringsIn partial fulfillment for the degreeMasters of Engineering Department of Computer Science2011?Copyright By Ronald J. Erickson 2011All Rights ReservedThis Project Thesis for Masters of Engineering Degree by Ronald J. Ericksonhas been approved for the Department of Computer SciencebyDr. C.H. Edward ChowDr. Xiaobo ZhouDr. Chuan Yue ______________________________DateThis Project Thesis for Masters of Engineering Degree by Ronald J. Ericksonhas been approved for the Department of Computer SciencebyDr. C.H. Edward ChowDr. Xiaobo ZhouDr. Chuan Yue ______________________________DateErickson, Ronald J. (M.E., Software Engineering)Design of a Parametric Outlier Detection SystemProject Thesis Directed by Professor C.H. Edward ChowIntegrated circuit defects are only discovered during the electrical test of the integrated circuit. Electrical testing can be performed at the wafer level, package level or both. This type of testing produces a data-log that contains device specific electrical characteristic data known as parametric data. The testing of the integrated circuit has become increasing expensive in an ultra competitive market. The traditional method of setting upper and lower limits to discover non-compliant circuit defects has become only a portion of the criteria required to discover potential defects as the geometries of these integrated circuits continue to decrease.Integrated circuit manufacturers are now increasingly employing additional methods of failure detection by adding test structures on the wafer. Other techniques employ high percentage failure mode screening on these integrated circuits to discover device marginality. However, marginality testing can only be accomplished after extensive characterization of the circuit on numerous wafer lots over standard process corners to establish device trends. Another technique includes a design for manufacturing structure that detects a single parametric measurement that is potentially out of family when compared to the device simulation. All of these standard types of failure detection work well with an environment that builds many wafer lots and thousands of a single mercial circuit manufacturers will typically lower the FIT “failures in time” rate requirements of their specific integrated circuit. This is especially true when the integrated circuits of the potential amount of returned merchandise are lower than the costs of more rigorous electrical testing. Currently there is no universally industry adopted way to detect out of family circuit parametrics within a rigorous FIT requirement. In addition, these circuits have an end use that is used in a mission critical environment and where the product quantities are small. This thesis paper and accompanying project is an attempt to address the opportunities and challenges of creating a more rigorous parametric outlier detection system to detect out of family measurements on embedded circuits with very high FIT requirements, very small sample sizes, used in mission critical devices.For my family Crystal, Chelsea, and Isabel thank you for your supportTo my parents Gerry and Janice for always compelling me to continue my educationAcknowledgementsAnyone who has ever gone back to a university to obtain a high level degree knows the effort that is involved in the creation of a project of this magnitude. This project thesis was no exception. While I have many people to thank for helping me throughout this project, I would like to pay special mention to a number of individuals.First, I would like to sincerely thank Professor C.H. Edward Chow. Dr. Chow you are an excellent instructor and I really enjoyed the time you have taken mentoring me throughout the tedious graduate school process. Your time is very valuable, and I thank you for sharing a portion of that time over the last couple semesters. This project thesis would not have been successful without your direct guidance.To the members of my project thesis committee, Dr. Xiaobo Zhou and Dr. Chuan Yue, thank you for your feedback, support, and of course your encouragement throughout this process.And finally to Mr. Troy Noble: Thank you for being a great motivator and intuitive in asking and answering my questions. In addition, thank you for the personal mentoring in electrical engineering and software development.TABLE OF CONTENTSChapterINTRODUCTION....………………..……………………………………1Background…………...…………………………………………2General Circuit Manufacturing Techniques …………………7Design for Manufacturing Architectures..……………………8Motivation……………...…………….…………………………10Approach Overview ………………………….……………….12SOFTWARE ENGINEERING.……………………………………….13Software Engineering Basic Stages………………….……..14Capability Maturity Model Integration………………………..16Software Engineering Advanced Stages…………………...20STATISTICAL ANALYSIS……………………………………………23Goodness of Fit...……………………………………….……..25The Null Hypothesis…...….…………………………………..27Anderson Darling……………………………………….……..28SYSTEM DESIGN & ARCHITECTURE......………………………..30Database………………………………………………………..31Automatic Data Extraction…………………………………….32Automatic Statistical Modeling ...…………………………….33Testing and Evaluation of Prototype......…….……………...36Analysis of Results………………........…….………………...38CONCLUSIONS.............................................…….………………..44Lessons Learned......…….…………………………………...46BIBLOGRAPHY..........................................…………….…...……………..50APPENDICIESData Set Parametric Outlier Application Dataset 1………..…..53Normal Distribution Test Case Data Set Parametric Outlier Application Dataset 2………..…..54Normal Distribution Test Case Data Set Parametric Outlier Application Dataset 3………..…..55Normal Distribution Test CaseData Set Parametric Outlier Application Dataset 4………..…..56Bimodal Distribution Test CaseData Set Parametric Outlier Application Dataset 5………..…..57Bimodal Distribution Test CaseData Set Parametric Outlier Application Dataset 6………..…..58Normal Distribution with Negatives Test CaseData Set Parametric Outlier Application Dataset 7………..…..59Extreme Outlier with Multiple Ties Test CaseData Set Parametric Outlier Application Dataset 8………..…..60Multiple Ties Normal Distribution Test Case Data Set Performance Graded Binder Sample 195……………61Large Data-Set Non-Normal Distribution Test CaseData Set Performance Graded Binder Sample 196……………62Large Data-Set Non-Normal Distribution Test CaseLIST OF TABLESTable Critical Level Table……………..….……..………………………………..…28LIST OF FIGURESFigureWafer Dice with Street Intact …….……..……………………..……………...3Wafer Dice with Street Sawn…….……..……………………………………...4Astronaut Repairing the Hubble Telescope…………………………………10Software Engineering Basic Stages………………. …….…….…………...13Software Engineering Advanced Stages………... …..….…….…………...21Normal Distribution Curve………………………………….…….…………...34Bimodal Distribution Curve………..……………………….…….…………...36Cluster Algorithm Flowchart……....……………………….…….…………...41CHAPTER 1INTRODUCTIONIntegrated circuits are designed and simulated by designers and then sent off to a wafer fabrication to be built on wafers. Unfortunately wafer fabrication manufacturing and design defects are only discovered during the electrical test of the integrated circuit. This testing of the circuit electrically can be performed at the wafer level, package level or both. Electrical testing produces a data-log that contains specific device characteristic data known as parametric data. Testing of the integrated circuit has become increasing expensive in an ultra competitive market and circuit manufactures are exploring new methods of discovering failures. Traditionally circuit manufactures would use a method that would set an upper and lower limit on a parametric measurement to detect circuit failures. This methodology has become only a portion of the criteria that is now required to discover potential defects of small geometry integrated circuits. Integrated circuit manufacturers are now employing additional methods of failure detection including wafer test structures and specialized BIST (built in self test) circuitry. After extensive characterization of the circuit on many devices, a high percentage failure mode trending on these devices can be used to detect lower parametric marginality. Design for manufacturing simulations is limited in that they only employ detecting a limited parametric measurement to the original design simulation. All of these failure detection schemes work well with environments that produce thousands and thousands of the single integrated products. Most commercial non-mission critical integrated circuit companies will typically lower the FIT rate requirements of their specific integrated circuits. This is because the end users of the product are within a commercial market. Another reason commercial manufactures can lower the FIT requirements is because the potential returned merchandise costs are usually lower than the costs of more rigorous testing of the integrated circuit. There currently is a need to create a more rigorous, trusted automated parametric outlier detection system on embedded circuits with special requirements. These special requirements include high FIT requirements, small sample sizes, and high reliability on life and death critical systems. These systems can be airport scanners, avionics circuitry, military and satellite circuitry.BackgroundAs global markets become increasingly competitive some companies are utilizing a fab-less wafer manufacturing system. These companies design, assemble, and test the integrated circuits applications like aerospace, high-altitude avionics, medical, networking and other standard consumer electronics in their facility. Then pay to have the wafers manufactured at an outside facility. In this type of business model the company does not own a wafer fabrication facility. To stay price competitive companies must use many wafer manufacturing facilities throughout the world, depending on the design geometries of the transistor structures being manufactured within the integrated circuit design. Wafer Dice with Street IntactTo mitigate a portion of the risk using differing wafer fabs, most companies have begun inserting standardized transistor cell structures in the streets of the wafer. (Marinissen, E. Vermeulen, B., Madge, R. Kessler, M., and Muller, M, 2003) The wafers streets are the areas between the die and is illustrated above. However to save silicon the streets are the area of where the saw cuts the wafer producing individual die. These test structures are only available for testing while the wafer is intact at wafer probe. The saw processes destroys these test structures and are no longer available for testing after sawing, see illustration below. Wafer Dice with Street SawnHigh reliability products are electrically tested multiple times prior to shipment to ensure compliance to the device specification. In some cases this testing is tailored to the specific industry for which they are designed (Marinissen, E., Vermeulen, B., Madge, R. Kessler, M., and Muller, M, 2003). Companies must consistently monitor large amounts of test data at pre-defined electrical test steps during the manufacturing process to ensure compliance (Perez, R., Lilkendey, J., and Koh, S.1994). Contained within this test data are device specific electrical parametric readings that are composed of DC and AC electrical parameters pertaining to the individual device performance. To maintain a high yielding product as the geometries continue to decrease, the need to monitor device parametric data has become an increasingly important tool in discovering discrete product defects (Moran, D., Dooling, D., Wilkins, T., Williams, R., and Ditlow, G 1999). Naturally the business model of the fab-less microelectronic company introduces even more risk to the development of quality wafers than a company that owns their own fab and can tailor the process to their needs. The wafer fabs write contracts on less restrictive wafer parametric acceptance. A less restrictive wafer acceptance criterion is geared to increase fab starts and deliverables on a is more positive for the fab. This philosophy leads to the potential of large deviations in material parametrics that must be screened for out of family measurements. Some consumer products companies have resorted to only electrical testing at the wafer level. (Bhattachatya, S., and Chatterjee, A 2005). Generally because it cheaper to replace the device than it is to electrically test the device at package level. Most commercial manufacturers have been willing to trade reliability for economics. Designers of high reliability systems cannot accept this trade-off for the following reasons. Mission critical systems require high reliability because replacement of faulty parts is difficult or impossible after deployment, or component failures may compromise national security. Furthermore, typical service life tends to be longer for military systems and aerospace applications, so high reliability and mission critical devices must be tested at higher level at package to ensure compliance.Wafer fab minor deviations within the doping of the transistor can create very different wafer structure parametric data (Singh, A., Mani, M., and Orshansky, M 2005) (Veelenturf, K 2000). These types of deviations can be easily normalized and thus predicted if there are many parametric wafer samples. Deviations are much harder to predict on small lot quantities that can have months or years between lot manufacturing. Radiation tolerant circuits typically are only built in small quantities and sold after a lengthy well documented qualification process. Some companies have developed a manufacturing process to produce a range of specific radiation-hard electronic products using rad-hard by design techniques or by adding rad-hard doped trenches. These processes are somewhat different from the ones used in commercial foundries because they include a modified process that produces a circuit with greater radiation tolerance. The processes are generally developed between the manufacturing fab and the fab-less circuit design facility. Commercial foundries typically use standard material for all material manufactured in the facility. By contrast radiation tolerant circuits require nonstandard materials, incorporating epitaxial layers, insulating substrates; doped trenches in the silicon that enhance radiation tolerance. The use of non-standard materials makes out of family detection much more difficult due to the inherent parametric leakage these techniques introduce. This is where better predictability to detect outliers without many samples on non-standard processes is driving the requirement to build a statistically based robust detection tool.General Circuit Manufacturing TechniquesAll integrated circuit defects are discovered through the electrical test of the integrated circuit at the wafer level. The general flow of high reliability devices is to re-test at the packaged device level to detect assembly, environmental and accelerated life-time infant mortality failures. A portion of these defects that affect integrated circuits throughout the process of manufacture at a wafer fabrication facility, through the assembly, environmental aspects of tests are listed belowWafer fabrication defects transistor doping, oxide breakdown, interconnect and poly stringersopens, shorts, resistive metals, bridging materials metallization stability issues, dissimilar metals like AL & CU.Potential device assembly defects wafer back-grind, wafer saw, die preparationdevice package defects including trace length, & integrity, package leadssolder ball integrity, die attach, wire bond, hermetic lid sealEnvironmental defects on the package and die are caused by simulatingtemperature cycle, temperature shock, mechanical vibrationcentrifuge, high humidity, salt environmentaccelerated and dynamic electrical burn-in causing infant mortality total ionizing dose (TID), gamma radiation effects on gate oxides As a minimum the digital circuit DC parametric tests performed at electrical test are Continuity, power shorts, input leakage, output tri-state leakagequiescent power, active power, input and output thresholdsminimum voltage operationHowever, these digital circuit measurements usually include the AC electrical parametersdata propagation, input set-up, output/input holdinput rise, input fall, clock duty cycleDesign for Manufacturing ArchitecturesAll high reliability IC manufacturers must electrically test their product at the wafer level to ensure compliance to the product specification and save package and assembly costs. Design for manufacturing techniques (DfM) imply that device parametric data can potentially discover the mismatches between the manufacturing process of the circuit and the circuit designer simulation within test structures in street locations of a die reticle (Gattiker, A 2008) These parametric defect techniques only detect limited parameters of quiescent current device sensitivity and are limited to a very specific device function (Wang, W., and Orshansky, M 2006). Test structures capable of testing interconnect circuit dependencies, or the race conditions that can occur between circuit blocksMARYL (Manufacturability Assessment and Rapid Yield Learning) minimizes the risk of starting high volume manufacturing in a waferfab. Process engineers consistently study the relationships between the process parameters within the company owned wafer fabrication facility. Then modify the process and evaluate the electrical parameters of the integrated circuits and predict a final yield of the ongoing process (Veelenturf, K 2000). Some other DfM tools utilize the circuit from a many building blocks approach. This type of DfM tool takes the knowledge of smaller building block circuits in an attempt to solve a part of a larger problem through simulations. This type of simulation on a real world integrated circuit is based upon the performance or execution time of a particular product. To include a specific product line in a specific wafer fab and is only utilized as a simulation and not real silicon. (Moran, D., Dooling, D., Wilkins, T., Williams, R., and Ditlow, G 1999). Standard DfM techniques work well within the commercial market because these designs do not contain rad-hard non-standard materials or manufacturing requirements. Essentially a mature process wafer test structure will deterministically behave as predicted without the rad-hard circuit requirements. However within a small fab-less business model the wafer manufacturer will assume a no-liability stance on non-standard material due to the complexity and variance from one wafer lot to another. Add in the fact that manufacturing these wafers is generally more risky to yield significant die. MotivationA small fab-less rad-hard circuit company needs a tool that can process relatively small amounts of individual device data based on small lots of the electrical characteristic data. This small amount of data can be skewed by the implied marginality effects of a Commercial Radiation Hardened (CRH) process within the transistor structure on a wafer to wafer variability. These specialized circuits require nonstandard starting materials, incorporating epitaxial layers, insulating substrates; doped trenches in the silicon that enhance radiation tolerance, but introduce marginality into the circuits with increased sub threshold leakage effects. To compound the issue, this data being analyzed within these small lots must be used to guarantee a very high FIT rate. On highly reliable products that provide service within a mission critical sub system. Astronaut Repairing the Hubble TelescopeTo complicate matters if these devices are placed in service on a system that is in a high altitude or a space application? It is becomes unbelievably expensive to send an astronaut to replace a small faulty integrated circuit or subsystem in space. For instance the 2009 repair of the Hubble space telescope was valued at 1.1 billion dollars. It is highly visible when a space system fails, the publicity involved in a failure of this magnitude, and the potential government fines can easily drive a small company into bankruptcy and out of business. Typically rad-hard device lot quantities are small and the product offering are very broad to cover simple logic devices, to memories of DRAM, Flash, and SRAM. Additional devices types could be protocols of spacewire, European CAN, multiple clock generator chips, or single and multiple processors. All of these products could be built of differing technologies from 0.600?m to 90n meters by differing wafer fabrication facilities throughout the world. Radiation tolerant methodologies increase the sub-threshold leakage of the integrated circuit by design. These additional leakage paths provide pathway to slough off excessive charge that could potentially damage the gate oxide of the circuit transistors. These structures are required when experiencing an event of total ionizing dose, single effect upset, and single effect latch-up on the integrated circuit. The need to analyze the validity of the parametric data being captured in real time during the testing and manufacture of these circuits is of great importance.Approach OverviewThis automatic parametric outlier detection system prototype will utilize an existing algorithm named Anderson-Darling to test for normal distribution on the dataset. This test is a general normality test designed to detect all departures from normality within a dataset, and is believed to be the most powerful test for normality. The importance of checking the data-set for normality is very important or the consequential methodology for choosing outliers may produce incorrect results. If a dataset is deemed a normal Gaussian distribution by the automated Anderson Darling test The prototype will perform an inner 75% quartile outlier detection technique on the dataset and present the user with a histogram distribution of the data, and the Anderson-Darling calculated value. The lower 12.5% calculated value, and the upper 87.5% value are also displayed for comparison to the data-set for outlier identification. If a dataset is deemed non-normal by Anderson Darling testing, the dataset will be presented in a histogram format requiring user intervention. The user will be required to visually inspect the data and set the upper and lower fences for a normal distribution. Once the user has placed the upper and lower fences on the data distribution the remaining steps for normal and non-normal user intervention datasets will be the same. The prototype will perform an inner 75% quartile outlier detection technique. Create the histogram, Anderson-Darling non-normal calculated value, the lower 12.5% value, and the upper 87.5% value are then displayed for comparison to the data-set for outlier identification.CHAPTER 2SOFTWARE ENGINEERINGSoftware engineering is dedicated to the design, implementation, and maintenance of software code, software applications, and software systems. Software engineering integrates significant mathematics, computer science and general best engineering practices. High quality, more affordable, easy to maintain software is generated with a methodic approach to the key stages of the software. This chapter defines the basics of creating software engineering procedures and methodologies that produce reliable operating software.Software Engineering Basic StagesSoftware Engineering Basic Stages The basic stages within a software engineering life-cycle are the software requirements, design, construction, testing, and maintenance.A software requirement is defined as a property that must be exhibited in the software product being created that will solve a real-world problem. Requirements documents identify the project problem that the software will solve. This is completed through the elicitation of stakeholders, documenting these requirements, and analyzing the requirements by the people and organizations that are affected by the software. When an agreement on each of the requirements, and priorities for these requirements are set on the software to be created the project can move to the next stage; design.Software design is the process of defining the software architecture, the user and system interfaces. It also defines other characteristics of a system and any component that affects the software product to the requirements that were documented in the requirements stage. The use of tools similar to UML unified modeling language produces class diagrams or other upper level coding representation diagrams. These diagrams describe the static design of the software objects and the relationships these objects have with to each other in the software application. These diagrams are generally the basis used to describe a software design, and complete the design stage.Software construction refers to the methodical detailed creation of operating, defect free, user friendly software through the combination of writing and debugging the code. The construction follows the design foundation built upon the class diagrams or representation diagrams defined in the design stage. The construction of the unit tests that ensure individual units of code are fit for use in the system. In addition to the construction of the integration tests that will verify all the source code with the final system are built in the construction stage.Software testing consists of the dynamic validation of the requirements documentation while operating in the system environment. Generally the quality of the software under test performs the process of executing the program or application with the intent of finding software defects or errors. The software verification ensures the software operates correctly. Unit testing ensures individual units of code are fit for use in the system. Integration testing will verify the source code operates within the final system environment.Software maintenance is the most expensive stage in the software life cycle as it covers the operational lifetime through the eventual retirement of the software. When the software is delivered or placed in operation and errors are discovered a software engineer must fix the issues for continued use. But errors are not the only activity associated with software maintenance. In addition operating environments can change, operating platforms can change, or new requirements and functionality may be necessary? (Abrain, A. 2004)Capability Maturity Model Integration (CMMI)Capability Maturity Model Integration is a measure of the software process maturity framework within an organization. The CMMI assess the process maturity of software engineering company by setting specific benchmarks within the software creation process. The CMMI contains five descriptive benchmark levels to measure a company’s ability to produce quality software, these levels are described below:Maturity Level 1 Initial Maturity Level 2 Managed Maturity Level 3 Defined Maturity Level 4 Quantitatively Managed Maturity Level 5 OptimizingMaturity Level 1: The initial level is characterized by processes and projects that are usually unplanned from project management perspective. These projects and processes are very informal, subject to last minute changes, and based on improvised poorly documented requirements that eventually lead to project scope issues. Companies and the projects that software engineer to this maturity level are usually small start-up companies that tend to take short cuts when schedules are due. The companies are also prone to over commit, over promise, under-deliver, and don’t have repeatability on past successes with new projects.Maturity levels above level 1 have specific goals and requirements for the software engineering process. These goals and requirements must be met for a company to be certified at a higher level. As these levels progress higher from a level 1 to level 5 implies the higher level contains the requirements of the lower levels and the requirements of the higher level Maturity Level 2: The managed level has requirements that are documented and managed within the project. All processes have controlled documented planning stages and are measured at major milestones. The status of these processes and the software products are also reviewed at major milestones and the completion of major tasks. Requirements management identifies the problem the project will solve through the elicitation of stakeholders, documenting requirements, and analyzing requirements. These requirements are then agreed upon and priorities set through content management. Project planning identifies the scope of the project; estimates the work involved, and creates a project schedule based on the scope and work estimates. Project monitoring and control is handled through the use of a program manager that will hold regular meetings with project staff. This program manager will capture meeting notes and update the schedule to current work completed. Then communicate these project results to management and customers to manage expectations. Supplier agreement management handles all project subsystems, hardware, software, manuals, documentation, parts and materials that will be used on the project.Measurement and analysis involves gathering quantitative data about the software project and analyzing that data to influence project plans and effort expended on the project. Process and product quality assurance involve formal audits by a separate quality assurance individuals or groups. Technical peer reviews are required at different levels within the project, and in-depth reviews of all work performed. Configuration management outlines how the project team will control and monitor changes to the project code as the project progresses are required for a company to maintain maturity level 2.Maturity Level 3: Now defined processes are understood and characterized these processes are described in documented standards, procedures, and tools. Processes performed across the organization are consistent except for the specific variances to the standard guidelines that contained within a process tailoring document. Additional requirements for level three from the previous level include additional requirements analysis. Additional technical solutions that will design, develop, and implement solutions to requirements document. Project verification that ensures the software operates correctly and validation that ensures that the correct product was built per the requirements documentation. The addition of an organizational process improvement with a focus on initiating process improvements on processes and process anizational process definitions that establish and maintain a usable set of process assets and work environment organizational standards, and then completes training to these standardsIntegrated project management that tracks the activities related to maintaining the project plan, monitoring plan progress, managing commitments, and taking corrective actions. Risk management that will assign risk items to the project before and during the project as they are recognized. The project team will have to work to mitigate or remove risk altogether as the project progresses. Integrated teaming involves the teaming of software engineering to system engineering. The integrated supplier management within the project is related to management reporting and management. Decision analysis and resolution are comprised of decisions that may move the project timeline, could have a major impact on system performance, or that could have legal implications. Organizational environment for integration will integrate product and process development. Product integration will assemble the product and ensure that the product functions properly when integrated in the system for maturity level three.Maturity Level 4: The addition of quantitatively managed objectives of quality and process performance. These objectives are then used as the criteria in managing processes. Quality and process performance is controlled using statistical and other quantitative techniques, and is now quantitatively predictable based on these techniques. The organizational process performance views how well the processes are performing, these performance matrices can be parametrics of effort, cycle time, and defect rate. Quantitative project management is the project’s defined process and the ability to statistically achieve the project’s established quality and process-performance objectives.Maturity Level 5: Finally optimizes these processes by continually improving on quantitative understanding of processes variation causes. Through both incremental and innovative technological improvements level 5 constantly tries to improve process performance. Organizational innovation and deployment innovative and incremental improvements that improve the software processes and technologies of the company. Causal analysis and resolution that trace defects to the root cause and then modifies the processes at fault to reduce or eliminate those defects. (CMMI, 2010)Software Engineering Advanced Stages Software configuration management identifies the configuration of software at distinct points during the software engineering lifecycle. With only one purpose to control and track changes to the software configuration and completed through the systematic and methodical use of version control software tools. The use of configuration management will maintain the integrity and traceability of the software configuration for the life cycle including benchmarks.Software engineering management is the project management and measurement of software at various stages with the software lifecycle by monitoring plan progress, managing commitments, and taking corrective actions.Software Engineering Advanced StagesSoftware engineering process defines the implementation, assessment, measurement, management, modification, and eventual improvement of all the processes used in software engineering.Software engineering tools and methods are the use of tools that provide the assistance to the software developer in the development of software project. The purpose of the software engineering tools and methods are to decrease the software development time, increase the quality, reduce errors and software defects during the construction and maintenance stages of a software lifecycle. Software quality identifies the software quality considerations which occur during the software life cycle. Through formal audits by quality assurance group of the process and product, these audits provide a quality measure of how well the software performs and conforms to the original design and requirements. It also manages and controls the content management systems, then defines the rules for the use of these systems.Knowledge areas of related disciplines include computer engineering that is a merge of electrical engineering and computer science. Project management plans, organizes, secures, and manages all project resources in an effort to successfully complete a software project. Computer science is the study and implementation of techniques to perform automated actions on computer and server systems. Quality management seeks to improve product quality and performance that will meet or exceed requirements documentation for the software product. Management software ergonomics deals with the safety and compliance of the software systems to rules and regulations within the company, state, and environment. Mathematics is the learning or studying of patterns of numbers, counting, measurement, and calculations of quantity, space, and change. Systems engineering is effort required to integrate the software to the system platform that it will run. (Abrain, A. 2004) A combination of all of these disciplines is required on every software engineering project. It is these skill set requirements that are important to the success of any software engineering project but usually require the resources of many experts. CHAPTER 3STATISTICAL ANALYSISAn outlier is defined as an observation point that is inconsistent with other observations in the data set. A data point within the data-set, an outlier has a lower probability that it originated from the same statistical distribution as the other observations in the data set. An extreme outlier is an observation that will have an even low probability of occurrence than the rest of the data. Outliers can provide useful information about the process or the designed product. An outlier can be created by a shift in the mean of the data-set or the variability of parameters in the process. Outliers provide a numerical gauge of measure into the quality, yield, measurement errors, and process deviations of a product. Though an observation in a particular sample might be a candidate as an outlier, the process might have shifted and caused this outlier observation. Sometimes the erroneous result is a gross recording error or an un-calibrated measurement error. Measurement systems should be shown to be capable for the process they measure to correct significant digits and calibrated for accuracy. Outliers also occur with incorrect specifications that were based on the wrong distributional assumptions at the time the specifications are generated. Incorrect specifications can also occur if the sample size used was not large enough, or from diverse samples of the process at differing times of manufacture.Once an observation is identified by means of mathematical or graphical inspection for potential outliers, root cause analysis should begin to determine whether an assignable cause can be found for the outlier result. If the root cause cannot be determined, and a retest cannot be justified due to equipment issues, the potential outlier should be recorded for future reference and quality control standards. These recorded outliers are then used for process improvement at future time within the process flow. Outlier processing of a data-set can potentially introduce risk in the evaluation if the assumption that the data-set comes from a normal distribution; when actually the data-set is from a non-normal distribution. To ensure that errors are not introduced into the outlier analysis, the importance of testing the distribution for normality in the distribution before performing outlier analysis is critical. Because the characteristic property of the normal distribution is that 68% of all of its observations fall within a range of ±1 standard deviation from the mean, and a range of ±2 standard deviations includes 95% of the data points. If the outlier calculations are not calculated on normal distributions the mathematical properties and assumptions made begin to deteriorate and add error to the results. Outlier calculations using the standard deviation, quartile, or CPK analysis only show the variation of the data-set from the mean by usually ignoring the tails of the data-set and can misleading. Small data-sets present the unique problem that they essentially can be easily viewed by the user quickly as appearing to not be from a normal distribution. Only after additional analysis using tools on the distribution can these data-sets be proven to be from or not from a normal distribution. This project utilizes the Anderson-Darling test for normality; and was developed by two mathematicians with statistical backgrounds, Theodore Anderson and Donald Darling in 1952. When used to test if a normal distribution adequately describes a set of data, it is one of the more powerful statistical tools for detecting most non normality distributions. The equation can also be used as an estimation of the parameter for the basis of a form of minimum distance calculation, but was not used for this purpose in this project. Statistically it is an evidence test performed on the data set to give confidence it from non normal distribution. The given probability distribution then identifies the probability for each value of a random discrete variable within a given data set. Then provides a single p-value for quick evaluation of the normalcy contained with a data-set Goodness of Fit The most commonly used quantitative goodness-of-fit techniques for normality within a distribution are the Shapiro-Wilks, and the Anderson-Darling tests. The Anderson-Darling is commonly the most popular and widely used because it the accuracy of the test. Because of this fact the original requirements defined that Anderson-Darling would be used exclusively on this project. The Anderson Darling test is merely a modification of the Kolmogorov-Smirnov test but gives more weight to the tails. The Kolmogorov-Smirnov test is distribution free because the critical values do not depend on the specific distribution being tested; this is the difference with the Anderson Darling test which makes use of the specific distribution of the data set in calculating the critical values. Because of these facts the Anderson Darling creates a more sensitive normality test but the critical values must be calculated for each distribution. (Romeu J 2003)Both Shapiro-Wilks and the Anderson-Darling tests use a confidence level to evaluate the data set for normality, this value is referred to as a p-value or it is also known as an alpha level. A lower p-value limit correlates to a higher value of confidence and proves greater evidence that the data did not come from the selected distribution and therefore is not a normalized distribution. This double negative of not rejecting the null hypothesis is somewhat counter intuitive. Simply put the Anderson Darling test only allows the data reviewer to have a greater level of confidence as the p-value increases that no significant departure from normality was found. A unique explanation or an interpretation of the null hypothesis is below.Just as a dry sidewalk is evidence that it didn't rain, a wet sidewalk might be caused by rain or by the sprinkler system.? So a wet sidewalk can't prove that it rained, while a not-wet one is evidence that it did not rain.11Annis,C.StatisticalEngineering Charles.Annis@, 2009The Null HypothesisThe definition of the null hypotheses is: H0: that the data-set follows a specified distribution. This hypothesis regarding the distributional form of the data-set is rejected at the chosen (α) significance level and if the Anderson-Darling A2 test statistic is greater than the critical value obtained from the critical value table below. The fixed values of α (0.01, 0.05, 00.25, 0.10.) are generally only used to evaluate the null hypothesis (H0) at various significance levels. The (α) significance value or sometimes called the p-value of 0.05 is typically used for most applications, however, in some critical instances; a lower (α) value may be applied. This thesis project used the (α) significance value limit of 0.05 as the limit of p-value comparison to the application output when checking a data-set for normal distribution. It is this value that is outputted by the application that is used for the basis of user involvement on distributions that do not have a p-value greater than 0.05.The critical values of the Anderson-Darling test statistic depend on the specific distribution. The tables of critical values for many of the distributions are not normally published, the exception are the most widely used ones like normal, log-normal, and Weibull distributions. The Anderson-Darling quantitative goodness-of-fit techniques for normality tests are typically used to test if the data is from a normal, log-normal, Weibull, exponential, or from logistic distributions. The assumption is made that data-sets tested against this application for normalcy fall into the category of published critical value distributions.Due to the vast differing data distributions that can be analyzed by an Anderson-Darling test, it was chosen as the best generic test for normality for this thesis project application. The published confidence levels for normal, log-normal, Weibull distributions are published below in the critical values table. The confidence levels are calculated from the p-value in the form of 100*(1 - p-value).?Confidence Levelα-levelCritical Value A299.0%0.010.63195.0%0.050.75292.5%0.0250.87390.0%0.101.035Critical Level TableAnderson–Darling Because the Anderson-Darling statistic belongs to the class of quadratic empirical distribution that are based on the empirical distribution function statistical model. Essentially a step function that jumps for 1/n at each of the n data points, The Anderson Darling test statistic A, and the A2 equations are shown below these for the normal, lognormal, and Weibull distributions.n is the sample size of a continuous probability distribution. The equation is well suited for smaller sample data set size calculations. The result is Anderson Darling test statistic A; The A2 value calculated is then compared to an appropriate critical value from the table showing confidence level, α-value, and critical value of the null hypothesis from the critical value table shown above. In summary for simplicity of this thesis project the Anderson-Darling test for a normal distribution proves the data is from a normal distribution if the p –value > α-value and if A2 -value < Critical-Value from the critical value table shown above. In the real world the need to make decisions and then get on to other business matters is incredibly important. The normal distribution function using Anderson-Darling will produce quick estimates that are useful for such decisions. This is true even if the data-set is small and appears to be non-normal. It can still be assumed a normal distribution if the histogram curve looks non-normal, providing the p-value of the goodness-of-fit statistic shows the distribution is a normal Gaussian bell curve p > 0.05.CHAPTER 3SYSTEM DESIGN & ARCHITECTUREThe requirements documents of this project was to take a distribution of data, analyze the data for a normal Gaussian bell curve distribution, perform outlier analysis by some means and then indentify potential outliers within that data set. The data will then be presented to user as a histogram and outliers are shown. If the data-set did not come from a normal distribution then present the data-set in a histogram format so the user can set the lower and upper fences of the data-set before applying 75% percentile analysis.This project investigated the design of a prototype design that will utilize multiple existing algorithms like an Anderson-Darling, statistical based outlier analysis in a user friendly automated system. The goal of this design was to combine various existing outlier detection techniques and make necessary improvements so that the small company outlier detection system used on small data-sets would operate similar to other elaborate expensive architectures in making better business decisions. Specific software requirements included the platform base code that would be C# used to as the prototype test vehicle. This requirement was changed in the project due to unforeseen small company infrastructure issues after an agreement with stakeholders was completed. This requirement change added a significant amount of coding that had to be converted to the C# from the original base code. Database This project did will not focus on the database design, however the automated system inputs and future capability learning criteria was an original requirement that accessed a database. SQL server a widely-used and supported database can be utilized in this post project design relatively easily because of the hooks built-in to the .NET framework and the C# code. The basis of connection and implementation is enabled through the use of the SqlConnection class in the C# program to initiate a connection to the SQL database containing parametric data. The SqlConnection class is used in a resource acquisition statement in the C# language and must have the Open method called; this allows the query of the database with SqlCommand’s. The usage of this SqlConnection in a using type resource acquisition statement for use in Sql for the C# language. The SqlConnection has a constructor that requires a string reference pointing to the connection string character data. This connection string is often auto-generated by the dialogs in Visual Studio that the automated system must include in the SqlConnection code before the use of the SqlCommand object to perform a database query. By using the using statement construct in the C# it will lead to a more reliable, stabile C# database project. This application could not gain access to the database in a remote laptop atmosphere due to potential ITAR International Traffic in Arms regulations. Therefore, for demonstration purposes the project data-sets were hard coded into varying arrays and tested for functionality against test cases documented in a cited paper appendices. Automatic Data ExtractionThe automatic data extraction tool can overlap any parametric data-set to complete the viewing of any data-set from more than one viewpoint. The parametric historical data based on wafer manufactures like ON Semiconductor wafer fab 0.6um products or TSMC Semiconductor wafer fab 0.25um products are easily implemented within the parametric selections that are called and analyzed for outlier detection. For instance, the tool can view parametric historical data based on the PIC product identification code. 0.25um products like SRAM synchronous random access memory that statically does not need to be periodically refreshed. And DRAM dynamic random access memory that stores each bit of data in a separate capacitor within an integrated circuit. Both of these memories store the data in a distinctly different way and therefore have some differing data based on the architecture of the integrated circuit. However some parametric values will correlate between both memories built at the same wafer foundry, and can used to detect parametric trending within the wafer fab. This trending could be as simple as lower resistivity metal lines that exhibit faster access timing ac’s between both devices. These trends can be used to validate transistor design models, and help designers match the design simulations to real world silicon. This application will provide an important statistical data loopback to the original design models for verification. All data-sets were hand generated or based on cited works due to potential ITAR regulations. Real device parametric data was not used for this thesis project.Automatic Statistical Modeling Automated data modeling and goodness-of-fit tests are contained within the normalcy test of Anderson-Darling. The Shapiro-Wilk test is a slight modification of the Anderson Darling test but not as powerful, the Skewness-Kurtosis tests are better predicting larger data-sets and therefore were not chose over the functionality of the Anderson Darling. The only test implemented in this project was Anderson-Darling test for the null hypothesis. Through this test the user can gain a greater confidence that the data-set being analyzed is from a normal distribution and will provide greater confidence in to the outlier analysis being performed will be correct. This project implemented a 75% percentile analysis of the normalized data identified by the Anderson-Darling test described throughout this document as the basic default operation. The implementation followed a screening for the bottom 12.5% of the data or visually the left side of the bell curve. Then the tool screens for outliers on the upper 12.5% beginning at 87.5% by calculating these values. The histogram is then configured to show the distribution with the new lower and upper fences set. The toolset has other statistical modeling abilities including standard deviation, inter quartile analysis as options for the user to modify if needed but the default automated operation is the inner 75% percentile analysis..Normal Distribution CurveIf the data-set is determined by Anderson-Darling analysis to be bimodal or a non-normal distribution, it will be shown in a histogram format requiring user intervention. Bimodal distributions are a continuous probability distribution with two different modes; these modes appear as distinct peaks in the distribution diagram. The user is required to pick the upper and lower fence of the data to be analyzed for 75% percentile analysis, in the case of a bimodal distribution the user will have to pick one of the distinct peaks, or remove the entire population. This is the only non-automated portion of the project that requires user intervention by setting distribution of the data-set. Once the toolset knows what data in the data-set needs be analyzed for outliers the toolset can continue to perform the modeling on the data-set to locate potential outliers Future enhancements to the outlier detection system for non-normal data sets will utilize machine learning. All user defined changes to data-sets will be captured into the database. These user defined changes will specifically capture the user, the parametric data type, data-set, test equipment, lower, and upper fence chose by the user. Then based on the decisions made by engineers on the automated non-normal distributions, a future learning algorithm can start to be developed. However this project and tool-set is launched as a beta version prototype model for evaluation. Based on the evaluation of the beta version additional requirements may be realized and required in the released version. The small company requirements for this toolset deem the immediate release of a beta version for wafer test quiescent sub threshold leakage current measurements for out of family devices. (Gama, J., Rodrigues, P., and Sebastiao, R 2003)The effect of this software development methodology will be realized in the maintenance stage of this design. A well documented fact that the maintenance stage is the longest and the most expensive to the software project. Every attempt will have to be made to understand the costs involved with future enhancements and requirements. Releasing an incomplete beta version of the software for user evaluation, changes in scope, the need for future requirements and enhancements will certainly be a possibility as the known problem domain is still probably unknown? Bimodal Distribution CurveTesting and Prototype EvaluationThe outlier detection system design implementation utilized small author generated data-sets as the basis for the test cases used on the automated system implemented in C#. The exception of two data-sets described in the Asphalt Study cases 195 and 196. (Holsinger, R., Fisher, A., and Spellerberg, P 2005) These data-sets were listed in the document cited within paper as Precision Estimates for AASHTO Test Method T308 and the Test Methods for Performance-Graded Asphalt Binder in AASHTO Specification M320. For specific testing of the Anderson-Darling implementation, there are extensively documented test cases in the appendices of this document. The use of a non-automated Excel spread sheet with Anderson-Darling test statistics A and A2 calculations for the Anderson-Darling Normality Test Calculator. (Otto, K.2005) Identified data-sets are random representations of real data-sets but still are limited to utilize only a specific single device parameter within these randomly created data-sets, with the exception of the asphalt study case 195 and 196. These data-sets were not pre-loaded into a database, but instead hard coded into the application for ease of use and testing criterion for initial prototype correct functionality. Outlier analysis is then performed on a default inner 75% percentile analysis. The inner 75% percentile involves indentifying data values that fall with the first 12.5% of the data and the last 12.5% of the data from the mean as outliers with the normal distribution. The application can also filter for detecting outliers and extreme values based on interquartile ranges. These equations are in the form Q1 = 25% quartile, Q3 = 75% quartile,IQR = Interquartile Range difference between Q1 and Q3 OF = Outlier Factor EVF = Extreme Value FactorMild value detection:Outliers: Q3 + OF*IQR < x <= Q3 + EVF*IQR Outliers Q1 - EVF*IQR <= x < Q1 - OF*IQRExtreme value detection:x > Q3 + EVF*IQR and x < Q1 - EVF*IQRAlso presented in a histogram format of the standard sigma deviation from the mean, the user can pick any number of sigma or standard deviations from the mean data of the histogram. The application also presents the data in an all data within the data-set pass histogram format. This mode is used if the user needs to see a histogram format of all data points within the data-set. The all data format histogram is presented to the user with the ability to manually set the fences for upper and lower limits. This functionality allows the user to manually pick the outliers based on an upper and lower fence choice. It can also manipulate any data-set outlier to the user requirements and will be tracked and saved in the data-base as described in section Automatic Statistical Modeling section of this paper to eventually become part of a machine learning update version.Analysis of ResultsThe Anderson-Darling test for normality are one of three general normality tests designed to detect all departures from normality, and is sometimes touted as the most powerful test available. The Anderson-Darling test makes use of the specific distribution in calculating critical values, because the critical values do depend on the specific distribution being tested. This has the advantage of allowing a more sensitive test and the disadvantage that critical values must be calculated for each distribution or data-set that it is tested against. The Anderson-Darling test while having excellent theoretical properties does appear to have limitations when applied to real world data. The Anderson-Darling mathematical computation is severely affected by ties in the data due to poor precision. Ties are data points or values that match exactly other data points within the data-set. When a significant number of ties exist, the Anderson-Darling will frequently reject the data as non-normal, regardless of how well the data fits the normal distribution. The prototype application will still present the data in a histogram format for the user to custom set the upper and lower limits of the data-set. After a short analysis the user will notice the data-set is normally distributed due to the histogram being presented by the application. The user can still perform the 75% inner percentile analysis, or any other available in the application. While potentially creating a need for the user to evaluate more data-sets that have multiple ties of the same data, it by no means detracts from the functionality of the application, validation to the original requirements were still met. Additional conditional statements were hardcoded into the application that dealt with the mathematical calculation of Anderson-Darling A and A2 test statistic by checking for a unique divide by zero calculation. When computing these values the standard deviation is tested to be greater than zero or the application will set the value Anderson-Darling value to positive infinity. The calculations for the value of Anderson-Darling A and A2 test statistic by dividing by the standard deviation would result in a calculation that produces an application crash. Similar to the multiple ties limitation of the equation more serious mathematical errors occur when all the data values are the same resulting in a standard deviation of zero. Adjustments were not made to the base Anderson-Darling equation. Although the results of the check for normality on modified versions of the equation are documented within formal papers that adjust the base equation for a specific fit. These papers take the equation and modify the integral then verify the functionality by comparison by the use of some other normality test, like of Chi-Squares goodness of fit tests for instance. But even after equation modification the original limitation is still left intact with the inability to deal with multiple ties within the data-set. (Rahman, M., Pearson, M., Heien, C.2006)Investigated but not completed in this project, the added functionality to cluster the data by using a hybrid format with the standard deviation to overcome the limitation of Anderson-Darling on data ties. (Jain, A., Murty, M., and Flynn, P 1999) The K-Means Clustering algorithm based on data clustering methodology was developed by J. MacQueen in 1967 and then again by J.A.Hartigan and M.A. Wong in 1975. It is an algorithm to classify or to group objects based on attributes or features into a variable. The number of the group, K is a positive integer number and the grouping is done by minimizing the sum of squares of distances between data and the corresponding cluster called a centroid. The K-mean clustering only purpose is to classify the data into groups or “clusters” of data. (Technomo, K 2006)Clustering Algorithm FlowchartThen by using these clusters created in the K means clustering groups by passing in a hybrid format to verify against the standard deviation of the normal distribution. The data can then be automatically reviewed to verify if the clusters of data are within the standard deviation set for the distribution. If these clusters of data reside within the standard deviation set by the application usually a value of one standard deviation. Then the assumption can be made that the data within the distribution is from a normal distribution. If the data had too many ties within the data-set and would fail the standard Anderson-Darling p-value tests if the data-set were still normal. By contrast it would not fail this hybrid cluster with Anderson-Darling test and would not need user intervention to automatically to perform a 75% inner percentile analysis for outliers. The k-means algorithm can be used as an automated function when the Anderson-Darling p-value test fails on a data-set, as an auxiliary test. If the k-means clustering test also fails then the data-set would presented to the user in histogram format requiring user intervention as a non-normal distribution. Another mathematical equation the k-sample Anderson-Darling test for removing large number non-automated data-set histograms from this prototype. While the potential for this test to being statically the same as the k-means hybrid solution for Anderson-Darling standard normalcy test. The details and the equation of the test are described below and are not trivial by any means. This test for normal distributions is a nonparametric statistical procedure that tests a hypothesis. The hypothesis is that the populations from two or more groups of data were analyzed are identical. In this equation the test data can be either presented in groups or ungrouped because the Anderson-Darling k-sample test will verify if a grouped data-set can modeled as ungrouped data-set, grouped data-sets will look like bimodal distributions and this fail this test. The k-sample Anderson-Darling test is shown below, note that variable L is less than n if there are tied data-point observationshj = the number of data values in the combined data-set sample equal to zj Hj = the number of data-values in the combined data-set sample less than zj plus one half the number of values in the combined data-set sample equal to zj Fij = the number of values in the i-th group which are less than zj plus one half the number of values in this group which are equal to zj where k is the number of groups, ni is the number of hits in group i, xij is the j-th hits in the ith group z1, z2 ..., zL are the distinct values in the combined data set ordered from smallest to largest Implemented in pseudo code for k-means Anderson-Darling to appreciate the complexity of this algorithmThis prototype application validated all requirements in a loosely written requirements document. However after data analysis the chance that the Anderson-Darling test for normalcy would fail normally distributed data-sets became an unforeseen issue while meeting documented requirements. The issue arose that additional user intervention was required to process potentially normal data-sets. If additional time were available at the time of this write-up, a more elegant algorithm would be implemented and tested for a more automated version of the prototype. This elegant algorithm envisioned would certainly have to implement some portion of a clustering methodology like k-means, or a hybrid of k-means clustering or k-sample Anderson-Darling testing. CHAPTER 4CONCLUSIONSIntegrated circuits are designed and simulated by designers and then sent off to a wafer fabrication to be built on wafers. Unfortunately wafer fabrication manufacturing and design defects of integrated circuits are only discovered during the electrical test of the device. Circuit manufactures traditionally use a method that would set an upper and lower limit on a parametric measurement to detect circuit failures. This methodology has become only a portion of the criteria that is now required to discover potential defects of small geometry integrated circuits. Integrated circuit manufacturers are now employing additional methods of failure detection including wafer test structures. These methods include high percentage failure mode testing, design for manufacturing simulations, including lowering the FIT requirement because the potential returned merchandise costs are usually lower than the costs of more rigorous testing of the integrated circuit.Circuit design and manufacturing has become increasingly competitive. More companies are now utilizing a fab-less wafer manufacturing system. These companies design, assemble, and test the integrated circuits in their facility. Then pay to have the wafers manufactured at an outside facility where the company does not own the wafer fabrication facility. Being a fab-less microelectronic company introduces even more risk to the development of quality product because specialized flows and tailoring are not available in a non company owned facility; the benefit is reduced overhead capital expenses.Radiation tolerant circuits typically are only built in small quantities within a well documented qualification process. Some companies have developed a manufacturing process to produce a range of specific radiation-hard electronic products using rad-hard by design techniques or by adding rad-hard doped trenches. Commercial foundries typically use standard material for all material manufactured in the facility. By contrast radiation tolerant circuits require nonstandard materials, incorporating epitaxial layers, insulating substrates; doped trenches in the silicon that enhance radiation tolerance. The use of non-standard materials makes out of family detection much more difficult due to the inherent parametric leakage these techniques introduce and the use of non-standard materials and processes.A small fab-less rad-hard circuit company needs a tool that can process relatively small amounts of individual device data based on small lots of the electrical characteristic data. This small amount of data is subject to marginality effects of a rad-hard circuit effects that cause wafer to wafer variability. These specialized circuits require nonstandard starting materials, incorporating epitaxial layers, insulating substrates; doped trenches in the silicon that enhance radiation tolerance, but introduce marginality into the circuits with increased sub threshold leakage effects. To compound the issue, this data being analyzed within these small lots must be used to guarantee a very high FIT rate. Then used on potentially high reliable products that provide service within a mission critical sub system. Lessons LearnedThis automatic parametric outlier detection system prototype will utilized an existing algorithm Anderson-Darling to test for normal distribution on the dataset. This test is a general normality test designed to detect all departures from normality within a dataset, and is generally believed to be the most powerful test for normality. The importance of checking the data-set for normality was very important or the consequential methodology for choosing outliers may produce incorrect results. If a dataset was deemed a normal Gaussian distribution by the automated Anderson Darling test? The prototype would then perform an inner 75% percentile outlier detection technique on the dataset, and present the user with a histogram distribution of the data. Non-normal datasets were presented in a histogram format requiring user intervention. The user was required to visually inspect the data and set the upper and lower fences for a normal distribution. The remaining steps are the same operations as those discussed in normal distributions.The Anderson-Darling test for normality is one of three general normality tests designed to detect all departures from normality, and is sometimes touted as the most powerful test available. The Anderson-Darling test, while having excellent theoretical properties, does have limitations when applied to real world data when a data-point in the set ties or matches another data-point .When this happens the Anderson-Darling mathematical computation is severely affected by due to poor precision analysis of the data-set tails. Therefore when a significant number of ties existed within a data-set the Anderson-Darling frequently rejected the data as a non-normal distribution. This was done regardless of how well the data fits the normal distribution. This action required user intervention on a false negative condition, the automated functionality was then compromised b y the additional user intervention.Validation to the original requirements was completed at project end to the original requirements. The requirements phase of this project evidenced a lack of appreciation for the true complexity of the project. This was especially true as certain aspects of the functionality of the Anderson-Darling test for normality within the system automation features from the application to be less than optimal and require more user intervention than anticipated. This lack of appreciation for the complexity of evaluating a normal Gaussian distribution on every data-set presented to the prototype system is simple in theory, but became a large part of this project. Eventually became a larger portion of the project and required additional research to indentify future enhancements to the system. The problem domain was documented as a set of requirements by managerial stakeholders only to produce a robust beta-release outlier detection system. The additional requirement of detection within small data-sets was identified in the requirements documentation. This requirement evaluated small dataset distributions using mathematical equations for normalcy required significant hours of research. The requirements also included evaluation of the dataset into a histogram format on non normal distributions. After user intervention a 75% percentile analysis was always a requirement, but as the project entered the testing phase this default situation occurred more often than desired in an automated sub-system, although it still met validation requirements.During the design phase a significant amount of additional methods that became more complex to implement as important algorithms of the system. This became especially true when passing data back and forth from complex mathematical equations within numerous arrays’s of the data as it was calculated and maintaining these values. The use of case diagrams would have helped in the design phase as the design was mainly. This of course equated to additional time being required within the test phase because of the complexity of the Anderson-Darling mathematical equation to check for normal distributions. These deterministic equations easily produced divide by zero errors that were not taken into account during the design phase, and the number of ties within data-sets cause the tool to have false normalcy failures. Post prototype validation analysis of the requirements revealed minor departures of the software system within certain data-sets, this analysis revealed potential non-automated limitations of the software system. Therefore additional research was started in the final testing stage of the parametric outlier design to solve these non-automated issues. Clustering methodologies created in the k-means clustering groups could be passed in a hybrid format to verify against the standard deviation of the normal distribution. This data can then be reviewed to verify if the clusters of data are within the standard deviation of the distribution. If these clusters of data reside within the standard deviation set by the application usually a value of three standard deviations. Then the assumption can be made that the data within the distribution is from a normal distribution. The other promising methodology would be the use of the k-sample Anderson-Darling test a nonparametric statistical procedure that tests the hypothesis that the data from a data-set of two or more groups of data is identical and therefore from a normalized distribution. The k-means Anderson-Darling appears to be the most promising solution, however the mathematics required to implement, verify correct operation, and validate to requirements would be a significant task. Future enhancements could evaluate both methodologies of clustering for processing and computational accuracy. The methodologies certainly require additional research before implementation, in addition to more robust test modules for testing these complex algorithms.BIBLOGRAPHYAbrain, A.: SWEBOK, Guide to Software Engineering Body of Knowledge, IEEE Computer Society, 2004.Bhattachatya, S., and Chatterjee, A. Optimized Wafer-Probe and Assembled Package Test Design for Analog Circuits. ACM: Transactions on Design Automation of Electronic Systems (TODAES), Volume 10 Issue 2, April 2005.CMMI - Capability Maturity Model Integration Version 1.3, Nov 2010Dy, J, and Brodley,C.. Feature Selection for Unsupervised Learning. : The Journal of Machine Learning Research, Volume 5, December 2004. Gama,J., Rodrigues,P., and Sebastiao,R.: Evaluating Algorithms that Learn from Data Streams. ACM: SAC '09: Proceedings of the 2009 ACM symposium on Applied Computing, March 2009.Gattiker, A.: Using Test Data to Improve IC Quality and Yield. IEEE Press: ICCAD '08: IEEE/ACM International Conference on Computer-Aided Design, November, 2008 Jain, A., Murty, M., and Flynn, P. Data clustering: a review. ACM Computing Surveys: Volume 31,?Issue 3, Pages: 264 – 323,?September 1999.Jebara, T. Wang, J., and Chang, S.: Graph Construction and b-Matching for semi-Supervised Learning. ACM: Proceedings of the 17th ACM international conference on Multimedia, October 2009.Holsinger, R., Fisher, A., and Spellerberg, P., Precision Estimates for AASHTO Test Method T308 and the Test Methods for Performance-Graded Asphalt Binder in AASHTO Specification M320. National Cooperative Highway Research Program, AASHTO Materials Reference Laboratory, Gaithersburg, Maryland, February, 2005Marinissen, E. Vermeulen, B., Madge, R. Kessler, M., and Muller, M.: Creating Value through Test: DATE '03: Proceedings of the conference on Design, Automation and Test in Europe - Volume 1, March 2003.Moran, D., Dooling, D., Wilkins, T., Williams, R., and Ditlow, G.:Integrated Manufacturing and Development (IMaD). ACM: Supercomputing '99: Proceedings of the 1999 ACM/IEEE conference on Supercomputing, Jan 1999.Otto, K.: Anderson-Darling Normality Test Calculator, Normality Test Calculator.xls, Version 6.0, 2005.Perez, R., Lilkendey, J., and Koh, S, Machine Learning for a Dynamic manufacturing Environment. ACM: SIGICE Bulletin, Volume 19, Issue 3 , February 1994.Pyzdek, T.: Histograms: When to Use Your Eyeballs, When Not, Looks can be deceiving. Quality Digest Magazine, Volume 31, Issue 3, March 2011.Rahman, M., Pearson, L., and Heien, H.: A Modified Anderson-Darling Test for Uniformity. BULLETIN of the Malaysian Mathematical Sciences Society (2) 29(1) (2006), 11–16Romeu J.: Selected Topics in Reliability and Assurance Related Technologies, Reliability Analysis Center, Volume 10, Number 5, May 2005Saab, K., Ben-Hamida, N., and Kaminska, B.: Parametric Fault Simulation and Test Vector Generation. ACM: Proceedings of the conference on Design, automation and test in Europe, January 2000. Singh, A., Mani, M., and Orshansky, M.: Statistical Technology Mapping for Parametric Yield. IEEE Computer Society: ICCAD '05: Proceedings of the 2005 IEEE/ACM International conference on Computer-aided design, May 2005.Technomo, K.: K-Means Clustering Tutorial, pdf file 2006 with reference to website, , 2006.Veelenturf, K.: The Road to better Reliability and Yield Embedded DfM tools. ACM: Proceedings of the conference on Design, automation and test in Europe, January 2000.Wang, W., and Orshansky, M.: Robust Estimation of Parametric Yield under Limited Descriptions of Uncertainty. ACM: ICCAD '06: Proceedings of the 2006 IEEE/ACM international conference on Computer-aided design, November 2006.Walfish, S.: A Review of Statistical Outlier Methods: PharmTech - A Review of Statistical Outlier Methods, Pharmaceutical Technology, August 2006APPENDIX AData Set Parametric Outlier Dataset 1APPENDIX BData Set Parametric Outlier Dataset 2APPENDIX CData Set Parametric Outlier Dataset 3APPENDIX DData Set Parametric Outlier Dataset 4APPENDIX EData Set Parametric Outlier Dataset 5APPENDIX FData Set Parametric Outlier Dataset 6APPENDIX GData Set Parametric Outlier Dataset 7APPENDIX HData Set Parametric Outlier Dataset 8APPENDIX IData Set Performance Graded Binder Sample 195APPENDIX JData Set Performance Graded Binder Sample 196 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download