Etsmtl.ca



Stewart G. Crawford & M. Hosein Fallah

Software Development Process Audits - A General Procedure[1]

AT&T Bell Laboratories, Holmdel, New Jersey 07733. USA

ABSTRACT

To assist development organizations in improving software quality and productivity, the AT&T Bell Laboratories Quality Assurance Center created a Design Quality Group to independently evaluate the software development processes and associated development products of our AT&T projects. These software development process audits examine software engineering techniques and tools in practice, as they fit into the overall development environment. Our strategy behind these audits is to assemble a team which, with the involvement of the developers and their managers, will: characterize the existing development process; identify project strengths and areas for improvements; and recommend possible improvements. This paper details the general approach behind this strategy.

1. BACKGROUND AND OVERVIEW

During the last few years, the AT&T-Bell Laboratories Quality Assurance Center (QAC) conducted several audits of software and hardware development processes and of documented requirements and craft interface specifications. Our objectives centered around identifying areas of potential quality and productivity improvements. The QAC performed these audits by invitation from the respective development organizations, and their positive response led the QAC to create a group dedicated to software development process audits. This paper defines our general approach; the specifics of any one particular audit, however, will vary depending on the breadth and depth of the audit scope negotiated with the software development project.

The audits we describe here mark a departure from the adversarial relationship generally implied by the term "audit". Our primary objective is to assist development organizations in improving software quality and productivity through independent evaluations of their software development process and associated development products; additional benefits include the transfer of proven ideas and technologies in software development from other

projects, consolidated upward feedback to project management, and pointers to further opportunities for education and training.

Several projects within Bell Labs have requested these development process audits to date, and feedback from their management has been. positive. QAC's organizational independence from the development organizations allows for a perspective free of the local bias surrounding development issues; our interviews and discussions with project management provided time for them to stop and take a long view of the global development issues. We can audit a project at anytime during the project lifecycle, but if done early enough, the response to the audit findings can improve the quality and scheduling of the product currently under development. In one particular project audited late in the design phase, our findings and recommendations led the project to enhance the baselining of intermediate lifecycle deliverables, comprehensively review the remaining project schedule, and improve the project tracking system; these activities revealed several trouble areas looming in the near future for that particular software release, and the project then addressed the issues before they became serious problems.

The distinction between a development process audit and a project management audit is primarily that of focus. The former focuses on the techniques, tools, documents, and procedures through which an idea becomes a supported product, while the latter focuses on the organization and management of that whole effort. The scope of these two types of audit will overlap somewhat since by their natures, a process and its management are closely coupled. Figure 1 illustrates these ideas: The upper dashed circle shows the scope of concerns in a process audit, while the lower circle depicts the scope of a project management audit.

[pic]

Figure 1. Overview of a Development Project

Our strategy behind these audits is to assemble a team which, with the involvement of the developers and their managers, will :

- Characterize the existing development process by :

o Analyzing development methodologies, software quality assurance plans, and related practices and procedures.

o Interviewing supervisory and non-supervisory personnel

o Attending project meetings, reviews, and inspections, and

o Reviewing selected development products;

- Identify project strengths and areas for further improvements; and

- Provide recommendations for improving the software development process and the quality of its software products.

Positive interactions and close coordination between the projects management and the audit team are important to the effectiveness of these audits, and these contributed greatly to our previous successes. The core audit is a time of the team’s almost continuous presence within the development process; this is the interval when we interview the staff and sit in on many project meetings. Our presence will be non-threatening only when management support of the audit is understood by the project team.

2. A GENERAL AUDIT PROCEDURE

. Prepare and Plan

Once a project requests an audit, some groundwork is needed before the activities can begin. The pre-audit activities include the following :

2.1.1 Clarify Global Goals and Objectives: Due to the flexibility built into our audit service, the initial request may not be sufficient to define the task adequately. Thus the first planning activity is to clarify what the audit's customer, usually some level of project management, is seeking through the audit.

Discussion with he project of the managers' needs and of our expertise are crucial to the proper initial focus of our effort.

2.1.2 Assemble an Audit Team: Depending on the size of the project, the team may consist of three to six people: one person should be designated as the team leader. Each team will include members with experience and expertise in process audits, software development, and software quality assurance. Additional subject matter experts may be called in as necessary.

2.1.3 Understand the Project: Understanding the project is a prerequisite. A tutorial or an overview presentation to the team by the project is a useful start. The team should also learn about the project by reading project descriptions and other related documents that may be available, and by talking to the project contacts.

2.1.4 Understand the Organization: In an effective audit, it is crucial to know who is doing what in the project. Organization charts are often outdated and the group titles are generally insufficient for identifying the functions and activities of the project staff. The organization chart should be reviewed with the project contacts to identify the functional responsibilities of the people.

2.1.5 Define the Audit Scope: Negotiation of the audit's scope with the project's management is the foundation for further activities. Each audit will be customized to the needs of the particular project, thus, the breath and the depth have to be defined. Depending on the scope, the audit may focus on some particular phase(s) of the development process, or its depth could vary depending on the audit objectives, resource constraints, and timing.

2.1.6 Develop an Audit Plan: The plan is the "road map" for the audit. The plan should reiterate and expand on the agreed scope (the Why the audit), answering the following questions: What will be audited? Who are the auditors? When can the various audit activities be expected to happen and finish? Where and How will the audit activities take place?. A preliminary draft of the plan should be reviewed with the Project's management to assure a mutual -consensus on the audit activities and schedule.

2. Characterize the Development Process

With the initial groundwork laid, the formal audit activities begin in earnest. Our intent is to uncover as much as possible within the established scope and time constraints. The team should analyze not only the methodology as it is written down, but the "de facto" methodology, i.e., the methodology as it exists in practice. Our methods of collecting data include analyzing the development process documentation, interviewing project personnel, observing project meetings, and reviewing development process products.

2.2.1 Analyze methodologies: The first step in characterizing the development process is to study the documented methodology. Besides providing a general process overview and background for all the ensuing activities, we can study completeness of lifecycle activity documentation, evaluate adequacy of the written methodology in providing clear and consistent guidelines for the development process, and formulate initial hypotheses to follow up in interviews. Any documented development methodology, Software Quality Assurance Plan (SQAP), or other documented practices and procedures should be included in this review.

In analyzing development methodologies, either in the form of a handbook or a series of documents, we would look for: task definitions, descriptions; and ownership; solution procedures for tasks; review and inspection procedures; a well defined lifecycle, broken into activities; well-defined handoffs between activities; and any standards currently in use. We would want to look to any related practices and procedures for measurement and tracking procedures, monitoring and control mechanisms, and efficient organizational interfaces. The above lists are not meant to be all-inclusive, but they are a good starting point.

In general, we would get the necessary documents through the project interface contact or the project librarian. After reading and annotating the documents, all the auditors, in a group meeting, would discuss the documents, evaluate their adequacy, and formulate the first set of hypotheses regarding the process as it is documented.

2.2.2- Interview Developers: The core audit is the main time for interviewing developers on site and attending any relevant project meetings. The auditors should interview as a group or in subgroups of at least two : multiple interviewers provide multiple viewpoints which helps balance any individual bias. Interviewing people near the project site but not in their offices, removes many distractions and may help people open up more since they're slightly removed from their usual environment.

We would want to interview people in the major functional areas of software development included within the scope of the audit. A non-inclusive list may include staff and their managers responsible for:

system definition

system requirements

system architecture

software requirements

software architecture

software design

software code

software unit test

software integration test

system test

customer interface

end-user training installation

software maintenance

problem reporting & corrective action

project management & configuration management

quality control

methodology documentation & updates

developer training

A primary reason for interviewing people is to characterize the "de facto" methodology. Does it follow the documented methodology? Is it consistently applied across people? Is it effective? Additionally, we would want to: gather information to support or refute pending hypotheses; formulate new hypotheses as new information is gathered; characterize the visibility of project tracking and the effectiveness of project control; and gather upward feedback for the project management. Basic investigative and journalistic interview techniques would be the auditors' tools for this task; summary question checklists can help guide initial interviews.

2.2.3 Attend Meetings: The auditors should have a blanket invitation to any technical, administrative, or informative meeting, either formal or informal, the project holds during the time-frame of the audit. The purpose behind attending these gatherings is to: observe project communication; characterize general effectiveness of meetings, reviews, and inspections; and characterize compliance of reviews and inspections to any relevant standards.

The entire team (for large meetings) or a subgroup (for small meetings) would discretely attend and observe a project meeting. Discussing the meeting after it officially ends with attendees who happen to linger on is often of value in resolving any questions the auditors may have. The meetings observed would normally occur during the core audit, but they may occur before or after that time also, depending on when representative project meetings are scheduled.

2.2.4 Review Selected Development Products: The team may decide to review in-depth a sample of products representative of the various lifecycle activities within the scope of the audit. Such reviews serve to: examine the products' fitness-for-use in downstream development activities; characterize compliance of the products to existing standards (project or industry); or explore project details for necessary background.

Individual auditors examine selected products within their field of expertise. These reviews would be oriented toward substantive, not editorial or typographical, comments. When multiple auditors review the same set of documents, they would need to discuss and integrate their comments.

3. Evaluate and Report

The team should get together frequently during the core audit period and review their observations. These meetings help ensure that the audit is progressing according to the plan, provide opportunities to formulate and revise hypotheses, and adjust the planned activities, when necessary. The auditors may decide to include other people in the list of interviewees, plan to attend additional meetings, or review other relevant documents.

Following the core audit, the teams findings should be brought together, combined, evaluated, and documented in an audit report. As part of this process, any open items should be followed-up, and the auditors should obtain additional information and seek clarification on issues that do not have the team's consensus.

In preparing the audit report the following should be considered :

• Report strengths as well as areas for improvement. Although the main objective is to identify the areas with potential for quality and productivity improvement, it is equally important to identify those aspects of the process that are done well. This recognizes the successful efforts within the project to produce quality software, aids the project by itemizing things done well which shouldn't be changed for the worse, and helps us identify good methods and practices that have potential for use on other projects.

• Separate major from minor concerns. The team may find many weak spots. The few major problems should be separated from the others and given special emphasis and discussion; this separation assures they don't get lost amidst everything else.

• State the problems succinctly and state why they are problems. The problems should be communicated to the project very clearly. By stating why the team thinks something is a problem, we provide the project with the assumptions, perceptions, and the reasoning that led to the team's conclusions. This information helps the project to assess the audit findings in light of their more detailed project knowledge.

• Present recommendations with due caution. When possible, the team should recommend viable solutions to problems they find while examining the development processes. But, the recommendations should be expressed as suggestions that the project may want to consider in developing its own plan of action. The team should be cognizant of the complexities of the technological, functional, and organizational aspects of a development project as they relate to the recommended solutions. while the team's recommendations may provide a good start for addressing a problem area, the people within the project are in a better position to decide what solution is most likely to work.

The audit report should be reviewed with the project managers before being finalized. Such a review helps to remove any misunderstanding that the team may have. It also provides an opportunity to find out if there are any major disagreements between the team and the project managers, and document the opposing views if necessary in the report. With the issuance of the final report, the team may also make a formal presentation of the findings and provide a forum for discussion of the recommendations.

The report is meant only for the managers of the project. It is their prerogative to distribute the report either up or down managerial lines or outside the project. The team members will forward any request for an Audit Report to the appropriate project manager. The team members may report some generalities about the audit to their management as needed to keep them informed of the audit activities.

4. Follow-up

We will continuously try to improve the audit process. Feedback from the projects provide valuable information for achieving this objective. A follow-up with the project right after an audit may identify some of the changes that could strengthen the audit process. The effectiveness of the audit, however, can only be assessed in due time. Hence, the team leader should make another follow-up eight to twelve months after the audit. In this follow-up, we solicit an assessment of the benefits realized (or expected to be realized) by the project so that the cost-effectiveness of the audit can be evaluated. The project may also have other suggestions on ways to further improve the audit process.

-----------------------

[1] Proceedings of the 8th international conference on Software engineering

London, England, Pages: 137 – 141, 1985.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download