The Software Development Process



Apply standards to script code

Inside this reading:

The software development process 2

Standards for design documents 4

Development system standards 5

Coding standards and style 6

Code documentation standards 8

Why conform to standards? 13

Security policy for scripting 14

Summary 16

The software development process

When working in a software development workplace, you first need to know the details of how they go about developing projects. Most software companies or departments will have a particular development model that they follow, that prescribes which activities will be performed, and when.

A Software Development Life Cycle (SDLC) is a conceptual model, or methodology for the development process. These models are theories of how to best go about the process of software development. Having selected a model, a programming project team will follow the prescribed process during the course of development.

There are a number of different models, including for example the 'Waterfall model' the 'Spiral model' and 'Agile Development'.

For our purposes, a very simple sequential model (the 'Waterfall' model) will suffice to describing the development process.

The Waterfall model is so called because the stages theoretically occur in strict sequence, without going back to revise previous stages. The stages may include:

• Problem definition.

• Scoping – Decide what the system boundaries will be.

• Analysis – Look at the structure of the problem.

• Design – Define the solution and create the algorithm.

• Code – Write the instructions in a computer language.

• Test – apply tests to ensure the code works as specified.

• Debug – Ensure the program produces expected results.

• Document – Write user and technical documentation.

• Install – Deploy the application to do its job.

• Maintain – Fix bugs, implement minor enhancements.

This model is a very simple view of the whole process but it serves as a starting point for understanding how good software programs are developed. Today, the Waterfall model is generally recognised as too simplistic for software projects of any significant complexity.

You will understand more about the software development process as you develop more complex programs, and much more if you work on a large software project.

Each of these stages can be a discipline in its own right, and in a large software engineering shop, each stage will have specialists. For example, some people may specialise in requirements analysis, some on analysis and design including such tools as the Unified Modelling Language (UML), others may specialise in the testing of software, still others may specialise in documentation and so on.

Abbreviated development cycle

Each stage in the Software Development Life Cycle is important to the success of a software project. The programs that you will develop while learning are necessarily small and simple, but we will ask you to adopt an abbreviated development cycle, as a token of this more extensive process. Here are the steps you should take:

• Analysis: Read the problem definition carefully and think about the problem. How general is the problem you are trying to solve? Has it already been solved in a different way? Is it related to other problems you have solved? What resources are required to solve the problem?

• Design: Develop an algorithm in abstract terms using pseudocode or flowcharts.

• Code: Translate the pseudocode to an actual program. For this step, you need to get the syntax exactly right, and learn about the details of the data structures used and the objects and methods you will use from the standard library.

• Test: Fix any syntax errors and execute the program. Test the program with a range of inputs, checking that the behaviour of the program is correct for each input. Debug the program to remove any logic errors.

• Document: We will restrict ourselves to documentation embedded in the program itself. Include an introductory comment at the start of you program, and liberal comments throughout explaining the workings of your program.

Scripts must be developed properly too

You may think that scripting represents a special case for development that does not require a development process. Especially if you are working alone, it is all too easy to develop 'quick and dirty' scripts that have not been properly designed, documented or tested. Some of the arguments you might use are:

• "I'll only use it once then throw it away"

• "No-one will ever use it but me, and I know how it works!"

• "It's just a few lines."

• "It has to be finished today!"

The problem is that these statements do not remain true in the longer term. Consider that:

• Very often, a small script will be adapted or extended to solve similar problems.

• If you have to do a task once, chances are you will have to do it again.

• After several months break, you may have trouble understanding you own code.

• If you change jobs, what legacy will you leave for your replacement?

• If you don't have time to do it right, do you really have time to write a script at all?

Agile development

The full waterfall model of software development may not be appropriate to scripting. You may want to look at the agile development methodology instead, which combines the design and coding phases, and places a greater emphasis on testing, and refactoring of software during development.

In the sections that follow, we will outline the types of guidelines that you might have to follow, as they relate to the work of a programmer.

Standards for design documents

You will need to become familiar with whatever standards are used for the specification and design of software. You may need to:

• Learn how to read specification documents used by the designers. Often, the functional specification is the ultimate source for what will be provided to the client.

• Learn design systems and notations such as the Unified Modelling Language (UML), or learn to use design software tools.

• Formally document you own designs.

• Translate design documents into tests and code.

Development system standards

Each software house will have a particular development systems, an environment and a way of working that you will need to learn. Aspects of the environment may include the following:

Computer systems

You will need to learn the operating system that your company uses. This will likely be either a UNIX-like or a Microsoft operating system.

File management

Programmers deal with a lot of files, and the bigger the project, the more important to keep all files in order. File management for software development will include:

• Ensuring your environment variables are set correctly.

• Adhering to the local conventions for file and directory naming.

• Using the conventions for directory structures set by the development language, or locally.

• Correctly using the revision control system. Revision control systems allow the tracking of all changes to a document, keeping parallel versions, undoing changes and so on.

Development Tools

You will be expected to use the same suite of tools that your team is using. This may include for example:

• Computer aided design software.

• The language itself (Java, Python etc).

• The integrated development environment.

• Unit testing software.

• Bug tracking software.

• Debugger.

• Project management software.

Standard libraries

You will be expected to know not only the programming language, but also the set of libraries or Application Programming Interfaces (APIs) that are to be used. Wherever possible, you must use the standard features which are available, rather than reinventing the wheel.

These libraries may include:

• The Standard libraries for the language.

• Special purpose libraries for things such a graphics, networking, XML processing and so on.

• Specific APIs for the target operating system or for third-party software.

Security

Apart from the security considerations of your developed code, the environment in which you work may require you to follow security procedures. This may include:

• Maintaining correct permissions on files and directories that you are managing.

• Prohibition of copying files to portable media.

• Shredding of printouts after use.

Coding standards and style

The job of the programmer is to start with an abstract design and finish with source code that meets the requirements. Typically however, there are many solutions to the same programming problem, and no two programmers will finish with the same solution. We say that programming problems in general are 'underconstrained'.

These many solutions may behave similarly when viewed from the outside, but some of them will be superior to others on closer inspection. For example they may:

• be clearer and easier to understand

• contain fewer errors

• be more reliable under high load

• degrade gracefully instead of crashing when facing unusual conditions

• make better use of the standard libraries

• be easier to maintain

• be easier to re-use in other situations

• be faster, or more efficient in resources

• solve the problem with less code

Not every aspect of code quality will necessarily be maximised within one 'best solution'. But good programmers will consistently make full use of the available language features, the standard library, and the theory of algorithms to find solutions that work well.

Coding standards are an attempt to improve the quality of code in an organisation with straightforward 'rules of thumb' about many aspects of code, such as:

• Dos and donts.

• Simple rules for consistent style that improve the clarity of code.

• Rules for good documentation.

• When and when not to use more advanced language features such as exceptions, object-oriented programming, threads.

• Advice about efficiency of code.

• Advice about common difficulties encountered.

Obviously, you can only go so far with this, because you cannot always condense programming expertise and experience into simple rules. Most examples of coding standards place more emphasis on the simple aspects such as coding style which can be briefly described. Some more extensive coding standards are essentially small handbooks or summaries of programming theory.

But even simple coding standards can make a difference to quality, especially when there are teams of people working on the same project. The mere fact of having consistency is itself worthwhile, because the various programmers don’t have to continually adapt to other people's personal choice of coding style.

Reading activity:

At this point you should read the Python Style Guide in full, and browse through a few other coding standards, such as the GNU coding standard or the Java coding guidelines mentioned in the resources. Search for 'coding standards' on Google for further information.

Code documentation standards

In this section we will discuss the documentation of the code itself. The purpose of documenting code is to:

• Explain how your code can be used. For example, you may have developed a module to, say, display postscript images on the screen. The users of your module need to know exactly how they can use your creation.

• Explain the code itself. When your code needs to be modified, whether it be an enhancement or a bug-fix, other programmers need to be able to understand the workings of your code.

Note that we talking about the documentation of code by the programmer, within the source code file itself. We are not talking about technical writers explaining the work of programmers, as often happens in textbooks, FAQs, or other guides.

The mechanisms for documenting code include:

• Write self-documenting code.

• Use comments effectively.

• Use documentation strings.

Self-documenting code

Even without the use of comments, some programs are relatively easier to read. We say that these programs are 'self-documenting'. This is not an accident, but is due to the discipline of the programmer. With practice and attention to detail, you can learn to write clear code, just as you can learn to write clear sentences.

You should always strive to make the meaning and intent of your code as clear as possible, even without comments. A few of the things you can do to make you programs self-documenting are:

• Give variables the most meaningful and accurate name for their purpose.

• Similarly with functions, choose a name that describes clearly what the function does.

• Avoid complex or confusing code constructs. Some languages like C and PERL are notorious for difficult syntax, but most Python code is relatively straightforward. You should always strive to find the clearest way to express yourself in code, as you do in English.

• Do not try to be too smart when programming. In other words, try to choose the clearest, simplest way to do something rather than the smartest or 'coolest'. For example, do not use obscure language features or complex data structures if this can be avoided.

• Indent your code properly. Most languages use delimiters such as braces { } to indicate the nested structure of the code. In these languages, the compiler does not care about indentation. Correct indentation must be maintained by the programmer in order to make the code structure apparent to the reader. Unfortunately, different programmers have different ideas about which indentation rules are best. With Python, you do not have this problem. The indentation indicates the control structure to the compiler. If you don't indent correctly, you will either have a syntax error, or your program will not work as expected. Thus the structure of a Python program is always clear and consistent.

There are many articles that mention self-documenting code on the internet. Search for the term on Google, and skim several of the articles. One excellent article is 'Principled Programming', by Daniel Read at .

Comments

A comment is a piece of text in the source code that is meant for human readers. The compiler or interpreter will ignore the comments.

In Python, a comment is introduced using the hash # character. For example:

# count to three

for i in (1, 2, 3):

print i

Comments can be also placed at the end of a line:

DOZEN = 12 # constant

Comments are used by code writers to explain the purpose and intentions of the code or communicate that to others.

When code is changed or modified by a team of writers, code versions and records of changes may become necessary. Code versions and records of changes are often required by organisations when commissioning scripts; you may need to investigate the commenting requirements for the code you develop.

Good commenting practice

Finding the right thing to say in a programming comment is not always easy. Here are a few pointers:

• Not every line has to be commented. If you have done a good job with self-documenting code, many lines will not require comments.

• Most variable definitions could benefit from a comment explaining how they will be used.

• Do not just restate the code with equivalent comments. The following are not useful comments, as they simply restate what the actual code is doing.

avgMark = totalMarks / numMarks # calculate avgMark

print 'Average mark:', avg_mark # Print avgMark

• Similarly, you don't usually need to explain the normal operation of simple programming constructs (such as loops).

• A single-line summary comment before each if statement or loop is often helpful.

• It is most important to comment special conditions, assumptions, or complex logic.

• While you are learning to program, it’s better to err on the side of too many comments.

• It is very useful to take some of your old code (from say six months ago) and quickly try to understand it again. You will soon notice where you have not provided enough comments.

Block comments

Block comments are comments that extend over multiple lines, forming a block. Block comments are used to give a more extensive description of a section of code.

Some languages have beginning and ending delimiters for multi-line comments, for example:

/* This is a

multi-line comment

in Java

*/

Python does not have a true multi-line comment like this, so you must begin each line of a block comment with a new # character like this:

# This is a

# multi-line comment

# in Python

This is not a problem however, because the most popular editors for Python include facilities to comment or uncomment multiple lines of text automatically.

There is another reason that Python programmers do not need a special multi-line comment syntax. Many uses of block comments are more useful as documentation strings, as we will see in the next section.

Documentation strings

Comments are very useful for other programmers to understand your code. But to read the comments, you must open the source code file itself and read each comment.

Often the comments (block comments especially) contain summary information that is very useful to programmers as they work. For example, to use a function that appears in another module, you need to know its exact name, what it does, how many parameters it takes, and what it returns. Again, to find these details, you would need to open the file and check the comment above that function.

Therefore, it is much better to have this summary information available online, so that the programmer can browse through the available features of a software package. The current trend is to automatically assemble the prologue information of functions, classes and modules into convenient online documentation, normally formatted as HTML.

This is where documentation strings come in. Instead of using a plain block comment at the start of a module or function, use a Python docstring.

Example activity:

Try this example. Create a python module called myprog.py with the following contents.

"""myprog.py - Test of doc strings

This module doesn't really do very much, its just me testing

the great features of Python documentation strings.

"""

def doNothing():

"""doNothing - this function does nothing

This function takes no parameters, returns no value, and

does nothing. It's pretty useless really, but its well

documented.

"""

return

The strings between the triple quotes are normal Python strings (we need three quotes so they can extend over multiple lines). The first string describes the module as a whole, and it must be the first declaration that occurs in the module. Similarly, the second string is the first declaration in the doNothing function. These strings are given the special name __doc__ (starts and ends with two underscores).

Python docstrings can be accessed from code. If you add the following lines to the code above as the main program:

print myprog.__doc__

print myprog.doNothing.__doc__

you will find that the two documentation strings will be printed.

Documentation strings are typically used as prologues to modules (source code files), functions, and classes. A prologue is documentation that is included before a significant section of code.

For a function, the prologue will include such information as:

• A one-line description of the function's purpose

• Description of each argument's type and purpose

• Description of the return value

• Examples of using the function

• A more extensive description of how the function works including exceptions raised, side-effects,

The prologue for a module may contain:

• the purpose of the code

• the author(s)

• the creation date

• details of each modification made to the file

• the version number,

• brief descriptions of the classes and services made available by the module.

The docstring for a stand-alone script should be usable as the usage message for the user of the script.

The Python environment includes a module called pydoc, which allows you to view the documentation for any module. See the Python library reference for more information.

For information about how to use Python documentation strings, see Python Enhancement Proposal 257 (PEP 257 'Docstring Conventions': dev/peps/pep-0257).

Documentation of Test Cases

Testing is a vital part of modern software development. The test cases you write are an integral part of the software, and should be documented.

A convenient way to achieve this is to use the Python doctest module, which allows test cases to be interwoven inside the documentation strings. The example below shows how this is used with a simple function to reverse a string. Note how the tests are included in the docstrings. This not only takes care of the testing, but make it absolutely clear what the function does, and so also enhances the documentation.

import string

import doctest

def reverse(str):

"""Reverse a string

Inputs: a string

Output: the input string, reversed

Note: this is a very slow way to reverse a string and

is for demonstration purposes only. You should use the

standard library to do this!

Examples/Test cases:

Reversing a string gives the same characters in reverse

>>> reverse('hello')

'olleh'

Reversing a single character string gives the same string

>>> reverse('x')

'x'

Reversing an empty string returns an empty string

>>> reverse('')

''

"""

rev = []

i = len(str) - 1

while i >= 0:

rev.append(str[i])

i -= 1

rev = string.join(rev, '')

return rev

if __name__ == '__main__':

doctest.testmod()

If you run this program using the verbose ('-v') option, you will se a report similar to this:

C:\python>testDocString -v

Trying: reverse('hello')

Expecting: 'olleh'

ok

Trying: reverse('x')

Expecting: 'x'

ok

Trying: reverse('')

Expecting: ''

ok

Test passed.

C:\python>

Why conform to standards?

Programmers can often be a headstrong and unconventional lot, and they will often question any standards they come across. This is a healthy attitude - it's important that the reasons for any particular conventions or standards be made clear, and the standards be revised if conditions change.

While you can question anything, you should nevertheless do your best to conform to the standards in place during the lifetime of any project. Usually they are the result of experience, and will help because each programmer can use and understand the work of others more easily.

With everyone doing things in a similar way, the mechanics of getting things done should go more smoothly. There is less likely to be arguments about details that are not concerned with the project.

Refusal to conform to standards can have the following detrimental effects:

• Wasted time and energy.

• Disruption to the work of the project.

• Conflict with others.

• Difficulties with integration of code.

• Problems with the maintenance of code.

• Problems updating the code to new standards.

• The security features of a product may be compromised.

Security policy for scripting

You will need to become conversant with the security policies in your organisation. A security policy is a set of protocols and procedures for implementing security in an IT environment.

Note that computer security is a complex field in its own right. Here we will merely give an introduction to some of the issues.

Administration scripting can be dangerous

It should be quite obvious that scripting for computer or network administration can be dangerous.

If you are the administrator, then scripts run under your account will most likely have permission to do everything you can do manually. Therefore if there is an error in your script, it could have disastrous consequences. For example, you would not be popular if you accidentally delete a bunch of files, or change everyone's password!

Run scripts at the lowest permission level possible

If a script can do its job at a lower level of permission than you yourself possess, it makes sense to run the script at this level.

Keep your scripts secure

One basic step you can take to enhance security is to keep the script code itself secure. Nobody but the administrators should be able to read, much less modify, your script files. Likewise, never run any script if you don’t know where it came from, or exactly how it behaves.

Any files that your script uses or produces should be similarly secured.

Security holes in scripts

You need to beware of security problems in your script code. Here is an example of a very silly script with a gaping security hole:

import os

str = 'v'

while str != 'exit':

str = raw_input(':')

os.system(str)

This script will execute any system command the user inputs. If this script were to run in administrator mode, any user would have permission to run any command they liked! You should be aware of the security implications of using functions like os.system, the exec statement or the eval or execfile functions.

Your script should clean up after itself, and not leave the system in any lower state of security protection than it was initially. For example, any temporary files should be deleted after use, any altered permissions should be restored to their initial values, and all programs implementing security-related features should be running (eg virus checking software).

Make sure that the script cleans up regardless of error conditions which may arise. Exceptions can help you here. For example this code is dangerous…

disable_security_feature()

do_something()

enable_security_feature()

…but this code is less dangerous:

try:

disable_security_feature()

do_something()

finally:

enable_security_feature()

Signing scripts

Script signing is a way of ensuring that a script comes from a trusted source, and that the script has not been modified since being signed. Certificates based upon public key cryptography are issued with the script. The certificate is checked against the certificates kept in the system

It is possible to produce your own certificates for in-house use to prevent possibly malicious or untested scripts from damaging your systems.

Summary

When developing software, knowing the basics of a programming language is not enough. When working on projects of any significance, you need to follow a host of rules and protocols, including conforming to the development process, the development environment, the coding standards, the testing standards, and so on.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download