Neural Network Applications Development



Neural Network Applications Development

Neural Network Implementations

Neural Network Development Methodology

- The Concept Phase

- The Design Phase

- Implementation

- Maintenance

Neural Network Implementations

Three possible practical implementations:

1. A software simulation program running on a digital computer.

2. A hardware emulator connected to a host computer - the so-called neurocomputer.

3. A true electronic or optical neural network.

Software Simulations

Cheapest and fastest implementation method.

Replicates temporal behaviour of a network by numerically solving the system's equations for successive instants of time.

Written in most hi-level languages, C most popular. Trend toward object-oriented languages.

Drawbacks:

Writing simulators can be time consuming.

Solution - prewritten shells (commercial neural net packages) with convenient user interfaces. User must specify type of architecture and interconnections, number of layers and nodes.

Simulations are slow. Time and resource requirements become unacceptable beyond a few hundred nodes.

Inherent uncertainty of modelling - some combinations of parameters might produce nonsense. Too sensitive to initial conditions.

Inherent error of numerical computation, eg, truncation errors, approximation of differential equations with difference equations.

Neurocomputers

A dedicated special-purpose digital computer optimised to perform operations common in neural network simulation.

Acts as a coprocessor to a host computer and controlled by program run on the host.

Tens to thousands of times faster than simulators. Systems available with over 10 million IPS (Interconnect updates Per Second). Can emulate neural networks with several million neurones. Speed still limited by the need to maintain arrays in memory with thousands or millions of elements.

True Networks in Hardware

Neurons fabricated on silicon chips. Commercially available. Limitation of a few thousand neurons per chip. Chips connected in parallel to achieve larger networks. Problems with interconnection and interference. Simple synapses are resistors which represent fixed-valued weights. Work progressing on modifiable synapses.

Hybrid electro-optical or fully optical neural networks are currently undergoing research and development and are expected to be available by the turn of the century.

Neural Network Development Methodology

Aims to add structure and organisation to neural network applications development to reduce cost, increase accuracy, consistency, user confidence and friendliness.

The Concept Phase

Involves

1. validating proposed application, and

2. selecting a neural paradigm.

Some problem characteristics suitable for neural network application:

Data intensive

Multiple interacting parameters

Incomplete, erroneous data

solution function unknown or expensive

requires flexibility, generalisation, fault-tolerance, speed

Examples: Character recognition, forecasting, signal processing,

Problems not suitable:

Mathematically accurate and precise applications,

solution involves deduction and step-wise logic,

involves reporting.

Examples: resource management eg, inventory

accounts, sales data analysis

Selecting a neural paradigm

Based on comparison of application requirements to neural-paradigm capabilities. Issues to be considered:

Neural network size. Problem should be scaled to a size suitable for neural computing capabilities. Statistical neural network paradigms such as Boltzmann work best for smaller networks (around a few hundred) compared with nonstatistical paradigms such as backprop.

Required output type. Four common interpretations of output: classification, pattern, real number, and optimisation.

Associative memory classification.

Autoassociative, eg, character recognition, signal reconstruction.

Heteroassociative, eg, process monitoring, financial trend analysis.

Training Method.

Supervised (must have correct output),

Unsupervised (less data-requirement and training time)

Time Constraints. Training time (significantly longer) and Execution time.

After the proposed application is validated and the neural network paradigm selected, its technical and economic feasibility assessment should be carried out to complete the concept phase.

The Design Phase

Specifies initial values and conditions at the node, network and training levels.

The node level. Decisions to be made:

Types of input -

binary, bipolar, discrete, continuous.

Transfer function -

sigmoid (continuous inputs),

hard limiting (binary, bipolar inputs)

lookup tables used for speeding up calculations

The network level. Number, size and connectivity of layers.

Number:

One or two layers - Hopfield, Kohonen, ART,

One or more hidden layers - Backprop, Neocognitron

Size (number of nodes in) of layers:

Input layer - determined by number of data sources (input vector components)

Hidden layer - needs experimentation. Too few - incorrect mapping, too many - inadequate generalisation.

Output layer - determined by number of classes or objects to be output.

Connectivity

Feedforward (no feedback, no intralayer connections, eg, backprop nets)

Feedback (eg, ART)

Intralayer connections ( eg, Hopfield - fully connected, Kohonen)

Other design questions to be addressed:

Setting of parameters for learning phase

Possible addition of noise to speed up convergence in gradient descent

Possible combination of different neural network paradigms to meet application objectives better.

References

1. Caudill & Butler, Naturally Intelligent Systems, MIT Press 1989, pp 227-240.

2. Bailey, D., & Thompson, D., How to Develop Neural- Network Applications, AI Expert, June 1990, pp. 38-47.

The Implementation phase

Objective:

A functional neural network based on the most accurate, consistent and robust model possible.

Gathering the training set

Four aspects -

Type of data to collect :

depends on learning algorithm, type of input and output nodes, and nature of the problem. Data should be representative including routine, unusual and boundary-condition cases; non-ambiguous and not too erroneous.

Amount of data to gather:

Increasing data amount increases training time but may help earlier convergence. May be limited by time and other resources availability. Quality more important than quantity.

Collection of data:

May be tedious if multiple data sources exist. Potential sources include - historical records, instrument readings, simulation results, test data.

Preparation of data:

May involve - preprocessing, normalising, converting into binary vector.

Selecting the developing environment

Two aspects - Hardware and software.

Hardware: Requirements based on speed of operation, memory and storage capacity, software availability, cost, compatibility etc.

Most popular platforms - workstations and PC's (with accelerator board option); mainframes, parallel processors also used.

Software: Two options -

Custom-coded neural paradigms - maximum flexibility

Commercial development packages - easy to use, selection based on :

neural paradigms implemented

user programmable paradigms

speed in training and recall

transportability

vendor support

extensibility

price

Implementing the neural network

Common steps involved:

1. Select appropriate neural paradigm

2. Set network size

3. Determine the learning algorithm

4. Create any screen displays

5. Load the training set

6. Initialise the network

7. Determine the halting criteria

8. Begin training the network

9. Monitor the training process.

Possible problems during training - stabilising in a local minima, overtraining. May require resetting the learning parameters and reinitialising, if addition of small amount of noise does not help.

Training results may be affected by the method of presenting data set to the network. Some commercial packages use a random rather than sequential order; others buffer weight updates from individual data sets before combining them.

To ensure optimal results, several variations of a neural network may be trained and each tested for accuracy. Adjustments may be made by varying the layers sizes and fine tuning the learning parameters

Testing and debugging the network

Testing can be done by:

1. Observing operational behaviour of the net.

2. Analysing actual weights

3. Study of network behaviour under specific conditions

1. Testing usually done by treating the network as a black box and evaluating its response to a series of test cases.

Thresholds for acceptance of network responses must be predetermined.

Test data should contain training cases as well as new cases.

Routine, difficult as well as boundary condition cases should be tried.

2. The weights entering and exiting nodes are analysed. Any nodes without significant connections (>0.1) is a candidate for deletion. Similarly, any significantly large weight might indicate a problem.

3. Specific nodes may be strongly activated and its function in network behaviour observed (is the input output relationship reasonable?).

In case of significant errors detected in testing, debugging would involve examining

the training cases (representativeness, accuracy, number);

learning algorithm constants;

neural network architecture, node characteristics, and connectivity;

training set-network interface, user-network interface

The Maintenance Phase

Consists of placing the neural network in an operational environment with possible integration, periodic performance evaluation, and maintenance.

Operational environment

[pic]

Although often designed as stand-alone systems, some neural network systems are integrated with other information systems using loose-coupling, tight-coupling or full integration.

In loosely-coupled systems, the neural network is a preprocessor, a postprocessor or a distributed component communicating by passing data files.

In a tightly-coupled system, the neural network is fully integrated and works in close cooperation with the rest of the system for problem solving, eg, neural networks embedded in expert systems.

Neuro-symbolic computing utilises the complementary nature of computing in neural networks (numerical) and expert systems (symbolic).

Neuro-fuzzy systems combine neural networks with fuzzy logic. Another type of hybrid intelligent systems combine ANNs with genetic algorithm methodology (See "Hybrid Intelligent Systems" by L. Medsker, Kluwe Academic Press, Boston 1995)

System evaluation

Continual evaluation is necessary to ensure satisfactory performance in solving dynamic problems, and to check damaged or retrained networks.

Autonomous systems like the ART should be monitored against biasing with rare situations.

Evaluation can be carried out by reusing original test procedures with current data.

Maintenance

Involves modification due to decreasing accuracy or enhancements.

Two categories:

Data modification

- carried out in two steps. First, the training data must be modified or replaced; second, the network retrained and re-evaluated.

Software changes

- include changes in interfaces, cooperating programs or in the structure of the network. If the network is changed, part of the design and most of the implementation phase may have to be repeated.

Backup copies should be used for maintenance and research.

Artificial Neural Networks - Looking Ahead

The promise of neural nets:

Ability to solve problems resistant to algorithmic solutions on digital computers, eg, pattern recognition, optimisation.

Potential as associative memory with ability to handle incomplete, erroneous data

Capability to generalise

Speed of operation due to inherent parallelism

Fault-tolerance due to distributed nature

An avenue for extending understanding of biological systems

Widespread application held back by limitations of present day hardware technology.

Major breakthroughs in fabrication technology, optical and opto-electronic technology expected within a decade(?).

Biological (natural) neural network systems using genetic engineering by the middle(?) of the century.

Artificial neural nets - a substitute for conventional technology?

- Unlikely to happen. More likely to appear in hybrid intelligent systems complementing expert systems, fuzzy logic and genetic algorithmic computing.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download