Neural Network Applications Development
Neural Network Applications Development
Neural Network Implementations
Neural Network Development Methodology
- The Concept Phase
- The Design Phase
- Implementation
- Maintenance
Neural Network Implementations
Three possible practical implementations:
1. A software simulation program running on a digital computer.
2. A hardware emulator connected to a host computer - the so-called neurocomputer.
3. A true electronic or optical neural network.
Software Simulations
Cheapest and fastest implementation method.
Replicates temporal behaviour of a network by numerically solving the system's equations for successive instants of time.
Written in most hi-level languages, C most popular. Trend toward object-oriented languages.
Drawbacks:
Writing simulators can be time consuming.
Solution - prewritten shells (commercial neural net packages) with convenient user interfaces. User must specify type of architecture and interconnections, number of layers and nodes.
Simulations are slow. Time and resource requirements become unacceptable beyond a few hundred nodes.
Inherent uncertainty of modelling - some combinations of parameters might produce nonsense. Too sensitive to initial conditions.
Inherent error of numerical computation, eg, truncation errors, approximation of differential equations with difference equations.
Neurocomputers
A dedicated special-purpose digital computer optimised to perform operations common in neural network simulation.
Acts as a coprocessor to a host computer and controlled by program run on the host.
Tens to thousands of times faster than simulators. Systems available with over 10 million IPS (Interconnect updates Per Second). Can emulate neural networks with several million neurones. Speed still limited by the need to maintain arrays in memory with thousands or millions of elements.
True Networks in Hardware
Neurons fabricated on silicon chips. Commercially available. Limitation of a few thousand neurons per chip. Chips connected in parallel to achieve larger networks. Problems with interconnection and interference. Simple synapses are resistors which represent fixed-valued weights. Work progressing on modifiable synapses.
Hybrid electro-optical or fully optical neural networks are currently undergoing research and development and are expected to be available by the turn of the century.
Neural Network Development Methodology
Aims to add structure and organisation to neural network applications development to reduce cost, increase accuracy, consistency, user confidence and friendliness.
The Concept Phase
Involves
1. validating proposed application, and
2. selecting a neural paradigm.
Some problem characteristics suitable for neural network application:
Data intensive
Multiple interacting parameters
Incomplete, erroneous data
solution function unknown or expensive
requires flexibility, generalisation, fault-tolerance, speed
Examples: Character recognition, forecasting, signal processing,
Problems not suitable:
Mathematically accurate and precise applications,
solution involves deduction and step-wise logic,
involves reporting.
Examples: resource management eg, inventory
accounts, sales data analysis
Selecting a neural paradigm
Based on comparison of application requirements to neural-paradigm capabilities. Issues to be considered:
Neural network size. Problem should be scaled to a size suitable for neural computing capabilities. Statistical neural network paradigms such as Boltzmann work best for smaller networks (around a few hundred) compared with nonstatistical paradigms such as backprop.
Required output type. Four common interpretations of output: classification, pattern, real number, and optimisation.
Associative memory classification.
Autoassociative, eg, character recognition, signal reconstruction.
Heteroassociative, eg, process monitoring, financial trend analysis.
Training Method.
Supervised (must have correct output),
Unsupervised (less data-requirement and training time)
Time Constraints. Training time (significantly longer) and Execution time.
After the proposed application is validated and the neural network paradigm selected, its technical and economic feasibility assessment should be carried out to complete the concept phase.
The Design Phase
Specifies initial values and conditions at the node, network and training levels.
The node level. Decisions to be made:
Types of input -
binary, bipolar, discrete, continuous.
Transfer function -
sigmoid (continuous inputs),
hard limiting (binary, bipolar inputs)
lookup tables used for speeding up calculations
The network level. Number, size and connectivity of layers.
Number:
One or two layers - Hopfield, Kohonen, ART,
One or more hidden layers - Backprop, Neocognitron
Size (number of nodes in) of layers:
Input layer - determined by number of data sources (input vector components)
Hidden layer - needs experimentation. Too few - incorrect mapping, too many - inadequate generalisation.
Output layer - determined by number of classes or objects to be output.
Connectivity
Feedforward (no feedback, no intralayer connections, eg, backprop nets)
Feedback (eg, ART)
Intralayer connections ( eg, Hopfield - fully connected, Kohonen)
Other design questions to be addressed:
Setting of parameters for learning phase
Possible addition of noise to speed up convergence in gradient descent
Possible combination of different neural network paradigms to meet application objectives better.
References
1. Caudill & Butler, Naturally Intelligent Systems, MIT Press 1989, pp 227-240.
2. Bailey, D., & Thompson, D., How to Develop Neural- Network Applications, AI Expert, June 1990, pp. 38-47.
The Implementation phase
Objective:
A functional neural network based on the most accurate, consistent and robust model possible.
Gathering the training set
Four aspects -
Type of data to collect :
depends on learning algorithm, type of input and output nodes, and nature of the problem. Data should be representative including routine, unusual and boundary-condition cases; non-ambiguous and not too erroneous.
Amount of data to gather:
Increasing data amount increases training time but may help earlier convergence. May be limited by time and other resources availability. Quality more important than quantity.
Collection of data:
May be tedious if multiple data sources exist. Potential sources include - historical records, instrument readings, simulation results, test data.
Preparation of data:
May involve - preprocessing, normalising, converting into binary vector.
Selecting the developing environment
Two aspects - Hardware and software.
Hardware: Requirements based on speed of operation, memory and storage capacity, software availability, cost, compatibility etc.
Most popular platforms - workstations and PC's (with accelerator board option); mainframes, parallel processors also used.
Software: Two options -
Custom-coded neural paradigms - maximum flexibility
Commercial development packages - easy to use, selection based on :
neural paradigms implemented
user programmable paradigms
speed in training and recall
transportability
vendor support
extensibility
price
Implementing the neural network
Common steps involved:
1. Select appropriate neural paradigm
2. Set network size
3. Determine the learning algorithm
4. Create any screen displays
5. Load the training set
6. Initialise the network
7. Determine the halting criteria
8. Begin training the network
9. Monitor the training process.
Possible problems during training - stabilising in a local minima, overtraining. May require resetting the learning parameters and reinitialising, if addition of small amount of noise does not help.
Training results may be affected by the method of presenting data set to the network. Some commercial packages use a random rather than sequential order; others buffer weight updates from individual data sets before combining them.
To ensure optimal results, several variations of a neural network may be trained and each tested for accuracy. Adjustments may be made by varying the layers sizes and fine tuning the learning parameters
Testing and debugging the network
Testing can be done by:
1. Observing operational behaviour of the net.
2. Analysing actual weights
3. Study of network behaviour under specific conditions
1. Testing usually done by treating the network as a black box and evaluating its response to a series of test cases.
Thresholds for acceptance of network responses must be predetermined.
Test data should contain training cases as well as new cases.
Routine, difficult as well as boundary condition cases should be tried.
2. The weights entering and exiting nodes are analysed. Any nodes without significant connections (>0.1) is a candidate for deletion. Similarly, any significantly large weight might indicate a problem.
3. Specific nodes may be strongly activated and its function in network behaviour observed (is the input output relationship reasonable?).
In case of significant errors detected in testing, debugging would involve examining
the training cases (representativeness, accuracy, number);
learning algorithm constants;
neural network architecture, node characteristics, and connectivity;
training set-network interface, user-network interface
The Maintenance Phase
Consists of placing the neural network in an operational environment with possible integration, periodic performance evaluation, and maintenance.
Operational environment
[pic]
Although often designed as stand-alone systems, some neural network systems are integrated with other information systems using loose-coupling, tight-coupling or full integration.
In loosely-coupled systems, the neural network is a preprocessor, a postprocessor or a distributed component communicating by passing data files.
In a tightly-coupled system, the neural network is fully integrated and works in close cooperation with the rest of the system for problem solving, eg, neural networks embedded in expert systems.
Neuro-symbolic computing utilises the complementary nature of computing in neural networks (numerical) and expert systems (symbolic).
Neuro-fuzzy systems combine neural networks with fuzzy logic. Another type of hybrid intelligent systems combine ANNs with genetic algorithm methodology (See "Hybrid Intelligent Systems" by L. Medsker, Kluwe Academic Press, Boston 1995)
System evaluation
Continual evaluation is necessary to ensure satisfactory performance in solving dynamic problems, and to check damaged or retrained networks.
Autonomous systems like the ART should be monitored against biasing with rare situations.
Evaluation can be carried out by reusing original test procedures with current data.
Maintenance
Involves modification due to decreasing accuracy or enhancements.
Two categories:
Data modification
- carried out in two steps. First, the training data must be modified or replaced; second, the network retrained and re-evaluated.
Software changes
- include changes in interfaces, cooperating programs or in the structure of the network. If the network is changed, part of the design and most of the implementation phase may have to be repeated.
Backup copies should be used for maintenance and research.
Artificial Neural Networks - Looking Ahead
The promise of neural nets:
Ability to solve problems resistant to algorithmic solutions on digital computers, eg, pattern recognition, optimisation.
Potential as associative memory with ability to handle incomplete, erroneous data
Capability to generalise
Speed of operation due to inherent parallelism
Fault-tolerance due to distributed nature
An avenue for extending understanding of biological systems
Widespread application held back by limitations of present day hardware technology.
Major breakthroughs in fabrication technology, optical and opto-electronic technology expected within a decade(?).
Biological (natural) neural network systems using genetic engineering by the middle(?) of the century.
Artificial neural nets - a substitute for conventional technology?
- Unlikely to happen. More likely to appear in hybrid intelligent systems complementing expert systems, fuzzy logic and genetic algorithmic computing.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- neural network online learning
- online neural network trainer
- neural network problems
- neural network in machine learning
- neural network vs ai
- neural network structure
- neural network examples
- neural network vs machine learning
- artificial neural network introduction
- neural network vs artificial intelligence
- types of neural network algorithms
- artificial neural network application