ViewIT 2



Discoteq 2.0

Advisors: Patrik Sundin and Krister Dackland

Examiner: Per Lindström

Abstract

This document describes my master's thesis work named Discoteq 2.0. The thesis was performed at AerotechTelub in Växjö during the fall of 2002.

Discoteq 2.0 is a distributed environment for development of applications handling configuration data and testing equipment, mainly for the aerospace industry. The name “Discoteq” comes from the quick access to function schemas and configuration information that is possible from several types of clients.

Discoteq 2.0 builds on an earlier version of a data access platform named Discoteq 1.x. This platform has grown to a point where it has become hard to manage the different configurations and versions of it. Discoteq 2.0 tries to solve this problem by using a new type of versioning concept, and by being highly extendable and flexible.

The thesis is focused on how the use of new technologies can create a manageable distributed system that is also easy to learn; a software developer with no specific knowledge of Discoteq subsystems should be able to quickly fix problems and make changes in the functionality of data access patterns.

Index

1 Introduction 7

1.1 AerotechTelub 7

1.2 Task 7

1.3 Definitions 7

2 Background 9

2.1 What is Discoteq? 9

2.2 Discoteq 1.x 10

2.3 .NET 11

Visual 12

C#, the CLI and the CLR 12

.NET Framework 14

.NET Remoting 14

3 Implementation and Design 17

3.1 Overview 17

3.2 Roles 17

3.3 Discoteq 2.0 18

Discoteq Types 18

Discoteq Server 19

Discoteq Compiler 20

Discoteq Application 21

3.4 Code Generation 21

Plugins 21

Example VFD 22

Example Code (Application) 22

Step-by-Step Description 23

Example Code (Wrapper) 24

3.5 XML Help Files 25

3.6 Discoteq Expressions 26

Type Mappings 26

Simple Expressions 26

Complex Expressions 27

3.7 GUI (Discoteq Logic Assembler) 28

4 Discussion 29

4.1 Platform 29

4.2 Programming Language 29

4.3 Database System 29

4.4 Transport Model 29

4.5 The extra compilation step and XML 30

Why XML? 31

Why generate C# code? 31

4.6 Distributed Approach 31

4.7 Thin vs. Rich Clients 32

4.8 Automatic Compilation 32

4.9 Versioning 32

4.10 Security and Compression 33

4.11 Developer roles 34

4.12 Use Cases 34

Discoteq Application Design 34

Discoteq Client Design 35

Discoteq Plugin Design 36

4.13 AECMA 1000D Integration 36

Data Modules 36

Document Format 36

Integration 37

5 Conclusion 39

References 40

List of Figures

Figure 1: A Discoteq system 10

Figure 2: The Discoteq 1.x system 11

Figure 3: .NET overview 12

Figure 4: Simple overview of Discoteq 2.0 17

Figure 6: The Discoteq Server 19

Figure 7: The Discoteq compilation process 20

Figure 8: Discoteq Application block diagram 21

Figure 9: Discoteq Logic Assembler 28

Figure 10: ICodeStore 31

Figure 11: Discoteq developer role hierarchy 34

Figure 12: Theoretical DMC-based filesystem tree 37

List of Tables

Table 1: Comparison between C#, C++ and Java 13

Table 2: Discoteq roles 18

Table 3: ILinkedTable 19

Table 4: Examples of type mappings 26

Table 5: Discoteq versioning example 33

Table 6: Method versions 33

Table 7: Application and method versions 33

Introduction

Discoteq 2.0 is a distributed environment for development of applications handling configuration data and testing equipment. The name “Discoteq” stands for "distributed environment for configuration data and testing equipment".

1 AerotechTelub

AerotechTelub is a service company in the Saab Group, with first-hand knowledge of what it means to be a third party supplier. The company specializes in aviation, command, information, communication and sensors.

The range of the company's services covers:

• Technical services principally consultancy, integration, information, maintenance and testing services

• Operation and maintenance

• Systems suppliers principally for test/measuring, simulation, traffic control, telecommunications contracts and customer adapted information systems.

2 Task

At 2002-12-31, AerotechTelub shall have a version 2.0 of the Discoteq system ready for use. The development shall be made through a master's thesis covering 20 weeks.

By using knowledge of the current Discoteq system, current Discoteq design patterns and the .NET platform, Discoteq 2.0 shall be able to be used in future applications and target environments in a cost-efficient way.

The purpose of the development of Discoteq 2.0 is to make use of existing knowledge of data transportation and implement these in a modern development environment such as .NET. Further, the aim is to break the parallel development that Discoteq development means to the SOAP protocol.

Discoteq 2.0 shall make use of web services and the SOAP protocol. In addition, an editor for XML templates supporting both the current Discoteq protocol and the Discoteq 2.0 protocol should be developed.

As a part of the Discoteq 2.0 development process, an examination of possibilities to use the following technologies and concepts with Discoteq 2.0 should be made:

• MSMQ as transfer protocol

• Data compression

• Security aspects

• Performance aspects

• ECMA 1000D support

3 Definitions

In this document the following definitions will be used:

• Discoteq 2.0 is the system described by this thesis

• Discoteq 1.x is the predecessor to Discoteq 2.0

• Discoteq (without any version number) refers to Discoteq 2.0 and any systems based on, or extending, Discoteq 2.0 - except when used in an obvious context describing Discoteq 1.x.

• A VFD or Discoteq Function Description is an XML document that controls the behavior of (parts of) a Discoteq application.

• A Collected VFD is a VFD that contains the merged XML of all VFD:s belonging to a certain Discoteq Application.

Background

The following sections strive to familiarize the reader with the technologies used in this thesis.

1 What is Discoteq?

The Discoteq platform originates from the feasibility study on ELDIS39, a system that shows electrical schemas and configuration data for the military aircraft JAS39 Gripen. The study was made during the year 2000, but it was not until the spring of 2001 that the Discoteq platform was separated from the ELDIS39 product, and got its current name.

The Discoteq philosophy can be summarized in three major points:

• Make software development more effective.

• Facilitate easy maintenance, thus reducing maintenance costs.

• Allow a single client to act upon several different kinds of services, such as databases and specific hardware.

The Discoteq system is a typical Client-Server system, providing quick data access from several data sources. What differentiates Discoteq from other systems is that the end-clients are often graphical interfaces that show several layers of related data. This way of presenting data often means simultaneous access to several types of data sources, be it databases, hardware systems or filesystems.

End-user applications built on the Discoteq system are used in administrative applications, especially in areas such as military equipment maintenance. A typical Discoteq system could look like this:

[pic]

Figure 1: A Discoteq system

The central point in the system is one central Discoteq server that distributes requests to other components, such as databases, yet another Discoteq server or hardware. Internally there are one or more administration clients which administer the overall functionality of the Discoteq server and the specific functionality of a Discoteq application. Externally, there are rich clients with Windows systems that can be connected to the Discoteq server directly, or thin clients that connect through a web server which is in itself a Discoteq client.

2 Discoteq 1.x

Discoteq 1.x is a system written mostly in Microsoft Visual BASIC and is utilizing COM and DCOM components to form a distributed system for data storage and retrieval. It was written to be the platform for the ELDIS[1] system, which displays hyperlinked electrical wire schemas and equipment information.

In brief, Discoteq 1.x consists of a client-side COM object, a “hub” and several DCOM server objects. The client issues an initial request or a hyperlink action, which the hub then distributes to affected servers. When processing is done, the hub collects the server responses and the collected document is sent back to the client. The logic in the hub is simple; in most cases it is limited to searching and replacing within client requests and merging server responses.

[pic]

Figure 2: The Discoteq 1.x system

Discoteq 1.x communication is based on an XML protocol, which has been extended a number of times to cope with new data sources and objects; the result is a hotchpotch of public standards and internal specifications. This is why the idea of a new Discoteq came to mind - something that would be extensible yet easy to use and maintain.

3 .NET

As the requirements of the Windows programming models were increasing, Microsoft kept on extending them to suit the customers’ needs. The result was very powerful but also extremely complicated models such as COM, DCOM and COM+. The problem with these models is that things were severely getting out of hand with a complexity so high, it was matched by a learning period of about six months [[i]] for an experienced developer. This is why Microsoft started from the beginning, building up both a new programming model and a new platform[2] centered on web services and the Internet; the result was .NET (pronounced “dot-net”).

In February 2002 [[ii]] Microsoft launched Visual , and the product was well received by the developer community. The general consensus is that Microsoft has listened to the users and has created a very useful, powerful and easy-to-learn programming model while retaining most of the compatibility for earlier models.

The .NET platform is also seen as a way for Microsoft to battle the growing Java and Linux presence on the Internet; at the end of year 2002, less than 28 percent [[iii]] of all web servers on the Internet were powered by Microsoft solutions.

[pic]

Figure 3: .NET overview

Some integral parts of .NET are:

• Visual

• The .NET Framework

• The .NET programming languages; C#, , managed C++ and J#

• The Common Language Runtime (CLR)

• .NET Remoting and web services

Visual

True to their concept, Microsoft always releases a full IDE (Integrated Development Environment) with every major release of their compilers. Guides makes it very easy to start with development, and a powerful debugger makes debugging an easy and intuitive task. In addition to this, the IDE sports a context-sensitive, color-coding editor and quick help access to the whole MSDN (Microsoft Developer Network) library.

C#, the CLI and the CLR

C# (pronounced c-sharp) and the CLI (Common Language Infrastructure) are specifications developed by Hewlett-Packard, Intel, Microsoft and several ECMA members [[iv]]. C# is a new language tailored for .NET, while the CLI describes a stack-based intermediate language. All .NET-managed languages (C#, J#, and Managed C++) compiles to this intermediate language, which has been developed previously and in parallel to C#. Both C# and the CLI are public ECMA standards, free to implement by anybody.

C# was developed with .NET in mind, semantically resembling Java more than any C dialect. The language provides some very powerful concepts, such as code constructs for properties and indexers, metadata attributes and operators. However, some good features from other languages are missing, such as internal classes (from Java) and constant parameters (from C++). The language was submitted to ECMA in August 2000 and the first edition of the public standard was ratified in December 2001.

A quick comparison of C#, C++ and Java:

| |C++ |C# |Java |

|Executable |Native |Runs in a VM |Runs in a VM |

|Inheritance |Multiple |Single with interfaces |Single with interfaces |

|Memory allocation |Manual |Automatic or Manual |Automatic |

|Class grouping |Flat (header files) |Hierarchical (namespaces) |Hierarchical (packages) |

|Synchronization |With external APIs |Internal (lock) |Internal (synchronized) |

|Runtime reflection |Low (only dynamic type casts)|Very high (dynamic discovery |High (dynamic discovery) |

| | |and dynamic compilation) | |

|Global code |Global definitions, |Global definitions and static|Static fields in classes. |

| |constants, variables and |fields in classes. | |

| |functions. | | |

|Primitive type usage |High |Medium (objects are |Low |

| | |automatically converted to | |

| | |primitive types and vice | |

| | |versa) | |

|Nested classes |No nesting |Only static class nesting |Full class nesting; even has |

| | | |anonymous internal classes. |

|Operator overloading |Almost all operators are |Some operators are |No support for operator |

| |redefinable. |redefinable. Provides special|overloading. |

| | |access for indexers. | |

|Properties |N/A |Built-in constructs. |By design patterns. |

|Metadata |Static and sometimes |Special language constructs. |N/A |

| |compiler-dependent |Fully customizable. | |

| |directives. | | |

Table 1: Comparison between C#, C++ and Java

The CLI (Common Language Infrastructure) defines the Intermediate Language (IL), which behaves much like an assembly language. It also specifies the rules on how to generate IL code from high-level languages.

The CLR is a closed-source virtual machine that compiles and executes IL code, much resembling the JVM [[v]]. Although the CLI is public, Microsoft didn’t want to make the CLR public also, mainly because of high-performance optimization algorithms [[vi]]. This is why Microsoft has released a shared-source[3] version of a CLI implementation, called Rotor [[vii]], containing both subsets and supersets of the CLR. Programming languages that compiles to IL and are managed by the .NET CLR are called managed or contains the text ".NET", as in managed C++ and .

One difference between the JVM and the CLR/IL is that the latter is focused on cross-language rather than cross-platform; this means that old applications can be migrated to .NET in an easy way, and use components written in any other managed language. For example, old COBOL business systems could be migrated to managed COBOL [[viii]] and use components from the .NET class library, such as GUI components.

While the JVM supports fully interpreted execution and mixed-mode execution, the CLR does not support interpreted mode execution at all, instead all IL code is compiled to native code before execution. Even though the IL is stack-based just like the JVM, the former seems to have an easier task of optimizing the code to form efficient native code. Microsoft even claims that in some ways IL is a better thing than native code, because the final code of a managed application can be very tightly optimized for the architecture it will run on during the IL to native compilation.

.NET Framework

The .NET Framework is much like the Java API; it is a class library that contains frequently used classes, such as classes for collections, file I/O and GUI components. The .NET Framework is of course compiled to IL to facilitate language interoperability.

There is a drawback to the .NET Framework model though; the source code is not readily available so deeper understanding can be hard to attain. The Java SDK download includes the class library source code [[ix]], but a license is forbidding redistribution of the package. The Rotor project does not include the Framework, although there is an open-source initiative called Mono [[x]], which aims to port most of .NET to several platforms, including Linux.

Microsoft has also chosen to divide the framework in more or less platform-dependent parts. is a very Windows-centric database API, as is the GUI API; with a name such as Windows Forms there can be no doubt what operating system this API is targeted for.

.NET Remoting

.NET Remoting is the .NET replacement of DCOM and COM+ that is written especially with the .NET Framework in mind. It is surprisingly easy to use, but lacks some basic functionality such as encryption and authenticating.

The functionality, features and performance are comparable to CORBA[4] and RMI[5], but the ease of implementation is outstanding. With only a few lines of code, a class or an object can be remotely published. It’s possible to use HTTP or a binary protocol for transfers, and the objects can be serialized to SOAP format or .NET binary format.

By enabling the user to customize each step of the remoting and serialization process Microsoft has taken a huge step away from the proprietary formats of DCOM and COM+. For example, it is fairly easy to implement compression and encryption of the data stream, whatever serialization format or transport protocol is used. There are also many ways to marshal[6] objects; the simplest way is to create a .NET Remoting service inside a Windows Console application, then marshal the object type as a singleton[7] object. It is also possible to run an object inside Microsoft IIS[8], or run it as a standalone Windows NT service.

.NET Remoting is seen by many as the real power of .NET, even if the big hype is on .NET web services. Web services are simply too limited, having only singleton/single-call functionality.

Implementation and Design

The following sections describe the Discoteq 2.0 implementation on a high to medium level, and highlight some major points in its design.

1 Overview

The Discoteq 2.0 system is based on a .NET platform using .NET Remoting to communicate. .NET Remoting provides several ways to access remote objects, and this particular implementation allows the use of HTTP/SOAP and the TCP/binary protocol. A service object receives requests to create new remote objects, which are automatically generated from XML description files (VFD:s).

[pic]

Figure 4: Simple overview of Discoteq 2.0

The software used in the development is the Visual Studio .NET environment, including the .NET CLR and framework. The underlying operating system for development and testing has been Microsoft Windows 2000 Professional. To install and run the software, the only thing that is needed is a platform capable of fully supporting the Microsoft .NET framework.

2 Roles

To easier be able to describe the full Discoteq process, the human roles in the Discoteq system has to be explained. The roles are the following:

|Discoteq user |The end-user of a Discoteq client. |

|Discoteq client developer |Develops clients for Discoteq. |

|Discoteq plugin developer |Develops plugins for the Discoteq language. |

|Discoteq application developer |Writes VFD:s for Discoteq applications. |

|Discoteq developer |Can handle anything from the three former categories. |

|Discoteq administrator |Administers the Discoteq system; manages Discoteq windows services and webpages, |

| |chooses where to install the Discoteq system, notifies clients of upgrades and so on.|

Table 2: Discoteq roles

3 Discoteq 2.0

Discoteq 2.0 has three basic blocks of functionality, the Discoteq Server, the Discoteq Compiler and the automatically generated Discoteq Applications. Additionally, it has a type library that is used in all these three blocks.

Discoteq Types

The Discoteq type library was created to make a smoother transition from Discoteq 1.x, and to reduce the amount of code that has to be written both in XML templates and in clients. As the Discoteq types are integral parts of all Discoteq modules and should therefore not be changed very often, much research was put on making the types simplistic yet powerful. An image class and a real-time streaming class had to be cut from the project due to lack of time. The following sections describe the public Discoteq classes and interfaces.

IParameters, VParameters

The IParameters interface describes a list of objects where each object is linked to an index and optionally a name. This interface makes it possible to send and return a variable amount of arguments to and from a Discoteq method. Although IParameters is immutable, the VParameters class, which implements IParameters, has additional methods to add parameters to the collection.

ITable, VTable

ITable is an immutable interface that describes a table of objects, with named and indexed columns. The VTable class implements ITable, and provides methods to add and modify cells, rows and columns. There are also advanced methods to convert standard data formats into an ITable.

IColumn, IRow

These are interfaces supporting row and column operations on an ITable. The implementations of these interfaces are internal to VTable. IRow extends the IParameters interface, and only adds an Index property, which can be used to get the rows numeric index in the table. IColumn provides column name and object type information.

ILinkedTable, VLinkedTable and ILinkedColumn

These specialized Discoteq classes enable the use of "links" for table cells. Link functionality is applied to columns, which means every cell in that column is linked to a specific remote Discoteq class and method. When a link is executed, the parameters for the method are taken from static values or from cells on the same row where the link was executed.

An example of the linking ability is the following table of car models, generated by the Discoteq call Show_CarModelsByRegionAndType("Europe","*").

|Model |Type |

|Ferarri F40 |Sport |

|Volvo 440 |Family |

Table 3: ILinkedTable

The natural thing would be to link the "Model" column to a more detailed specification of the car model, while the "Type" column could be linked to another lookup for any cars in the region having the specified type.

Thus the action for the "Model" column would be Show_CarModelDetails, and the parameter would be column="Model". The action for the "Type" column would be Show_CarModelsByRegionAndType and the parameters would be value="@Region"[9] and column="Type".

Discoteq Server

[pic]

Figure 5: The Discoteq Server

The Discoteq Server is comprised of the following components

• Discoteq Windows Service

The Windows Service component is a container for the Discoteq Service and Discoteq ServiceControl components. It presents the Discoteq developer with an easy way to install and execute the Discoteq service.

• Discoteq Service

The service component is the component responsible for the marshalling of the CodeStore and AppInstantiator and is responsible for the starting of the compiler loop (if not manual compilation is chosen).

• Discoteq ServiceControl (IServiceControl)

The ServiceControl component presents an interface to the Discoteq administrator who can use it through a webpage or a Windows application, it enables and administrator to start and stop the automatic compilation, and to manually compile a specific application.

• Discoteq Code Store (ICodeStore)

The CodeStore component stores references and allows access to the compiled assemblies, the generated code and the collected XML files of any Discoteq applications that are registered with the service.

• Discoteq Application Instantiator (IAppInstantiator)

This is the component that is called by the client applications, whether from within a Discoteq client assembly or directly from within an application.

Discoteq Compiler

[pic]

Figure 6: The Discoteq compilation process

The Discoteq compiler is only comprised of itself and any Discoteq Plugins that might be loaded. Internally there is a three-phase compilation process: XML document collection, XML to C# code generation and Assembly and DLL generation.

Discoteq Application

[pic]

Figure 7: Discoteq Application block diagram

A Discoteq Application holds information generated from a Discoteq XML Function Description. The following information is held within an application object:

Version-independent:

• The collected XML document that has not yet been split in version-dependent parts.

• The compiled version-independent assembly containing one single application class.

For each version:

• The XML document

• The compiled interface and client helper classes

• The compiled wrapper

4 Code Generation

This section contains an example on how C# code is generated from a VFD file, which means the reader is assumed to have some knowledge of C# and XML.

Plugins

To grant third part developers and ourselves with an easy way of extending Discoteq, the method of having plugins for code generation were conceived. Each element in a VFD has a corresponding plugin, which parses the element and converts it into C# code. Plugin classes can be grouped in .NET assemblies and loaded dynamically by the Discoteq Service.

Example VFD

i = i + 1;

Example Code (Application)

using System;

using System.Collections;

using System.Runtime.Remoting.Metadata;

using Discoteq;

using Discoteq.Types;

namespace Discoteq.DemoNamespace {

public class _DemoApplication : MarshalByRefObject {

private delegate object

_MethodCallDelegate(System.Reflection.MethodInfo mi,

object caller, object[] callobjs);

private object _MethodCall(System.Reflection.MethodInfo mi,

object caller, object[] callobjs) {

return mi.Invoke(caller,callobjs);

}

public object __caller;

public void Class_Method__V_1_0(System.Int32 i,

System.Collections.IDictionary __Meta) {

try {

try {

i = i + 1;

} catch(DiscoteqException __vex) {

throw __vex;

} catch(Exception __ex) {

throw new

DiscoteqException("/Method/code",__ex);

}

} catch(DiscoteqException __vexception) {

Console.Error.WriteLine(__vexception);

throw __vexception;

} catch(Exception __exception) {

Console.Error.WriteLine(__exception.Message);

Console.Error.WriteLine(__exception.StackTrace);

throw new Exception("Unhandled Remote Exception in Method: " +

__exception.Message + ": \n" + __exception.StackTrace,

__exception);

}

}

}

}

Step-by-Step Description

The first node that is parsed is the root element, which is required to contain an attribute named "application" that defines which Discoteq application the code belongs to.

...

The Discoteq compiler generates the following code for the above element (the notation "..." means irrelevant code or code that will be filled in later):

namespace Discoteq.DemoNamespace {

public class _DemoApplication : MarshalByRefObject {

...

This class is called the "application class" of the Discoteq application, and will contain all logic of the Discoteq application. The class name is not relevant at this stage, because a wrapper takes care of all calls to this class.

The next node to be parsed is the "Class" element. All elements at this level are Discoteq class definitions, except some few special elements.

...

The "param" element belongs to the method definition, and is parsed as a part of the method. In this case, it defines that the method has one integer (32-bit) parameter named "i". The "@" sign that defines a Discoteq variable is mandatory, but only to enhance readability as it is not possible to provide anything else than a variable name to the "param" element. Further, the method gets an extra parameter named "__Meta" which is a collection of session variables. The try-catch block is designed to forward most exceptions to the remote client.

The class name, method name and method version are then combined into a single method name. The client does not see the version number, as the adding of version numbers is another thing the wrapper takes care of.

public void Class_Method__V_1_0(System.Int32 i,

System.Collections.IDictionary __Meta) {

try {

...

} catch(DiscoteqException __vexception) {

Console.Error.WriteLine(__vexception);

throw __vexception;

} catch(Exception __exception) {

Console.Error.WriteLine(__exception.Message);

Console.Error.WriteLine(__exception.StackTrace);

throw new Exception("Unhandled Remote Exception in Method: " +

__exception.Message + ": \n" + __exception.StackTrace,

__exception);

}

}

The code element is chosen because it is a very simple element; basically it only adds the textual contents to the C# code stream.

i = i + 1;

Each element is surrounded with a try-catch statement that has the responsibility of converting any exceptions into a Discoteq exception. The Discoteq exception contains a reference to the source VFD file and the element that the code is generated from.

try {

i = i + 1;

} catch(DiscoteqException __vex) {

throw __vex;

} catch(Exception __ex) {

throw new

DiscoteqException("/Method/code",__ex);

}

Example Code (Wrapper)

This is the wrapper code. The wrapper implements a specific version of a Discoteq application interface, and translates all "nonversioned" method calls to "versioned" method calls.

namespace Discoteq {

public class DemoWrapper : MarshalByRefObject, DemoApplication {

private Discoteq.DemoNamespace._DemoApplication _app;

public DemoWrapper(IDictionary meta) {

_app = new DemoNamespace._DemoApplication();

_app.__caller = this;

__Meta = meta == null ? new Hashtable() : meta;

__ResultMeta = new Hashtable(__Meta);

}

private IDictionary __Meta;

private IDictionary __ResultMeta;

public IDictionary Meta {

set {

__Meta = value == null ? new Hashtable() : value;

}

get {

return __ResultMeta;

}

}

public void Class_Method(System.Int32 i) {

__ResultMeta.Clear();

foreach(object __key in __Meta.Keys)

__ResultMeta[__key] = __Meta[__key];

_app.Class_Method__V_1_0(i, __ResultMeta);

}

}

}

5 XML Help Files

Each tag handled by a plugin is required to have an XML help file. The help file format is not yet formally defined, although the layout is simple.

The following lines are the XML document header, and the "vilahelp" element with a mandatory version of "2.0".

The "tag" element tells which element name to describe, and on which level in the VFD the element exists.

The next element, "desc", contains the description of the VFD element.

Makes a method call

Each attribute to the VFD element is described by an "attr" element. The "name" attribute specifies the name of the VFD attribute, and the optional "enum" attribute presents a vertical-bar separated list of possible values. If the "q" attribute is set to "1", the attribute is mandatory (has to exist exactly once).

The contents of the "desc" element describes the attribute.

Specifies the type of this call

If the attribute has a default value, the "def" element is specified with the default value of the attribute.

local

Each value in the optional "enum" list should be described by a "vdesc" element. It contains the description for the value defined by the "value" attribute.

X is a value

Y is a value

Special child tags only applicable to this VFD element can be specified in additional "tag" elements. It follows the above specification, but has no "level" attribute.

Specifies method parameters

...

The "ex" element provides the help file writer to add usage samples.

...

6 Discoteq Expressions

Discoteq expressions were created because an easy system of being able to specify integer values, text values and variable references were needed. The problem is that XML files, which the Discoteq language is based on, needs all attribute values to be surrounded with double quotes, a method also used in various program languages to separate string values from function calls and numeric values.

Type Mappings

Type mappings makes the Discoteq expression syntax more readable. Types are mapped using a simple search and replace, where simple type names are mapped to basic .NET objects. In Table 4 some examples of mappings can be found.

|Simple type |.NET type |

|bool |System.Boolean |

|byte |System.Byte |

|char |System.Char |

|int |System.Int32 |

|uint |System.UInt32 |

|long |System.Int64 |

|float or |System.Single |

|single | |

|table |Discoteq.Types.ITable |

Table 4: Examples of type mappings

Simple Expressions

Having another set of quotes inside the XML attribute quotes would be cumbersome for a Discoteq developer, so the type of an expression is decided by the plugin, if the developer does not force the type by using a typecast operator. The different ways of writing a simple expression can be seen below.

"@variable"

Results in the value assigned to the variable with the name "variable".

"#meta"

Results in the value from the metavariable with the name "meta".

"$const"

Results in the value from the constant named "const".

"$enum.const"

Results in the value from the constant named "const" in the enumeration named "enum".

"value"

Results in the string "value".

"1.0"

Normally results in the string "1.0", but if the attribute is explicitly defined as a float value, it results in the float value of 1.0.

"(double)1"

Results in the double floating-point value 1.0. The typecast operator can also be prefixed to variables, meta variables and constants.

"\@variable"

Results in the string "@variable" as the backslash is treated as a string escape character.

"\\nvalue\n"

Results in the string "\nvalue\n" (containing two newlines).

Complex Expressions

Complex expressions are combinations of simple expressions and operators. All C# operators except [=] (assignment) can be used. The assignment operator is instead mapped to the equality operator, and the characters [] are mapped to the inequality operator.

To make complex expressions less ambiguous, all strings must be enclosed in single quotes [']. If single quotes have to be used within the string, a backslash [\] character must precede each single quote. The following are some examples of complex expressions:

"@sItem == 'chair'"

-> false

"String.Format('Time: {0},{1}',@iHours,@iMinutes)"

-> "Time: 12,34"

"@xData.SelectSingleNode('*[@attr=\'name\']')"

-> (XmlNode)

7 GUI (Discoteq Logic Assembler)

To simplify the Discoteq server development process further, a GUI (Graphical User Interface) was created. The behaviour depends on the Discoteq XML help files, which means that the GUI does not have to be updated when new plugins are created (except for the copying of XML files).

[pic]

Figure 8: Discoteq Logic Assembler

The GUI provides pop-up help for all elements and attributes that have definitions in the XML help files. Additionally it is possible to copy and paste textual XML data to and from the tree.

Discussion

Before beginning with the implementation, and during the project, research had to be made on what techniques to use, and what things were in scope of the thesis. This section discusses that research, and what was learned from it.

1 Platform

As the "Background" section tells, the .NET platform is a new and promising product from Microsoft. The reason to implement Discoteq 2.0 in .NET is twofold:

First, almost all of the development at AerotechTelub is today performed on Microsoft Windows NT-based platforms. Most developers are therefore much more acquainted with the Microsoft platforms than with any other platform.

Second, the company wants to test the viability of the .NET platform by using it for an internal project that is not business-critical.

2 Programming Language

With the platform set to be .NET, the choice of C# as the primary programming language for the project was easy: No other language in the .NET platform really supports all the features of .NET in a straightforward way. Later in the development cycle when it was decided to compile the XML code, the choice became even more natural as the easiest way to compile code from within .NET is to use C#.

3 Database System

Microsoft SQL Server 7.0 was chosen as the primary relational database to work with. It was chosen because it was readily available at the time of implementation, and because the test database is using Microsoft SQL-tailored stored procedures. The choice was not entirely without benefit for the development process either; a specialized SQL API in .NET (within ) provided good examples, which made development very fluid, compared to if other third-party solutions would have been used.

4 Transport Model

To start with, the thesis was specified to build upon web services and use SOAP [[xi]] as a primary transport protocol. After minor research, it was decided that the .NET implementation of web services did not have the functionality that was needed; the lack of state information (i.e. persistent objects) would make future versions of Discoteq hard to maintain. Instead, the implementation would be based on .NET Remoting, which has numerous advantages over .NET web services [[xii]].

Apart from the functionality limitations of the .NET web services, .NET Remoting also gives a much faster response time and a higher throughput, especially when using the binary protocol [[xiii]]. Both these factors could be crucial in later versions of Discoteq, if real-time measuring (fast response time) and very large images (high throughput) will be implemented.

.NET Remoting

• can use the binary protocol

• provides state management

• provides ways to marshal existing objects

• can be deployed in several ways

Even though .NET Remoting supports the highly efficient binary protocol, the specification requires the SOAP protocol to be used, mainly because of the compatibility it provides. The SOAP protocol is based on XML [[xiv]], which is a hierarchical text format. Although XML is a well-defined and flexible format, it's hardly the end-it-all of data formats.

The first problem with XML is that it is inefficient for large amounts of data: XML character data sections are enclosed in so called XML elements, and to find the boundaries of these elements, the whole XML file has to be filtered through an XML parser. This uses processing power and places a high load on system memory, as the whole file has to be in memory when parsing. Adding to this, each data section that will be used, most probably has to be copied to a new memory location for further processing.

The second problem with XML is its inability to contain binary data; all data must be textual. To overcome this limitation, binary data is encoded using Base64 encoding ([[xv]] section 6.8). Unfortunately, this method makes the data 33% bigger in size, and needs another working copy of the data in system memory.

To overcome these problems, a way of encoding SOAP messages into the MIME [15] format has been proposed [[xvi]]. Even though Microsoft is one of the companies behind this proposal, there is no support in current .NET Remoting for this type of messages. MIME allows the definition of separate data blocks that can optionally contain length information, making parsing inside data unnecessary. MIME also supports 8-bit binary encoding, which means no conversion to text has to be made.

Still, MIME encoding had to be tested, so a MIME API (complete with a Base64 encoder/decoder) was written in C#. However, the resulting application proved to be inefficient; quick tests showed severe slowdowns both when encoding and decoding MIME messages, so the usage of SOAP with MIME attachments had to be postponed until a later Discoteq version.

Another transport method that did not even pass to the testing stage was the idea of using MSMQ[10] in conjunction with .NET Remoting. This method was quickly scrapped during discussions, simply because there would be no need for such a distributed system - at least not for direct Discoteq communication.

5 The extra compilation step and XML

The extra compilation step - going from XML code to C# code - can be seen as an unnecessary and complicated task. The explanation is divided in two sections; the reason for XML and the reason to generate C# code instead of parsing the XML directly.

Why XML?

XML is used for VFD:s mainly because it has been used with success in the former version of the system and the Discoteq 1.x developers are sufficiently familiar with the XML data format. XML is furthermore a well-defined standard and is widely supported by everything from text editors to parser APIs.

Although the XML format is awkward in heavy scripting, the scripting of Discoteq functions should not become too complicated, as the real computing should be made in subsystems to Discoteq, such as an SQL database or a C++ DLL. For example, there are no loop tags in the original specification of the Discoteq language.

XML constructs can be tailored to the scope of Discoteq so less code has to be written, a greater overview can be attained, and documentation can be automatically generated in different levels of detail. As the Discoteq language is very high-level and fairly restricted, there could even be possible to extract functionality descriptions for methods without having the developer write a single comment in the code.

Why generate C# code?

Discoteq 1.x parses XML every time a request is made, and testing showed a worse-than-linear slowdown of execution when the XML documents grew large and complicated (the main reason for this is due to complicated and generalizing XPath [[xvii]] lookups). To battle the slow processing of XML files, Discoteq 2.0 generates several blocks of C# code for each Discoteq application defined in the available VFD:s. The code is then compiled into .NET Assemblies, which means a very fast and efficient execution.

There are some dangers in doing this; the code-generation process is fairly complicated and susceptible to programming errors. The good thing is that this only needs to be done once for every tag in the Discoteq language, and if the code behaves correctly in all situations, there is a very strong possibility it will continue to do so even if the code for other tags are changed or added. Partly, this is thanks to the forced insertion of try-catch[11] blocks around the generated code for each tag.

6 Distributed Approach

The idea of having a code "store" came from current version control systems such as CVS and Microsoft SourceSafe. However the automatic generation of C# code and DLL files does not make these systems fit for Discoteq. The Discoteq code store is defined by the interface ICodeStore, which has the following layout:

[pic]

Figure 9: ICodeStore

Apart from being able to download Discoteq client assemblies and WSDL, it is also possible to get the collected VFD for any available applications, or just the latest modification time of the collected VFD. This enables other Discoteq Servers to fetch this document from a central server, enabling a truly centralized code store.

Overall, the Discoteq system is explicitly distributed which means there are no automatic algorithms to distribute the workload over several computers. Instead, the Discoteq application developer must specify how an application should be distributed, and the Discoteq client developer or user must specify what server(s) to contact. To make things easier there are VFD constructs that can be used for data distribution; for example, the concurrent element makes child elements (which can only be method calls) execute in parallel.

7 Thin vs. Rich Clients

Discoteq 1.x uses thin clients because all results are sent as XML data, which is parsed directly into HTML pages with little logic other than XSL [[xviii]] styling on the client side. However, as the processing ability of most clients has grown substantially since the first appearance of Discoteq 1.x, there is no longer a need for thin clients, and mobile units will be using web-based access anyway.

Instead, straightforward development on the client side has been favored in front of complex formatting schemes. There were formatting additions planned for the Discoteq type library, but the limited time did not allow for inclusion of this in the Discoteq 2.0 release. .NET clients are by definition "rich" clients because the .NET framework and CLR is needed for execution.

8 Automatic Compilation

The automatic compilation was thought of as a helper for true development servers where quick results are needed. A thread is repeatedly scanning the file system and remote code stores for any changes, and if changes are found, it recompiles the affected Discoteq applications.

9 Versioning

Discoteq includes a versioning concept that differs from most others; instead of just having the source versioned, the compiled application is also versioned. This means that in many cases, old clients can continue to work even if the server code is upgraded. The reason behind this scheme is the need for long-lived applications.

The version numbering follows the standard .NET scheme, which is major[.minor[.release[.build]]]. When compiling VFD:s the following rules apply to versioning:

• Release and build version numbers are ignored.

• Different major versions are always incompatible.

• If the minor versions differ, the closest lower method version with the same major version is selected.

• If a no other match is found, and a method that has no version exists, that method is selected.

Here are some examples:

|Client version |Discoteq method version |Compatible |

|2.3 |2.3.1.18 |Yes - Equals |

|1.0 |2.0 |No |

|2.0 |1.0 |No |

|1.3 |1.5 |No |

|1.5 |1.3 |Yes |

|2.2 |none |Yes |

|0.9 |none |Yes |

Table 5: Discoteq versioning example

To clarify the process even more, consider the following method versions:

|Find 2.1 |

|Search 1.0 |

|Search 2.1 |

|Search 2.3 |

Table 6: Method versions

These would be split into the following three application versions, with the specified methods:

|Version 1.0 |Version 2.1 |Version 2.3 |

| |Find 2.1 |Find 2.1 |

|Search 1.0 |Search 2.1 |Search 2.3 |

Table 7: Application and method versions

10 Security and Compression

As there is no NT-security style remoting channels included in the .NET Framework, there were plans to run parts of the system through a secure HTTPS connection using IIS to deploy the remote objects, but some bug or configuration error made this impossible. Not until the very last stages of Discoteq 2.0 development, Microsoft released a MSDN example of an NT-style security remoting channel. To make this work to full extent in the Discoteq system, additions to the class and method tags are needed, as well as internal user/group handling. The task of implementing this in new releases of Discoteq should not be too difficult.

Compression of data is an even simpler task. Several documents on the web describe various methods to implement compression into .NET remoting connections. There should be no problem to implement compression if there is any need for it.

11 Developer roles

The roles defined it the "Implementation/Roles" section includes three developer roles; the client developer, the application developer and the plugin developer. Because of these roles, which are highly (but not entirely) separated, knowledge of the Discoteq system can be distributed over several individuals. This is all connected to the efficiency goal of the Discoteq design; by not having to learn as much, a developer can be acquainted with a system much quicker, which saves time and resources.

There is one role that is omitted in the implementation section; the role of the Discoteq system developer(s). Naturally, this role is the most demanding of all, as it encompasses most of the knowledge from the other roles. Naturally, the plugin developer comes very near the Discoteq system developer, but not quite.

The roles do have dependencies though. To depict these dependencies, a role-hierarchy-tree is showed beneath.

[pic]

Figure 10: Discoteq developer role hierarchy

12 Use Cases

The use cases are limited to the development process as the client usage patterns can vary quite a lot. The development process is divided in three separate parts:

Discoteq Application Design

The VFD design process for a new Discoteq application is initiated by the need for some kind of new and separate functionality. For example, the addition of a new database system. The client developers might also want added functionality to existing Discoteq applications. The following points illustrate the building and modification of a Discoteq application.

• An application name is choosed. The name should be sufficently unique, but short enough to not bloat client code. In this example the application is called "AircraftDB".

• Logic groups are formed, called Discoteq classes. When handling a database, these could for example be: Search, Modify and Retrieve.

• Discoteq methods are added to the Discoteq classes. The methods provide the logic and functionality, and should be named as unambiguous as possible. For example, a method under the "Retrieve" class could be named "AircraftCockpitLayout", doing the obvious thing: Retrieve the cockpit layout for a specific aircraft model. The method should also get a temporary version number, for example, version 1.1.

• Logic is added to the methods using the available VFD language elements.

• The VFD is compiled and tested. While working with the different methods, either the "build" or "release" numbers should be updated to reflect work in progress.

• When a method is deemed "release" quality, the version number should be updated to reflect this. For example, the minor number could be increased, creating the version 1.2.

• If functionality or parameters need to be changed after this point, a copy of the method should be made, and the version number should once again be updated to reflect a method in development (version 1.3).

• This creates two methods, one providing the old functionality, and one method providing the new functionality. When the old functionality is not needed anymore, the old method is simply deleted.

Discoteq Client Design

The client design process is probably the most straightforward of all, at least after having downloaded the correct assembly.

• A new client is needed for a certain Discoteq application, for example, a client showing an image of a cockpit layout in a selected aircraft model.

• The client developer downloads the latest version of the Discoteq client assembly for the Discoteq application "AircraftDB".

• The assembly is added to the project, and if this is done in Visual Studio .NET, the method signature is also added to the context-sensitive help.

• Apart from forming the GUI and the logic behind displaying the image, only three lines has to be written to connect and execute the wanted Discoteq method:

o Get access to the Discoteq client classes:

using Discoteq;

o Create a connection to a Discoteq application:

AircraftDBApplication adba = AircraftDBHelper.Connect(...);

o Finally, make the call:

adba.Retrieve_AircraftCockpitLayout(sAcID);

Discoteq Plugin Design

A Discoteq application developer (who writes VFD:s) sees that she is hampered by functionality limitations in the VFD language, so she contacts a plugin developer and describes the problem. The VFD developer could for example need a method that converts an image in uncompressed TIFF format to JPG format.

• The plugin developer then defines additional VFD elements and their functionality, and uses any of the available methods (either IPlugin or PluginBase) to implement the code generating classes. An XML documentation file for the element is also created. The XML element could in this case be named "imageConvert" and have the attributes "fromType", "toType" and "applyOn".

• After thorough testing, the code plugin classes are compiled to an assembly and incorporated into the Discoteq server.

• The XML help file is distributed to the Discoteq application developers, who can use them as HTML documentation or through the GUI.

13 AECMA 1000D Integration

AECMA (Aircraft European Contractors Manufacturers Association) has created a standard for technical publications, which is called 1000D [[xix]].

As part of the thesis, an evaluation of possible AECMA 1000D Integration in Discoteq 2.0 should be made, but the documentation available from the website was unfortunately not complete so only a quick summary has been made.

Data Modules

The smallest parts of 1000D documentation are called Data Modules. Data Modules uses a numbering scheme called DMC (Data Module Code). The code is divided in several parts, to facilitate easy identification of data modules. It resembles an IP address in functionality, as each part of the code narrows the search until one single document is found. The DMC can be easily applied to a relational database, which Discoteq 2.0 applications are mostly based upon. In addition, special DMC .NET types can be created. A theoretical DMC might look like this: AB-001-12-104; "AB" might denote the manufacturer, "001" the product type, "12" the product, and "104" the document number.

Document Format

The 1000D document format is based on SGML[12], and there are DTD:s[13] describing a number of standardized document types. The DTD:s can be converted into .NET classes, forming a type library for 1000D documentation. The documents can be stored as large text objects in a relational database, or in a filesystem tree with folder names based on DMC:s.

[pic]

Figure 11: Theoretical DMC-based filesystem tree

Integration

As it looks, no internal modification to Discoteq 2.0 has to be made in order to use AECMA 1000D documents. The task seems to come mostly on database development, a task outside the Discoteq 2.0 scope.

Conclusion

Even though some things had to be dropped from the project, both AerotechTelub and I are satisfied with the result. Discoteq 2.0 might very well become a platform for further developement of easy database access and hardware integration. During a test, the time to start developing a Discoteq 2.0 application, including assembly download, were estimated to around 10-15 minutes. By executing only two method calls, a Discoteq 2.0 connection and database access can be made. All this means a very short time-to-market, which also means cost-efficiency, an important keyword in the software business today.

The hardest work was to understand what was wanted from the Discoteq 2.0 system. As a result, the thesis has become less focused on a single aspect, and instead spread over several areas. There was also the need for a solid product, which needed many hours of implementation and testing to get everything ready for release at the time of the deadline.

I have also gained much personal experience during this time; both from having worked at a large company, and from working with the new .NET environment.

References

-----------------------

[1] Elledningsvisning (electrical wire display)

[2] There are several views on what a platform really is; I mean, “any hardware and/or software that is designed to further build upon” - just like a platform for a house.

[3] Shared-source is Microsoft’s own open-source model.

[4] The Common Object Request Broker Architecture, a specification created by the Object management Group

[5] Remote Method Invocation, part of the Java2 SDK

[6] Marshaling is the process of registering and publishing a remote object.

[7] A singleton is an object that is created once then called a number of times. A single-call object is created, called once then destroyed.

[8] The Microsoft Internet Information Server

[9] The @-sign denotes a variable reference, in this case to the first parameter in the Discoteq call.

[10] Microsoft Message Queuing

[11] Not only does the try-catch block handle exceptions; it also separates the code from other code blocks that are siblings of the current block.

[12] Standard Generalized Markup Language, not very unlike XML and HTML

[13] Document Type Declaration, defines the layout of a document

-----------------------

[i] Simon Robinson : Introduction to .NET, 2000-12-01 (2003-01)

[ii] Microsoft: Key Events in Microsoft History (2003-01)

[iii] Netcraft : Netcraft Web Server Survey, 2003-01-xx (2003-01)

[iv] Microsoft: ECMA C# and Common Language Infrastructure Standards, (2003-01)

[v] Sun Microsystems , VM Spec Introduction: (2003-01)

[vi] David Sims , Tim O'Reilly , Rael Dornfest : Microsoft Plans Shared Source .NET, 2001-06-27 (2003-01)

[vii] Microsoft, David Stutz: The Microsoft Shared Source CLI Implementation, 2002-03-xx,

(2003-01)

[viii] Fujitsu: NetCOBOL for .NET (2003-01)

[ix] Sun Microsystems: J2SE Downloads, 2002-12-04 (2002-09)

[x] Ximian, Mono : Home / Mono, 2003-01-17, (2003-01)

[xi] W3C, Nilo Mitra : SOAP Version 1.2 Part 0: Primer, 2002-12-19

(2003-01)

[xii] Microsoft, Priya Dhawan, Tim Ewald: Web Services or .NET Remoting: How to Choose, 2002-09-xx

(2003-01)

[xiii] Microsoft, Priya Dhawan: Performance Comparison: .NET Remoting vs. Web Services, 2002-09-xx

(2003-01)

[xiv] W3C, Tim Bray , Jean Paoli , C.M. Sperberg-McQueen , Eve Maler : Extensible Markup Language (XML) 1.0 (Second Edition), 2000-10-06

(2002-09)

[xv] N. Freed, N. Borenstein: RFC2045, Multipurpose Internet Mail Extensions (MIME) Part One, 1996-11-xx (2003-01)

[xvi] W3C, John J. Barton , Satish Thatte , Henrik Frystyk Nielsen : SOAP Messages with Attachments, 2000-12-11 (2002-10)

[xvii] W3C, James Clark , Steve DeRose : XML Path Language (XPath), 1999-11-16 (2002-10)

[xviii] W3C, Sharon Adler , Anders Berglund et al: Extensible Stylesheet Language (XSL), 2001-10-15 (2002-10)

[xix] AECMA: TPSMG Home, 2002-05-07 (2002-12)

-----------------------

Client

Server

ICodeStore

IAppInstantiator

IApplicationInfo

Remote

Application

Object

Remote

call

Fetch application wrapper and

reference to application object

Instantiation and

marshaling of

wrapper

Client Assembly

Session

Construction call

Downloading of Assembly (DLL) or WSDL

Discoteq

Client Developer

Service

From the

Discoteq

Compiler

Updating of the

Discoteq

Compiler

Discoteq

Discoteq

Internal

Application-

specific

Automatically

generated

Application

Logic

(version independent)

Wrapper 1.0

Wrapper 1.4

Wrapper 2.0

Interface 1.0

Interface 1.4

Interface 2.0

Inheritance

Call

DBMS

Hardware

Discoteq

(D)COM

Distribution to developers

through ICodeStore

interface

Discoteq.Server.ICodeStore

+GetAppNames:ICollection

+GetAppInfo:IApplicationInfo

+GetClientInstallPackage:VFile

+GetAppDoc:string

+GetAppTime:DateTime

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download