Introduction



-3810034925Accelerating Excel??2010 with Windows? HPC Server 2008?R25534025409575: Building Custom Clients and ServicesPublished: May 2010 By: Duncan Werner, Managing Member, Structured Data LLCAbstractThis article addresses how to build custom client applications and services for running Excel calculations on a Windows HPC 2008 R2 cluster.HPC Services for Excel supports a variety of new programming models that allow you to run Excel calculations in parallel on a Windows HPC 2008 R2 cluster. HPC Services for Excel is based on the Service-Oriented Architecture (SOA) programming model, and includes a built-in SOA client and a built-in service. The built-in components support a large number of use cases. In some cases you might want to create custom clients that use the built-in service, or build custom clients and services.This article walks through three basic patterns for creating client applications that use the built-in HPC/Excel service: the simple model, the interactive model, and the fire-and-recollect model. The article also walks through building a custom service library, and an application that utilizes the custom service. As we go through the code examples, we’ll discuss when it’s appropriate to use custom clients and services, and the types of applications that you can build.This is a preliminary document and may be changed substantially prior to final commercial release of the software described herein.The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS plying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, email address, logo, person, place or event is intended or should be inferred.? 2010 Microsoft Corporation. All rights reserved.Microsoft, Excel, SQL?Server, Visual Basic, Visual Studio, Windows, and Windows Vista are registered trademarks of Microsoft Corporation in the United States and/or other countries.The names of actual companies and products mentioned herein may be the trademarks of their respective owners. TOC \o "1-2" \h \z \u Introduction PAGEREF _Toc260748343 \h 4Intended Audience PAGEREF _Toc260748344 \h 4Download Files PAGEREF _Toc260748345 \h 5Before You Start: Prerequisites and Requirements PAGEREF _Toc260748346 \h 5On the cluster PAGEREF _Toc260748347 \h 5On the desktop PAGEREF _Toc260748348 \h 7Summary PAGEREF _Toc260748349 \h 7Building an HPC/Excel Client Application PAGEREF _Toc260748350 \h 8Building an HPC/Excel Client Application: Interactive Calculation PAGEREF _Toc260748351 \h 17Building an HPC/Excel Client Application: Fire and Recollect PAGEREF _Toc260748352 \h 27Review: Building HPC/Excel Client Applications PAGEREF _Toc260748353 \h 35Building a Custom HPC/Excel Calculation Service PAGEREF _Toc260748354 \h 35Deploying the HPC/Excel Service PAGEREF _Toc260748355 \h 45Building an HPC/Excel Client Application for the New Service PAGEREF _Toc260748356 \h 47IntroductionHPC Services for Excel is a comprehensive set of tools for running Excel spreadsheet calculations on a Windows HPC 2008 R2 cluster. With HPC Services for Excel, you can build applications that run Excel workbooks in parallel for faster overall calculation.HPC Services for Excel is based on the Service-Oriented Architecture (SOA), a feature of Windows HPC Server 2008 R2. In this model the cluster supports services, code that implement a defined set of method calls; and applications can act as clients, calling methods in those cluster services. This framework removes a lot of the complexity of building applications and running calculations on an HPC cluster. HPC Services for Excel includes default libraries that implement this client and service framework. The default client application is a library you can use from Excel – directly from Excel VBA – which we described in a previous article. The default service is an HPC cluster service which uses Excel on the cluster compute nodes to calculate workbooks in parallel. The simplest and easiest way to use HPC Services for Excel is to use the default client library, and run calculations directly from Excel VBA. Using that framework means you don’t have to write any code – you can design the cluster calculation completely from VBA.That framework supports a large number of use cases. However there are other cases in which the Excel plug-in isn’t appropriate: for example, if you don’t want to use Excel as the user interface. In those cases, it can be more appropriate to build a custom application using HPC Services for Excel. In most cases, even if you’re not using the default client library, you can still use the default HPC Services for Excel service – the application running on the cluster compute nodes – to manage the Excel calculation. In those cases you need only write the client code. In some other cases, you might wish to write your own custom service library as well.In this article we’ll address both of these cases: first building custom client applications, utilizing the default HPC/Excel service; and then subsequently building a custom service library, and an application that utilizes the custom service. As we go through the code examples, we’ll discuss when it’s appropriate to use custom clients and services, and the types of applications you can build.Intended AudienceThis article is intended for developers familiar with C# and .NET development generally. You should be comfortable building applications in .NET using Visual Studio.Building HPC cluster applications (clients and services) uses Windows Communications Foundation (WCF) tools and libraries. We’ll walk through the development steps, so you don’t need prior experience with WCF or HPC development.For training resources on C# and .NET application development generally, see additional training resources on WCF specifically, see FilesIncluded with this article are pre-built Visual Studio solutions for both VS 2008 and VS 2010. The steps below walk through the process of building client and service applications from scratch, but if you’d prefer to just follow along with the code, you can use these pre-built solutions.The download files also include Excel workbooks and some support files used to build, test, and deploy cluster applications. We’ll refer to the specific files in the development steps that follow.Before You Start: Prerequisites and RequirementsHPC Services for Excel is a set of tools included with the Microsoft HPC Pack 2008 R2 Beta 2 and later versions. You’ll need a Windows HPC Server 2008 R2 cluster installed and configured. Installing an HPC cluster is outside of the scope of this article; we’ll only address the specific configuration you will need for HPC Services for Excel. For more information on installing and configuring the cluster, see the documentation included with Microsoft HPC Pack 2008 R2. On your desktop, you’ll need the HPC Pack 2008 R2 client utilities and the HPC Pack 2008 R2 SDK. You’ll also need Microsoft Visual Studio to compile the example applications.On the clusterAfter you have an HPC cluster up and running, you’ll need to install Excel 2010 on the cluster compute nodes. You can use the standard Office 2010 installation kit to install Excel, or see the HPC Server 2008 R2 documentation for more information ways to automate the Excel installation.After Excel is installed on your cluster compute nodes, you can run the HPC diagnostic test to ensure that everything is configured properly. To run the diagnostic test, use HPC Cluster Manager (either on the cluster head node, where it’s installed by default, or on your desktop if you have installed the client utilities).To verify that the HPC service for Excel are configured correctlyIn the HPC Cluster Manager, click Diagnostics.In Tests, expand Microsoft and then select Excel.In the view pane, double-click Excel Runner Configuration Test.In the configuration dialog, select All Nodes and then click Run.In the Navigation Pane, click Test Results to see the progress of the test. The test that you are running appears in the view pane. After the test is complete, you can click on the test in the main window to see the results.If the test shows Failure, double-click the test and try to correct any errors in the popup window. Common errors you might see are that Excel is not installed on a compute node, or that Excel has not been activated. Make any changes you need and then re-run the diagnostic test using the instructions above. When the test shows Success, you’re ready to continue. The last thing you’ll need is a share directory. When we calculate a workbook on an HPC cluster, each of the cluster compute nodes actually loads and runs a copy of the workbook. To support that, you need to create a share directory that is visible both to the cluster compute nodes and to your desktop. If you have access to the cluster head node, create a share directory on the head node. This is convenient because you know that the cluster compute nodes can access the head node. If you can’t create a share directory on the head node, create the share directory anywhere within the domain that is accessible both to your desktop and to the cluster compute nodes.On the desktopOn the desktop, you’ll need to install the Windows HPC Pack 2008 R2 client utilities and the Windows HPC Pack 2008 R2 SDK (installing the client utilities and SDK will require Administrator permissions). To install the client utilities, run the HPC Pack 2008 R2 installer on your desktop. It will offer an option for installing just the client utilities. To install the SDK, run the HPC Pack 2008 R2 SDK installer. SummaryYou should now have:A Windows HPC 2008 R2 cluster installed and configured.Excel 2010 installed on the cluster compute nodes. A network share directory. The Windows HPC Pack 2008 R2 client utilities and SDK installed on your desktop. Building an HPC/Excel Client ApplicationHPC Services for Excel includes a default client and service framework for calculating Excel workbooks in parallel on a Windows HPC 2008 R2 cluster. The default client application is a library you can use from Excel – in VBA – which we described in a previous article. The default service is an HPC cluster service which uses Excel on the cluster compute nodes to calculate Excel workbooks in parallel. In this article we’ll address writing custom HPC/Excel applications – first client applications, using the default HPC Services for Excel service; followed by writing a custom HPC/Excel service and an application to utilize the custom service.To build the basic HPC/Excel client application:Prepare your environmentOnce you have all the prerequisites installed (following the Before you Start section, above), you are ready to run Excel calculations on the cluster. To run this application, however, we need one more step: we need to create a share directory for the example workbook.Each compute node in the cluster must be able to load the workbook for calculation. There are lots of ways to manage this, but we will design this application to access a workbook on a network share directory.Create a share directory that is visible by the cluster compute nodes. Typically we create this directory on the cluster head node. Make sure that the directory is readable by the cluster compute nodes, and writable by your user account. Then copy the Excel workbook “ExampleWorkbook1.xlsb” from the download files to this share directory.Create a new projectOpen Visual Studio and create a new project (in a new solution). In the project selection dialog, select Visual C# > Windows > Console Application. Name the new application “BasicClientApplication”.Add project referencesRight-click the new project and select Add Reference… We’ll need to add references to some standard .NET components and to the HPC SDK components. (You may need to add these references in several steps; if it’s easier, just add one at a time until you have all of the required references).In the .NET tab of the references dialog, add references to “System.ServiceModel” and “System.Runtime.Serialization”. In the Browse tab of the references dialog, locate the Windows HPC Server 2008 R2 SDK directory. By default this will be installed in “C:\Program Files\Microsoft HPC Pack 2008 R2 SDK”. Enter the “Bin” directory and add references to “Microsoft.Hpc.Scheduler.dll”; “Microsoft.Hpc.Scheduler.Properties.dll”; and“Microsoft.Hpc.Scheduler.Session.dll”.Add a reference to the Microsoft.Hpc.Excel library. This library is provided with the HPC 2008 R2 SDK and installed in the Global Assembly Cache (GAC), but it may not be visible in the .NET tab. To make this easier we’ve provided a copy of this library with the download files. In the Browse tab of the references dialog, locate the download files for this article and add a reference to “Microsoft.Hpc.Excel.dll”.You should now have the following references added to your project:Microsoft.Hpc.ExcelMicrosoft.Hpc.SchedulerMicrosoft.Hpc.Scheduler.PropertiesMicrosoft.Hpc.Scheduler.SessionSystem.Runtime.SerializationSystem.ServiceModelAdd directivesTo simplify the code, add directives for the library namespaces. At the top of the file add these directives:using Microsoft.Hpc.Scheduler;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;using Microsoft.Hpc.Excel;using Microsoft.Hpc.Excel.ExcelService;If any of these directives generate Visual Studio errors, double-check the list of references added in the last step. Implement the main methodIn this section we’ll write the application code. We’ll add code in a few separate sections, so we can describe each section in turn. At the end of this step we’ll include the full code as one block so you can compare it to the code you’ve added.First we create some basic parameters. In a “real” application you would probably use arguments to the application to collect these parameters. For now, we’ll hard-code these values into the application; that will make it easier to run the application from the debugger. static void Main(string[] args) { // change these values to match your cluster head node // and path to the test workbook (on a network share directory) const string headNode = "HeadNodeName"; const string workbookPath = @"\\path\to\share\directory\ExampleWorkbook1.xlsb";Change the values for headNode and workbookPath. For headNode, use the network name of your cluster head node. If you’re in the same domain, you can use just the machine name. If you’re in another domain or subdomain, use the fully-qualified name of the machine. For the workbookPath value, set the full path to the workbook in the share directory you created in step 1, above. In the next section we create and initialize the cluster session: Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo( headNode, "Microsoft.Hpc.Excel.ExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; info.Environments.Add("Microsoft.Hpc.Excel.WorkbookPath", workbookPath ); // step two: initialize the session and create the client object Console.WriteLine("Creating session and client"); Session session = Session.CreateSession(info);The SessionStartInfo class manages settings for the cluster session, such as the unit type (e.g. node or core) and the number of requested units. The second argument to the class constructor, “Microsoft.Hpc.Excel.ExcelService”, is the name of the cluster service we will use for calculation.Once again we are using the default HPC/Excel cluster service, which is provided when you install Windows HPC Server 2008 R2. Cluster services are identified by unique strings. Whenever we want to connect to the default cluster service, we use the string identifier. That tells the cluster to create a session with the specific service.The default HPC/Excel service uses an environment variable to identify the workbook. SessionStartInfo.Environments can be used to set environment variables visible to the cluster service. When we call a method to calculate a workbook (below), the service will start Excel and load the workbook specified by the environment variable.In the next section we create an instance of the BrokerClient class: BrokerClient<IExcelService> client = new BrokerClient<IExcelService>(session);BrokerClient is a generic type which uses an interface to create a specific class instance. The interface type – in this case, IExcelService – is specified in the Microsoft.Hpc.Excel library which we added as a reference in step 3, above.The Windows HPC Server 2008 R2 service-oriented architecture (SOA) model uses a number of generic types, including BrokerClient, to implement classes and methods. This architecture allows the code structure to be consistent for most cluster services, with only the specific interfaces and class objects changing. If you follow the code examples in this article, you should get a reasonable understanding of the generic types and how they’re used. Full documentation for the generic types is available on the web at (v=VS.85).aspx.After the session and client object have been created, we can send calculation requests to the cluster service: // step three: send requests for calculation Console.WriteLine( "Sending requests..."); for (int i = 0; i < 16; i++) { CalculateRequest request = new CalculateRequest(); request.macroName = "MyCalculationMacro"; request.inputs = WorkItem.Serialize(new WorkItem(new object[] { i })); client.SendRequest<CalculateRequest>(request, i); } client.EndRequests();In this section we’re sending 16 calculation requests to the cluster service, in a loop. Each request is created as a class object, of the type CalculateRequest. CalculateRequest is included in the Microsoft.Hpc.Excel library, which you added as a reference in step 3. In the first line of this code, we create an instance of CalculateRequest. CalculateRequest is a Message Contract object. Windows HPC Server 2008 R2 supports the Service-Oriented Architecture (SOA), which is itself built on Windows Communications Foundation (WCF). WCF defines a structure for building distributed applications and services, which are used in web service applications as well as in HPC cluster applications.Every WCF service provides an interface, a list of functions that client applications can call. In this case the interface is IExcelService (which you saw in the last code section when we created the BrokerClient object). Message Contract objects are a framework implementation which allow client applications to construct method calls using an object-oriented interface. This creates the method call as an encapsulated object which can be sent to the service. In the code above, we create an instance of the CalculateRequest object. (In the interface, this represents a service function called “Calculate”). We then set parameters of this object, which represent the arguments to the function. For example, the macroName parameter indicates a VBA macro that we want Excel to call when it runs a calculation.All of that taken together describes the internals of SOA / WCF services and how they are implemented in the framework. It’s not important to understand the specifics, as long as you understand how to use these objects. To send a request to a cluster service, you (a) create an instance of the request object; (b) set parameters on the object; and (c) call BrokerClient.SendRequest.The line request.inputs = WorkItem.Serialize(new WorkItem(new object[]{ i }));represents another parameter sent with the calculation request. This line is a little complicated so we’ll explain it in detail.As with the macroName parameter, this line is setting an argument to the calculate function which we will call on the cluster service. The data set here uses the class WorkItem to create a serialized object which can be passed to the function call. WorkItem is a class provided by the Microsoft.Hpc.Excel library.The reason for this structure has to do with how the Excel/HPC service works. The service is designed to call a VBA macro in an Excel workbook. VBA uses COM (the Component Object Model), but the service and client are written in .NET. For that reason we have to be somewhat careful when managing data. In particular, the two frameworks use different Array types. To support sending arbitrary data to Excel VBA – including arrays of values – we need to encapsulate and serialize the data.It’s not necessary to understand too much about the WorkItem class to build Excel/HPC applications. There are four important functions: Insert; Get; Serialize; and Deserialize. The Add and Get functions store and retrieve data in the object. The Serialize and Deserialize methods convert to object to and from byte streams.For the purposes of this application, we add one value to the WorkItem – the integer from the loop. Then we call Serialize to pass the data to the CalculateRequest object as a byte stream. We’ll see Get and Deserialize in later code sections.When the method runs on the cluster, whatever we’ve stored in the WorkItem will be passed to the VBA macro (specified by the macroName parameter). In VBA, this data will arrive as a generic Variant type which can be treated as an Array or as any simple type.The next line of this code section calls BrokerClient.SendRequest to send the request to the cluster for calculation. This is a generic method and includes the request type, here CalculateRequest. We pass two parameters to this method. The first is the request object itself, representing the method call and the method arguments. The second argument is a “UserData” object, which can be any value. The purpose of the UserData object is to match requests we send here with responses we receive later. Because the calculation runs in parallel, and runs asynchronously, it’s possible that we will receive responses in a different order than the order in which they were sent. If it’s important that we handle results in order, or if we want to match the input data to the calculation result, we can use the UserData parameter. In this case, the UserData value is just the integer representing the calculation number.Finally the last line of this section calls BrokerClient.EndRequests, indicating that we have finished sending requests and we are ready to receive the results. BrokerClient.EndRequests tells the scheduler to commit all open requests and it’s good practice to call this after each set of requests.Note: after you call BrokerClient.EndRequests, you cannot send any more requests using the same BrokerClient object. If you would like to send an additional set of requests, create a new BrokerClient object with a unique ID. We’ll explain this in more detail in the next section on interactive clients.In the next section of code we wait for the 10 calculations to complete: // step four: retrieve responses from the cluster Console.WriteLine( "Waiting for responses..." ); BrokerResponseEnumerator<CalculateResponse> responses = client.GetResponses<CalculateResponse>(); foreach (BrokerResponse<CalculateResponse> response in responses) { int userData = response.GetUserData<int>(); WorkItem workItem = WorkItem.Deserialize(response.Result.CalculateResult); object obj = workItem.Get<object>(0); Console.WriteLine("Request: {0}, response: {1}", userData, obj); }Once again there are a number of generic types, but this should start to become familiar. The method BrokerClient.GetResponses retrieves all responses from the service – these should match the 10 requests we sent. BrokerClient.GetResponses is another generic method, here using the type CalculateResponse. CalculateReponse mirrors the CalculateRequest object; taken together, these represent the method call and the result of a call to the service function “Calculate”. The BrokerClient.GetResponses method returns an enumerator object (again generic, with the same type CalculateResponse). To walk through the individual responses, we can loop over the enumeration using foreach.Each response object includes two important members: the calculation result and the UserData object. The UserData object is the same object we set when we sent the requests in the last code section. Again this was an integer value, representing the calculation number.The result of the function is a generic object representing the return value of the service function. In this case this is another WorkItem, so we use the method WorkItem.Deserialize to convert it to a class type, and then WorkItem.Get to retrieve the result value.The VBA macro we called during the calculation returned a generic Variant type. The HPC/Excel service converted this type to a WorkItem and then a serialized bit stream, which we’re deserializing here. As far as VBA is concerned, this value is just a generic object: it can be any primitive type or an array of values. When the WorkItem is deserialized here, we can use the Get method to retrieve values. In this case, the VBA macro is returning a simple double value; we can retrieve the value from the WorkItem and then process it.Using the UserData object and the result obtained from the WorkItem, we can display the calculation result to the user. The UserData matches the input value to the function, and the result is the calculation result from Excel on one of the compute nodes.The last section of code closes the client object and the session: // done - clean up Console.WriteLine("Cleaning up"); client.Close(); session.Close(); Console.WriteLine("Press return to exit"); Console.ReadKey(); }The complete main method, including all the code added above, should look like this:using System;using System.Collections.Generic;using System.Linq;using System.Text;using Microsoft.Hpc.Excel;using Microsoft.Hpc.Excel.ExcelService;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;namespace BasicClientApplication{ class Program { static void Main(string[] args) { // change these values to match your cluster head node // and path to the test workbook (on a network share directory) const string headNode = "HeadNodeName"; const string workbookPath = @"\\path\to\share\directory\ExampleWorkbook1.xlsb"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo( headNode, "Microsoft.Hpc.Excel.ExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; info.Environments.Add("Microsoft.Hpc.Excel.WorkbookPath", workbookPath ); // step two: initialize the session and create the client object Console.WriteLine("Creating session and client"); Session session = Session.CreateSession(info); BrokerClient<IExcelService> client = new BrokerClient<IExcelService>(session); // step three: send requests for calculation Console.WriteLine( "Sending requests..."); for (int i = 0; i < 16; i++) { CalculateRequest request = new CalculateRequest(); request.macroName = "MyCalculationMacro"; request.inputs = WorkItem.Serialize(new WorkItem(new object[] { i })); client.SendRequest<CalculateRequest>(request, i); } client.EndRequests(); // step four: retrieve responses from the cluster Console.WriteLine( "Waiting for responses..." ); BrokerResponseEnumerator<CalculateResponse> responses = client.GetResponses<CalculateResponse>(); foreach (BrokerResponse<CalculateResponse> response in responses) { int userData = response.GetUserData<int>(); WorkItem workItem = WorkItem.Deserialize(response.Result.CalculateResult); object obj = workItem.Get<object>(0); Console.WriteLine("Request: {0}, response: {1}", userData, obj); } // done - clean up Console.WriteLine("Cleaning up"); client.Close(); session.Close(); Console.WriteLine("Press return to exit"); Console.ReadKey(); } }}Running the client applicationYou should now be able to run the client application. Right-click the project and select Set as Startup Project; then run the application in the debugger from the menu Debug > Start Debugging. If everything works, you should see results similar toDepending on your cluster, the results may not be in the same order. In most cases the results will not be in numerical order, even though we sent the requests in numerical order. This is because different compute nodes may run the calculation faster or slower than others. We use the “UserData” parameter in the request and response objects to match requests to responses, and in the output of the application – whatever order the results are presented – the “response” value displayed should always be the square of the “request” value.If we wanted to sort the results, we could use that UserData parameter as a sort index. Or, if we had a number of input values to the function, we could use the full set of input values (as an array) as the UserData parameter, to better report results.Debugging the client applicationIf you receive compilation errors when you try to run the application, double-check the code against the code listing at the end of step 5, above. If Visual Studio indicates that one or more references are missing, check the references added in step 3.If the debugger raises an exception while the application is running, check the exception message for help on resolving it. Common issues at this point are Cluster settingsMake sure that the cluster head node specified in the code file matches your cluster head node address. Check that the workbook “ExampleWorkbook1.xlsb” is deployed in your network share (as described in step 1, above) and that the full path to the workbook is correct in the code file.PermissionsMake sure that your user account has permissions to run jobs on the cluster. In the HPC Cluster Manager application, click the Configuration tab at the bottom-left and then click Add or Remove Users.ConfigurationMake sure that the cluster is configured properly and that Excel is installed on the compute nodes, as described in the above section “Before you Start”.Verify that the settings and configuration are correct and try running the application again.Note: if you stop the debugger while the application is running, the cluster session may not close. In that case, subsequent calculations may be stuck in the “Queued” state. If you stop the debugger, open the HPC Cluster Manager application and click the Job Management tab at the bottom-left. If the application is not running but you see a cluster job with the name “Microsoft.Hpc.Excel.ExcelService” in the Running state, right-click the job name and select Cancel Job to cancel.Building an HPC/Excel Client Application: Interactive CalculationThe simple application described in the previous section demonstrates the easiest way to send calculation requests to an HPC cluster and retrieve calculation results. Although it’s simple to implement, there are some drawbacks to that method.In the first application, we sent a number of requests; called BrokerClient.EndRequests; and then used BrokerClient.GetResponses to collect the calculation results. The alternative is to use an asynchronous code pattern, with a BrokerResponseHandler. The BrokerResponseHandler class uses a delegate method to handle responses as they arrive. The asynchronous code pattern better supports Windows applications (stand-alone Windows Forms applications, or plug-ins to other applications such as Excel). The pattern used in the first application – the simple send requests / wait for results pattern – will block the application, preventing the user interface from updating. In most applications this is undesirable; we want to update the user interface to display progress, repaint the screen, or run other code functions.To build an interactive client application:Create a new projectYou can create this project in the same solution as the previous projects, or create a new solution. Create a new project and in the project selection dialog select Visual C# > Windows > Console Application. Name the new application “InteractiveClientApplication”.Add project referencesRight-click the new project and select Add Reference… We’ll need to add references to some standard .NET components and to the HPC SDK components. (You may need to add these references in several steps; if it’s easier, just add one at a time until you have all of the required references).In the .NET tab of the reference dialog, add references to “System.ServiceModel” and “System.Runtime.Serialization”. In the Browse tab of the reference dialog, locate the Windows HPC Server 2008 R2 SDK directory. By default this will be installed in “C:\Program Files\Microsoft HPC Pack 2008 R2 SDK”. Enter the “Bin” directory and add references to “Microsoft.Hpc.Scheduler.dll”; “Microsoft.Hpc.Scheduler.Properties.dll”; and“Microsoft.Hpc.Scheduler.Session.dll”.Add a reference to the Microsoft.Hpc.Excel library. This library is provided with the HPC 2008 R2 SDK and installed in the Global Assembly Cache (GAC), but it may not be visible in the .NET tab. To make this easier we’ve provided a copy of this library with the download files. In the Browse tab of the references dialog, locate the download files for this article and add a reference to “Microsoft.Hpc.Excel.dll”.You should now have the following references added to your project:Microsoft.Hpc.ExcelMicrosoft.Hpc.SchedulerMicrosoft.Hpc.Scheduler.PropertiesMicrosoft.Hpc.Scheduler.SessionSystem.Runtime.SerializationSystem.ServiceModelAdd directivesTo simplify the code, add directives for the library namespaces. At the top of the file add these directives:using Microsoft.Hpc.Scheduler;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;using Microsoft.Hpc.Excel;using Microsoft.Hpc.Excel.ExcelService;using System.Threading;If any of these directives generate Visual Studio errors, double-check the list of references added in the last step. The last directive – System.Threading – will be used for synchronization classes, which we’ll need because this is an asynchronous application.Write the application codeDouble-click the code file “Program.cs” in the new project to open the main method. We’ll add code in a few separate sections; at the end of this step we’ll include a full listing of the code file you can compare to your code.The first part of the code is the same as the first application, except that we don’t create the BrokerClient right away: static void Main(string[] args) { // change these values to match your cluster head node // and path to the test workbook (on a network share directory) const string headNode = "HeadNodeName"; const string workbookPath = @"\\path\to\share\directory\ExampleWorkbook1.xlsb"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo(headNode, "Microsoft.Hpc.Excel.ExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; info.Environments.Add("Microsoft.Hpc.Excel.WorkbookPath", workbookPath); // step two: initialize the session Console.WriteLine("Creating session and client"); Session session = Session.CreateSession(info);In this part we create the SessionStartInfo object with some specific parameters; create the cluster session; and create the client object. Make sure to update the value for the cluster head node and workbook path to match your settings.Next we add some variables which will be used in the calculation: // step three: create fields used to manage the interactive calculation int sentRequests = 0; int receivedResponses = 0; object lockObject = new object(); AutoResetEvent evt = new AutoResetEvent(false);The counters “sentRequests” and “receivedResponses” are used to manage individual sets of requests and responses, so we can monitor the progress of a running calculation. The AutoResetEvent object is used to signal the application that a set of requests has been completed. In the code, we’ll block on this object during a set of calculations – although the specific method will block, in a Windows application this will allow other threads to continue operating. The lockObject is used to synchronize response handlers – this is important because in an asynchronous application, multiple responses might arrive at the same time. If we use any global or class members in the results handler, we want to make sure that they are only used by one thread at a time. In the next block, we start an application loop. Every time the user presses a key, the application will send 100 requests and wait for responses: // step four: the interactive loop while (true) { Console.WriteLine("Press any key to send 100 requests, or press X to exit."); ConsoleKeyInfo cki = Console.ReadKey(false); if (cki.KeyChar == 'x' || cki.KeyChar == 'X') break; Console.WriteLine("Sending 100 requests..."); // create a client object for this batch of requests, using a unique ID string clientID = Guid.NewGuid().ToString(); BrokerClient<IExcelService> client = new BrokerClient<IExcelService>( clientID, session);At the beginning of the loop, we create the BrokerClient object. In this application, we’re creating a new BrokerClient object each time we send a batch of requests. You should use a single BrokerClient object to send one batch of requests, and then call BrokerClient.EndRequests to tell the scheduler to commit all outstanding requests. However after you call BrokerClient.EndRequests, you cannot re-use the same BrokerClient object to send another set of requests. Therefore in this code we create a new BrokerClient object each time we pass through the loop. To ensure that each BrokerClient is unique, we use the constructor that takes a clientID parameter; here, we’re using a GUID as the clientID to ensure each clientID is unique.Creating a new BrokerClient object does not consume a lot of system resources, and it does not affect the cluster session connection. As long as the session remains open, we can create as many BrokerClient objects as necessary for the application.After creating the BrokerClient object, we set the BrokerResponseHandler for the object: // set the response handler for the client. use an anonymous delegate method (in-line) client.SetResponseHandler((BrokerResponseHandler<CalculateResponse>) delegate(BrokerResponse<CalculateResponse> response) { lock (lockObject) { receivedResponses++; { int userData = response.GetUserData<int>(); WorkItem workItem = WorkItem.Deserialize(response.Result.CalculateResult); object obj = workItem.Get<object>(0); Console.WriteLine("Request: {0}, response: {1}", userData, obj); } if (receivedResponses == sentRequests) { evt.Set(); } } });BrokerResponseHandler is another generic type, so we use the interface type for the Excel calculation service. This is a delegate method, similar to event handlers in Windows applications. Whenever the client application receives a response from the cluster, the delegate method will be called. We use this method to handle responses.The first line of the delegate method uses the lockObject to synchronize method calls. That ensures that multiple threads will not execute the method simultaneously, and the code inside the block beginning with lock(lockObject) is thread-safe. Inside the synchronization block, we increment the receivedResponses counter. At the end of the block, we check this value to see if the complete set of responses (indicated by sentRequests) has been completed. If so, it will signal the event object evt (which will be used in sections that follow).Note: it’s not strictly necessary to synchronize this block just to increment the receivedResponses counter. You could also use the Interlocked.Increment API. However it’s a good idea to use synchronization if you are accessing any other global or class member variables. For example, if we were writing responses to a log file, we would want to ensure that only a single thread wrote to the file at once. Or if we were storing response values in an object that is not thread-safe, such as a List, we would want to ensure that access to the List was synchronized. In between, the code handles the response as in the first application: it simply writes responses to the console.Next the code clears out the request and response counters. It also resets the event object, used to signal when all responses have been received: // reset counters sentRequests = 0; receivedResponses = 0; // clear the event evt.Reset();In the last part of the loop, the code sends requests and cleans up the client object: // now send 100 requests for (int i = 0; i < 100; i++) { CalculateRequest request = new CalculateRequest(); request.macroName = "MyCalculationMacro"; request.inputs = WorkItem.Serialize(new WorkItem(new object[] { i })); client.SendRequest<CalculateRequest>(request, i); sentRequests++; } // call EndRequests to commit all requests client.EndRequests(); // and wait for the synchronization object evt.WaitOne(); // we're done with this client object, we can close it. we only need to // keep the session open to maintain the service connection. client.Close(); }Sending requests is identical to the first application: we create request objects, set parameters, and call BrokerClient.SendRequest. The only difference in this instance is that we increment the sentRequests counter for each request. After all requests have been sent, the application calls BrokerClient.EndRequests (to commit all outstanding requests), and then waits for the event object to be signaled. Remember that the event object will be signaled in the delegate method when all responses have been received (when the receivedResponses counter equals the sentRequests counter). While this code will block the thread, in a Windows application other threads could continue. In this example we simply block on the event object, but it’s also possible to use a timeout value and periodically run some other code while waiting for responses to arrive.After the event object has been signaled – indicating that all responses have been received – the code closes the BrokerClient object. Finally in the last section of code, if the user has pressed ‘x’ to exit the loop, the code cleans up the session and client objects and then exits. // done - clean up Console.WriteLine("Cleaning up"); client.Close(); session.Close(); Console.WriteLine("Press return to exit"); Console.ReadKey(); }The full main method should be implemented as follows:using System;using System.Collections.Generic;using System.Linq;using System.Text;using Microsoft.Hpc.Excel;using Microsoft.Hpc.Excel.ExcelService;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;using System.Threading;namespace InteractiveClientApplication{ class Program { static void Main(string[] args) { // change these values to match your cluster head node // and path to the test workbook (on a network share directory) const string headNode = "HeadNodeName"; const string workbookPath = @"\\path\to\share\directory\ExampleWorkbook1.xlsb"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo(headNode, "Microsoft.Hpc.Excel.ExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; info.Environments.Add("Microsoft.Hpc.Excel.WorkbookPath", workbookPath); // step two: initialize the session Console.WriteLine("Creating session"); Session session = Session.CreateSession(info); // step three: create fields used to manage the interactive calculation int sentRequests = 0; int receivedResponses = 0; object lockObject = new object(); AutoResetEvent evt = new AutoResetEvent(false); // step four: the interactive loop while (true) { Console.WriteLine("Press any key to send 100 requests, or press X to exit."); ConsoleKeyInfo cki = Console.ReadKey(false); if (cki.KeyChar == 'x' || cki.KeyChar == 'X') break; Console.WriteLine("Sending 100 requests..."); // create a client object for this batch of requests, using a unique ID string clientID = Guid.NewGuid().ToString(); BrokerClient<IExcelService> client = new BrokerClient<IExcelService>(clientID, session); // set the response handler for the client. use an anonymous delegate method (in-line) client.SetResponseHandler((BrokerResponseHandler<CalculateResponse>) delegate(BrokerResponse<CalculateResponse> response) { lock (lockObject) { receivedResponses++; { int userData = response.GetUserData<int>(); WorkItem workItem = WorkItem.Deserialize(response.Result.CalculateResult); object obj = workItem.Get<object>(0); Console.WriteLine("Request: {0}, response: {1}", userData, obj); } if (receivedResponses == sentRequests) { evt.Set(); } } }); // reset counters sentRequests = 0; receivedResponses = 0; // clear the event evt.Reset(); // now send 100 requests for (int i = 0; i < 100; i++) { CalculateRequest request = new CalculateRequest(); request.macroName = "MyCalculationMacro"; request.inputs = WorkItem.Serialize(new WorkItem(new object[] { i })); client.SendRequest<CalculateRequest>(request, i); sentRequests++; } // call EndRequests to commit all requests client.EndRequests(); // and wait for the synchronization object evt.WaitOne(); // we're done with this client object, we can close it. we only need to // keep the session open to maintain the service connection. client.Close(); } // done - clean up Console.WriteLine("Cleaning up"); session.Close(); Console.WriteLine("Press return to exit"); Console.ReadKey(); } }}Run the interactive applicationThis application requires a workbook installed on a share directory accessible to the compute nodes. If you’ve followed the previous sections and run cluster applications, you should already have this in place. If not, make sure that you have the workbook “ExampleWorkbook1.xlsb” accessible in a share directory, matching the value in the code file.Right-click the new project and select Set as Startup Project. Then run the debugger using the menu Debug > Start Debugging. If you receive any compilation errors, double-check the code against the list in the previous step. Double-check the references set in step 2 above. If you receive an exception when you run the application, check the section “Debugging” in the section for the first application. When the application runs, it asks you to press a key or press ‘x’ to exit. Press return to start the first set of requests. The code will send 100 requests. As responses are received, the results are written to the console. When all 100 responses have been received, the loop will start again and ask the user to press a key.Unlike the first application, in this model we can send requests multiple times in the same session. This pattern can be useful in any interactive application. When you run the application, you’ll notice an initial delay as the session is created. When you run the first set of requests, you’ll notice another slight delay. This second delay occurs as the service starts Excel and loads the workbook on all the compute nodes.The second time you send a batch of requests (without restarting the application), you’ll see that there is almost no delay. That’s because the session is already open, and the service has already loaded the workbook. We can skip those two steps and simply send requests. You can see that for an interactive application, using this model can have significant performance benefits. Although this is a console application, it’s important to understand that this model better supports Windows applications because using the asynchronous model will allow other application threads – such as user interface threads – to work while the cluster calculation is ongoing.Building an HPC/Excel Client Application: Fire and RecollectOne drawback to the calculation models presented above is that they require an interactive user session. You start the application, calculations run on the cluster compute nodes, and when results are complete they are delivered back to you on the desktop.Windows HPC Server 2008 R2 provides a new session type, “durable sessions”, which offers an alternative. Using durable sessions you can implement an application with a “fire-and-recollect” execution model. In this model, you send some number of requests to the cluster for calculation. After you have sent the calculation requests, you can end your user session (for example, you can log out of your workstation). At a later time, when you want to view the calculation results, you can reconnect to the cluster and retrieve the calculation responses.This execution model supports a number of new design patterns, many of which can be useful for Excel calculation. For example, if you have a number of long-running workbooks, you can send them to the cluster for calculation overnight, and collect the results the following morning. You can submit calculations from one location (the office) and collect results from another location (home), provided you can connect to the cluster domain.In this section we’ll build a new client application implementing the fire-and-recollect model.To build a fire-and-recollect application:Create a new projectYou can create this project in the same solution as the previous projects, or create a new solution. Create a new project and in the project selection dialog select Visual C# > Windows > Console Application. Name the new application “FireAndRecollectClientApplication”.Add project referencesRight-click the new project and select Add Reference… We’ll need to add references to some standard .NET components and to the HPC SDK components. (You may need to add these references in several steps; if it’s easier, just add one at a time until you have all of the required references).In the .NET tab of the reference dialog, add references to “System.ServiceModel” and “System.Runtime.Serialization”. In the Browse tab of the reference dialog, locate the Windows HPC Server 2008 R2 SDK directory. By default this will be installed in “C:\Program Files\Microsoft HPC Pack 2008 R2 SDK”. Enter the “Bin” directory and add references to “Microsoft.Hpc.Scheduler.dll”; “Microsoft.Hpc.Scheduler.Properties.dll”; and“Microsoft.Hpc.Scheduler.Session.dll”.Add a reference to the Microsoft.Hpc.Excel library. This library is provided with the HPC 2008 R2 SDK and installed in the Global Assembly Cache (GAC), but it may not be visible in the .NET tab. To make this easier we’ve provided a copy of this library with the download files. In the Browse tab of the references dialog, locate the download files for this article and add a reference to “Microsoft.Hpc.Excel.dll”.You should now have the following references added to your project:Microsoft.Hpc.Excel Microsoft.Hpc.SchedulerMicrosoft.Hpc.Scheduler.PropertiesMicrosoft.Hpc.Scheduler.SessionSystem.Runtime.SerializationSystem.ServiceModelAdd directivesTo simplify the code, add directives for the library namespaces. At the top of the file add these directives:using Microsoft.Hpc.Scheduler;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;using Microsoft.Hpc.Excel;using Microsoft.Hpc.Excel.ExcelService;If any of these directives generate Visual Studio errors, double-check the list of references added in the last step.Add an application setting for session IDIn this application, we will be using durable sessions to maintain calculation results without an active user session. Durable sessions are identified with a session ID. To reconnect to an existing session and collect results, we need to retain this session ID.You can store the session ID anywhere – in a file, in a database, on a piece of paper. For this application we’ll store the session ID in an application setting. That will simplify the application so you don’t have to remember the session ID.In the Visual Studio project window, right-click the new project and select Properties. Select the Settings tab on the left. Click the link to create a new settings file for the application.In the table that’s created, change the values of the first line to create a setting for the session ID. Set the values so they match this screen:This creates an integer value setting, with a scope of User (meaning there are individual settings files for each application user), and a default value of zero.Write the application codeMuch of the application code will resemble the earlier applications discussed in this article. We’ll use the simple method of sending requests and collecting responses, as in the first application. Unlike the first application, however, we’ll split the requests and responses into two sections.In this step we’ll add code in a few separate sections. At the end we’ll include the full code listing for the file so you can compare against your code.Double-click the code file Program.cs in the project window to open the code file. The first part of the code sets the SessionStartInfo parameters as with earlier applications: static void Main(string[] args) { // change these values to match your cluster head node // and path to the test workbook (on a network share directory) const string headNode = "HeadNodeName"; const string workbookPath = @"\\path\to\share\directory\ExampleWorkbook1.xlsb"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo(headNode, "Microsoft.Hpc.Excel.ExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; info.Environments.Add("Microsoft.Hpc.Excel.WorkbookPath", workbookPath);Next we check the setting, to see if there’s an existing session ID. Because the default value for the setting was zero, the first time you run this code there won’t be a session ID. // check if we have an existing session ID we can collect results from int lastSessionID = global::FireAndRecollectClientApplication.Properties.Settings.Default.lastSessionID;The next code block is used for that case, when there’s no session ID: if (lastSessionID == 0) { Console.WriteLine("No existing session found. Creating a new session for requests..."); Console.WriteLine("Creating session and client"); DurableSession session = DurableSession.CreateSession(info); Console.WriteLine("Created session with ID {0}", session.Id); // store the session ID in the settings so we can use it next time. // save the settings! global::FireAndRecollectClientApplication.Properties.Settings.Default.lastSessionID = session.Id; global::FireAndRecollectClientApplication.Properties.Settings.Default.Save(); BrokerClient<IExcelService> client = new BrokerClient<IExcelService>(session); Console.WriteLine("Sending requests..."); for (int i = 0; i < 10; i++) { Microsoft.Hpc.Excel.ExcelService.CalculateRequest request = new CalculateRequest(); request.macroName = "MyCalculationMacro"; request.inputs = WorkItem.Serialize(new WorkItem(new object[] { i })); client.SendRequest<CalculateRequest>(request, i); } client.EndRequests(); // close the client and session with the purge parameter set to false - // this tells the scheduler to preserve results from this calculation client.Close(false); session.Close(false); }The first part of this code block creates the session. Unlike earlier applications, here we use the “DurableSession” class. Except for the class name, the code is the same.Next we store the session ID in the application settings. That means the next time we run the application, it will find the old session ID.After storing the session ID, the code for sending requests is identical to code used in the first application. We create the client object; create a number of requests and send them to the cluster; and then call client.EndRequests to indicate we’re done.When we’re finished sending requests, we close the client object and session connections. In this case, we set the purge parameter to false – that tells the scheduler to preserve results from this calculation even after we’ve disconnected.For the fire-and-recollect application, that’s all we’re going to do when we create a new session. Remember that in this case, we want to close the application after sending the requests. You can completely exit the user session (log out of your desktop) without losing the calculation results.The second code block handles the case when you start the application and there is a session ID: else { Console.WriteLine("Found existing session ID {0}. Attempting to reconnect...", lastSessionID); DurableSession session = DurableSession.AttachSession(new SessionAttachInfo(headNode, lastSessionID)); Console.WriteLine("Connected to session"); // clear the session ID from the settings so we can send a new // request on the next time. save the settings! global::FireAndRecollectClientApplication.Properties.Settings.Default.lastSessionID = 0; global::FireAndRecollectClientApplication.Properties.Settings.Default.Save(); BrokerClient<IExcelService> client = new BrokerClient<IExcelService>(session); BrokerResponseEnumerator<CalculateResponse> responses = client.GetResponses<CalculateResponse>(); foreach (BrokerResponse<CalculateResponse> response in responses) { int userData = response.GetUserData<int>(); WorkItem workItem = WorkItem.Deserialize(response.Result.CalculateResult); object obj = workItem.Get<object>(0); Console.WriteLine("Request: {0}, response: {1}", userData, obj); } // we're done with this session so we can close it. Console.WriteLine("Cleaning up..."); client.Close(); session.Close(); }In this code block, instead of creating a new session we use the method DurableSession.AttachSession to connect to an existing session. The session ID, which was stored as an application setting, is used to identify the old session.The next lines remove the session ID from the application settings (and set that value to zero). That means that the next time you run the application, it will start again, creating a new session and sending requests (if that’s confusing, it will make more sense when you run the application in the next step).The next code creates the client object and collects responses from the cluster session. This is the “recollect” part, because we’re collecting results from an earlier calculation. The code for collecting results is identical to code in the first application.After collecting results, we close the session – now that we have the results we don’t need this session anymore. The complete code for the main method should look like this:using System;using System.Collections.Generic;using System.Linq;using System.Text;using Microsoft.Hpc.Excel;using Microsoft.Hpc.Excel.ExcelService;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;namespace FireAndRecollectClientApplication{ class Program { static void Main(string[] args) { // change these values to match your cluster head node // and path to the test workbook (on a network share directory) const string headNode = "HeadNodeName"; const string workbookPath = @"\\path\to\share\directory\ExampleWorkbook1.xlsb"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo(headNode, "Microsoft.Hpc.Excel.ExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; info.Environments.Add("Microsoft.Hpc.Excel.WorkbookPath", workbookPath); // check if we have an existing session ID we can collect results from int lastSessionID = global::FireAndRecollectClientApplication.Properties.Settings.Default.lastSessionID; if (lastSessionID == 0) { Console.WriteLine("No existing session found. Creating a new session for requests..."); Console.WriteLine("Creating session and client"); DurableSession session = DurableSession.CreateSession(info); Console.WriteLine("Created session with ID {0}", session.Id); // store the session ID in the settings so we can use it next time. // save the settings! global::FireAndRecollectClientApplication.Properties.Settings.Default.lastSessionID = session.Id; global::FireAndRecollectClientApplication.Properties.Settings.Default.Save(); BrokerClient<IExcelService> client = new BrokerClient<IExcelService>(session); Console.WriteLine("Sending requests..."); for (int i = 0; i < 10; i++) { Microsoft.Hpc.Excel.ExcelService.CalculateRequest request = new CalculateRequest(); request.macroName = "MyCalculationMacro"; request.inputs = WorkItem.Serialize(new WorkItem(new object[] { i })); client.SendRequest<CalculateRequest>(request, i); } client.EndRequests(); // close the client and session with the purge parameter set to false - // this tells the scheduler to preserve results from this calculation client.Close(false); session.Close(false); } else { Console.WriteLine("Found existing session ID {0}. Attempting to reconnect...", lastSessionID); DurableSession session = DurableSession.AttachSession(new SessionAttachInfo(headNode, lastSessionID)); Console.WriteLine("Connected to session"); // clear the session ID from the settings so we can send a new // request on the next time. save the settings! global::FireAndRecollectClientApplication.Properties.Settings.Default.lastSessionID = 0; global::FireAndRecollectClientApplication.Properties.Settings.Default.Save(); BrokerClient<IExcelService> client = new BrokerClient<IExcelService>(session); BrokerResponseEnumerator<CalculateResponse> responses = client.GetResponses<CalculateResponse>(); foreach (BrokerResponse<CalculateResponse> response in responses) { int userData = response.GetUserData<int>(); WorkItem workItem = WorkItem.Deserialize(response.Result.CalculateResult); object obj = workItem.Get<object>(0); Console.WriteLine("Request: {0}, response: {1}", userData, obj); } // we're done with this session so we can close it. Console.WriteLine("Cleaning up..."); client.Close(); session.Close(); } Console.WriteLine("Press return to exit"); Console.ReadKey(); } }}Run the fire-and-recollect applicationThis application requires a workbook installed on a share directory accessible to the compute nodes. If you’ve followed the previous sections and run cluster applications, you should already have this in place. If not, make sure that you have the workbook “ExampleWorkbook1.xlsb” accessible in a share directory, matching the value in the code file.Right-click the new project and select Set as Startup Project. Then run the debugger using the menu Debug > Start Debugging.If this is the first time you’ve run the application, it will tell you that it did not find an existing session. It will create a new session and report the session ID. Then it will send requests and complete. Press return to exit the application.If you look at the HPC Cluster Manager application, you’ll see that there’s a running job with your session ID in the Job Management tab.Now go back to Visual Studio, and run the debugger again. This time, the application will find the existing session ID, connect to the session, and report results. You should see a list of calculation results, as in earlier applications.Every time you run the application, it will either (a) create a session and send requests; or (b) connect to an existing session and collect results. The session ID is stored as an application setting, so you can log out of your workstation, log back in, and re-run the application, and you’ll get your results. Re-run the application a few times and you’ll see how this works.Review: Building HPC/Excel Client ApplicationsIn the previous sections we’ve walked through three basic patterns for constructing HPC/Excel client applications: the simple model, the interactive model, and the fire-and-recover model. How you select a model depends on the requirements of your application. To help you select the appropriate model, we’ll recap some of the very general strengths and weaknesses of each type:ModelStrengthsWeaknessesSimpleThe simple model is the easiest to construct and contains the least amount of code. A simple console application supports scripting or running on a fixed schedule, so you can integrate Excel calculations with your management and maintenance tools. The simple model is the least flexible. It blocks while waiting for results, so it’s not a good choice for Windows applications.InteractiveThe asynchronous application structure supports Windows applications and application plug-ins. The interactive model is the most complex to implement, and requires some understanding of event objects, synchronization and threading, and asynchronous code.Fire-And-RecollectThe fire-and-recollect model is the most robust because it does not require a user session: you can log out of your workstation and collect results at a later time. You can also collect results from different physical workstations, as long as you can connect to the cluster.The fire-and-recollect model is more complex and requires more code than the simple model. It adds complexity in the form of managing sessions, both for the user application and for the cluster itself. Using this model also requires more resources on the cluster.Of course these are very general considerations, and in many cases more than one of these models might support your application. The general structure is similar in each case, so it’s always possible to change from one to another as your needs change.Building a Custom HPC/Excel Calculation ServiceIn the previous sections of this article we’ve discussed writing HPC/Excel applications that use the default HPC/Excel cluster service. For many applications, the default service will be sufficient. The default service can call a VBA method and execute arbitrary code in VBA – allowing you to run virtually any Excel calculation or VBA code.In this section we’ll discuss writing a custom HPC/Excel service. When would you want to build your own HPC/Excel service? Using VBA in the workbook, the default service can do almost anything in Excel or on a Windows system.But there are still many good reasons to write a custom HPC/Excel service. These fall into a number of general categories:IsolationAlthough it’s possible to do almost anything from VBA within Excel, that’s not always what you want. Build a custom service to move complex or potentially dangerous actions – such as modifying the registry – into code and out of Excel. That prevents users from accidentally modifying the VBA code and making unexpected changes to the system.EncapsulationAlthough it’s possible to do almost anything from VBA, that’s not always ideal from a design and code management standpoint. You may prefer to write code in C# or – and use Excel only for calculation.Global BehaviorsIf you want to apply some behavior to every spreadsheet you calculate on the cluster, but you don’t want to modify every spreadsheet individually, write a custom service to perform common or general tasks before the calculation.Putting this general code in a custom service also means you have a single code base to update when you make changes, rather than updating every workbook.Application RequirementsFinally there are some things that you can’t do in Excel – or at least you can’t do before the workbook is loaded. For example, your workbook might require some Excel add-ins to be loaded before the workbook is opened (because they’re used in the Workbook.Open method). Or the workbook might require a password to open, and you don’t want to create an unprotected workbook.Or, for your purposes, you might simply not want to use VBA. In that case, you can build a custom service to interact with the workbook using C#, , or any other .NET framework language.In this example, we’ll build a service to use a workbook that does not include VBA. The workbook includes a basic options price calculation using the Black-Scholes model. Open the workbook “ExampleWorkbook2.xlsx” from the download files:If we wanted to use this workbook with the default HPC Services for Excel service, we could write a VBA method to fill in the cell values and collect the results. But it’s also possible to build a service that controls the workbook directly, without using VBA.Although this is a very simple example, it should illustrate that you have almost unlimited flexibility in how you interact with Excel on the cluster compute nodes when you build a custom service. To build an HPC/Excel service:Create a new projectIn Visual Studio, create a new project. From the project selection dialog, select Visual C# > WCF > WCF Service Library. Set the application name to “CustomHPCExcelService”. This will create a basic WCF service project.Note: you can use either the WCF Service Library or WCF Service Application project types. For the examples in this article, we’ll use the WCF Service Library. Select that project type to make the rest of the steps easier to follow.Update project propertiesIf you are using Visual Studio 2010, the default target framework for new projects is .NET Framework 4.0. To support Windows HPC Server 2008 R2, the target framework must be .NET Framework 3.5 or lower. Even if you’re using Visual Studio 2008, it’s a good idea to check this value.Right-click the new project and select Properties. In the Properties window, select the Application tab on the left. In the dropdown box for Target Framework, select “.NET Framework 3.5”. Visual Studio may ask you to reload the project; if so, click OK.Note: if you are using Visual Studio 2010, and you changed the target framework, you may see a warning about a reference to Microsoft.CSharp. It’s OK to ignore this warning; if you prefer, you can safely remove this reference from the references list in the project window.Add references to the projectAdd references for the Excel/HPC service files. In the Project Explorer window, right-click the new project and select Add Reference. In the references dialog, click the Browse tab. Then locate the HPC/Excel library.Add a reference to the Microsoft.Hpc.Excel library. This library is provided with the HPC 2008 R2 SDK and installed in the Global Assembly Cache (GAC), but it may not be visible in the .NET tab. To make this easier we’ve provided a copy of this library with the download files. In the Browse tab of the references dialog, locate the download files for this article and add a reference to “Microsoft.Hpc.Excel.dll”.Open the list of references in the project window. Right-click the reference for Microsoft.Hpc.Excel and select Properties to open the properties window. In the properties window, change the setting for CopyLocal to True. This will create a local copy of the library in the build directory, which will make it easier to deploy the service later.Create the service interfaceThe service interface defines the contract for use from external applications. Any methods you want to expose to client programs must be defined in the service interface.In the project window, double-click the file “IService1.cs” to open the service interface. First, change the name of the interface. In the code file, right-click on the name “IService1” at the top of the class declaration, and select Refactor > Rename. Change the name to “ICustomHPCExcelService” (Note that starts with a capital “I”, for Interface).Some methods are included automatically when the project is created. Delete these methods so you have a clean interface file. Delete the two methods “GetData” and “GetDataUsingContract”. Delete the second class defined in the file “CompositeType”. Make sure to delete the attributes as well. Your empty interface file should now look like this:namespace CustomHPCExcelService{ [ServiceContract] public interface ICustomHPCExcelService { }}Add two new methods to the interface. First we’ll add a test method, to validate the service, and then we’ll add a method to calculate the workbook.Add these two declarations to the interface:namespace CustomHPCExcelService{ [ServiceContract] public interface ICustomHPCExcelService { [OperationContract] int TestMethod(int inputValue); [OperationContract] double[] CalculateWorkbook(string workbookPath, double spotPrice, double strikePrice, double time, double sigma, double riskFree, double dividendYield); }}The test method is very simple: it takes one input value, an integer, and returns a single value. The CalculateWorkbook method is more complicated. In this method we have a number of parameters, representing first the Excel workbook (in the workbookPath parameter), and then values we’ll use in the calculation.If you reviewed the workbook at the beginning of this section, you saw that it was an option pricing model which included a number of parameters. The calculation method we’ll write here includes the same set of parameters – the service we will construct will insert these parameters into the Excel workbook, and retrieve the calculated prices.Also note that we’re returning an array – we want to return two values, the generated prices for call and put options based on the input parameters (again, look at the workbook: these are the values in rows 15 and 16).When you build a custom service, you have almost unlimited flexibility in the parameters you use and the return values you collect from the Excel calculation.Implement the interfaceNext we need to implement the interface – to write the code behind these new methods. In the project window, double-click the file “Service1.cs” to open the code file. First, rename the class. In the code file, right-click the name “Service1” in the class declaration and select Refactor > Rename. Change the name to “CustomHPCExcelService”. Remove the methods that were created automatically, so you have a clean class file.We’re going to use the Excel/HPC library (which we added as a reference in step 3), so add a declaration at the top of the file. Under the other “using” directives, addusing Microsoft.Hpc.Excel;You should now have an empty class file that looks like this:using System;using System.Collections.Generic;using System.Linq;using System.Runtime.Serialization;using System.ServiceModel;using System.Text;using Microsoft.Hpc.Excel;namespace CustomHPCExcelService{ public class CustomHPCExcelService : ICustomHPCExcelService { }}First we’ll implement the test method. We’ll use the test method when we build the first client application – it’s going to simply square the input value. Write the method as follows: public int TestMethod(int inputValue) { return inputValue * inputValue; }Next we’ll implement the Excel calculation method. This method will use the HPC/Excel library to start Excel, load a workbook, and run a calculation. To do that, we’ll use the class “ExcelDriver”, which is provided by the HPC/Excel library. The ExcelDriver class can create and manage instances of Excel running on HPC compute nodes. It exposes a number of methods to help you perform the most common Excel functions; and it provides a pointer to the Excel automation interface, so you have almost unlimited flexibility in what you can do with Excel.Declare a class member variable for the ExcelDriver class, at the top of the class file: ExcelDriver m_xlDriver = new ExcelDriver();Note that we’re initializing the member when the class is created. The ExcelDriver class does not create the Excel instance until it’s needed, so initializing the ExcelDriver instance here won’t affect performance.Next, write the calculation method: public double[] CalculateWorkbook(string workbookPath, double spotPrice, double strikePrice, double time, double sigma, double riskFree, double dividendYield) { double[] results = { 0, 0 }; m_xlDriver.OpenWorkbook(workbookPath); m_xlDriver.SetCellValue("C5", spotPrice.ToString()); m_xlDriver.SetCellValue("C6", strikePrice.ToString()); m_xlDriver.SetCellValue("C7", time.ToString()); m_xlDriver.SetCellValue("C8", sigma.ToString()); m_xlDriver.SetCellValue("C9", riskFree.ToString()); m_xlDriver.SetCellValue("C10", dividendYield.ToString()); results[0] = Convert.ToDouble(m_xlDriver.GetCellValue("C15")); results[1] = Convert.ToDouble(m_xlDriver.GetCellValue("C16")); return results; }This method uses the ExcelDriver object to control Excel and the Excel calculation. Recall that the ExcelDriver object was already initialized in the declaration, so it’s not necessary to declare it here. First we create an empty array for results. The next line calls the “OpenWorkbook” method on the ExcelDriver class. That tells the ExcelDriver to start Excel (if necessary – if it hasn’t already done so), and open the workbook we refer to in the workbookPath parameter.Next we use one of the functions included in the ExcelDriver class – SetCellValue – to insert values into the spreadsheet. There are a number of ways we can interact with Excel using the ExcelDriver class. The class includes methods for setting and retrieving values in spreadsheet cells, and a method for calling a VBA macro. The SetCellValue function takes a reference to an Excel range – here, we use cell addresses (“C5”, “C6”, etc.) but we could also use named ranges in the workbook.After we’ve set all the parameters, we know that Excel will recalculate automatically (just as if you typed these values into the spreadsheet). We can then collect the results values using another method of ExcelDriver: GetCellValue. GetCellValue works the same way as SetCellValue, using a reference to an Excel range.If you’re familiar with Excel automation, you can access the Excel object and manipulate it directly: use the App parameter of the ExcelDriver class.Note: if you do use the Excel automation object, you’ll need to add a reference to your service project. Add a reference to version 14 of the Excel interop libraries. You will need to install Excel 2010 on your development machine to have access to this library. The library is listed in the .NET tab of the references dialog, as Microsoft.Office.Interop.Excel version 14.0.0.0.Note that we’re not trapping exceptions in this method. If there is an exception, we want it to propagate to the calling code – in this case, that will be the HPC broker – so that it will be reported in the event log and to our client application.The complete code file should look like this:using System;using System.Collections.Generic;using System.Linq;using System.Runtime.Serialization;using System.ServiceModel;using System.Text;using Microsoft.Hpc.Excel;namespace CustomHPCExcelService{ public class CustomHPCExcelService : ICustomHPCExcelService { ExcelDriver m_xlDriver = new ExcelDriver(); public int TestMethod(int inputValue) { return inputValue * inputValue; } public double[] CalculateWorkbook(string workbookPath, double spotPrice, double strikePrice, double time, double sigma, double riskFree, double dividendYield) { double[] results = { 0, 0 }; m_xlDriver.OpenWorkbook(workbookPath); m_xlDriver.SetCellValue("C5", spotPrice.ToString()); m_xlDriver.SetCellValue("C6", strikePrice.ToString()); m_xlDriver.SetCellValue("C7", time.ToString()); m_xlDriver.SetCellValue("C8", sigma.ToString()); m_xlDriver.SetCellValue("C9", riskFree.ToString()); m_xlDriver.SetCellValue("C10", dividendYield.ToString()); results[0] = Convert.ToDouble(m_xlDriver.GetCellValue("C15")); results[1] = Convert.ToDouble(m_xlDriver.GetCellValue("C16")); return results; } }}Now rebuild the project. If you see any errors, double-check that your code matches the code listed above and try again.Create the service configuration fileThe service needs a configuration file to register with the HPC cluster. There are a number of settings in the configuration file, affecting things like transport layer buffers and security. Most of these are outside of the scope of this article. For now, use the service configuration template that’s provided with the download files. That configuration file provides all the options that you will need, and we’ll describe the specific changes you need to make. In the project window, right click the service project and select Add > Existing Item… and browse to the download directory. Select the file “ServiceTemplate.config” and click OK. Note: if you don’t see the file, check the “file types” drop-down in the Add Item dialog. Make sure this is set to “all files”, to allow you to see .config files.Change the name of this file. This is important because the name of this file is used when you connect to an HPC cluster and create a session with the service. Windows HPC Server 2008 R2 uses the names of configuration files to identify specific services. In the project window, right-click “ServiceTemplate.config” and select Rename. Name the file “CustomHPCExcelService.config”. Right-click the file again and select Properties. In the properties window, change the value of Copy to Output Directory to Copy if Newer. This will copy the configuration file to the build directory, which will make it easier to deploy the files later.In the service configuration file, the most important setting is the “ServiceRegistration” section. Double-click the CustomHPCExcelService.config in the project window to open the configuration file. Locate the ServiceRegistration section in the file (scroll down to find it, or search the file). Modify this section so it looks like this: <microsoft.Hpc.Session.ServiceRegistration> <service assembly="C:\HPCServices\CustomHPCExcelService.dll" contract="CustomHPCExcelService.ICustomHPCExcelService" type="CustomHPCExcelService.CustomHPCExcelService" includeExceptionDetailInFaults="true" maxConcurrentCalls="0" > </service> </microsoft.Hpc.Session.ServiceRegistration>The ServiceRegistration section controls how the HPC broker will locate and load the service library. The important attributes in this section areassemblyThe path to the library file, from the perspective of one of the compute nodes. In this example we will install the library on each compute node, in the directory C:\HPCServices. You can also install the library on a network share, using a UNC path.contractThe interface name, including any namespaces. The interface defines the contract – methods which client applications can call. In this case, the name has two parts: the namespace “CustomHPCExcelService”, and the interface name “ICustomHPCExcelService”.typeThe class name, including any namespaces. This is the assembly that the HPC service loader will attempt to load, so it must be a class which implements the interface described in the “contract” attribute. In this case, the name has two parts: the namespace “CustomHPCExcelService”, and the class name “CustomHPCExcelService”.(for more information on the rest of the attributes and elements in the configuration file, see the Windows HPC Server 2008 R2 documentation [link]).Create metadata and proxy code filesWhen we create an application that uses this new cluster service, we’ll need proxy code that describes the interface and the methods that are available. In this step we’ll add build events that automatically create the necessary metadata and proxy code files.Note: there are other ways to automatically create service references. For example, if you have a project in the same Visual Studio solution, you can often use “service discovery”. Here we’re creating metadata and proxy files using the SvcUtil tool, because it’s the most flexible way to do it – using SvcUtil will work even if you don’t have access to the original source project for the service.Right-click the project and select Properties. In the properties window, select the tab Build Events on the left. Click the Edit Post Build… button to edit the post-build step. Enter these lines:"C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\SvcUtil" "$(TargetPath)""C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\SvcUtil" /out:CustomHPCExcelServiceProxy.cs /a /mc *.xsd *.wsdl(Enter each command on a single line. In the code listing above the second command was too long, so it’s wrapped on the next line - but don’t enter a carriage return or newline).The first command creates metadata files from the new service. The metadata files are XML files which describe the service and the service methods (specifically, the SvcUtil tool will create XSD and WSDL files, which are themselves XML). The input to this command is the generated library (the .dll created when we build the project). The expression $(TargetPath) is a visual studio macro that inserts the full path to the generated dll file. We don’t need the metadata files except to create the proxy code file, in the second command.The second command uses the metadata files to create a proxy class that we can use when we build an application (in the next part of this article). The proxy class can be included in any project we develop in the future that uses this service. In the second command we use two arguments. The /a argument means “create asynchronous proxy methods”. That’s important for developing services which will run asynchronously or in parallel on an HPC cluster. The /mc argument means “generate message contract types”. That has to do with the way that client applications will utilize the proxy code, as you’ll see in the next part when we build an application that uses the service. The input to this second command are the metadata files – the XSD and WSDL files generated by the first command.By adding these commands to the build process, we know that the metadata and proxy file will be re-generated every time we rebuild the service. That means if you develop the service further – add, remove or change methods – you’ll always have up-to-date proxy code you can use to build client applications. You can run these same commands to create metadata and proxy code files for any WCF service library, even if you don’t have the code or the original project. As long as you have the .dll library, you can create the metadata and proxy files using SvcUtil.Deploying the HPC/Excel ServiceThere are two steps to deploying a service. Copy the service configuration file to the cluster head nodes, and copy the service library (the .dll) to the cluster. In the steps that follow, we will use clusrun – an application installed with Windows HPC Server 2008 R2 – to simplify installation and deployment.To deploy the service:Prepare a share directory on one of the cluster head nodesTo simplify deployment and installation, create a share directory on one of the cluster head nodes. You will need write access to this directory. We suggest creating a share directory because it makes deployment much simpler – after this directory is in place, you can do all the other steps from your desktop development machine using clusrun.Create a share directory on one of the head nodes – something like \\HeadNode\HPCTemp. Give your user account write access to this directory.Prepare to deploy the filesOpen an HPC PowerShell window and navigate to your project build directory. We will use PowerShell to deploy the files to the cluster head nodes and compute nodes.The download files include a batch script (.bat file) which includes all of the commands below. If you would like to use the batch script to deploy the files, open the file “DeployService.bat” and change the settings at the top indicating the scheduler name and your project directory. If you have installed your cluster using any non-standard settings, make changes as indicated in the batch file. The instructions below describe the individual steps – these are the same steps that the batch file uses to deploy the file.Copy the required files to the share directoryCopy the deployment files to the new share directory. There are three files we need: the service configuration file; the new service library .dll; and the HPC/Excel service library .dll file. If you followed the steps above, all three files should be in the project output directory (for the configuration file and the HPC/Excel library, we selected “copy to output directory” in the file properties).Do this with Windows Explorer, or from HPC PowerShell. In PowerShell, use the commands> copy CustomHPCExcelService.config "\\path\to\share\directory"> copy CustomHPCExcelService.dll "\\path\to\share\directory"> copy Microsoft.Hpc.Excel.dll "\\path\to\share\directory"Copy the service configuration file to the cluster head nodesThe service configuration file, which you created in the last part, needs to be installed on all cluster head nodes. You will need administrator permissions on the head nodes to install the file, because the service registration directory – where the file needs to reside – is installed by default under C:\Program Files.Use clusrun to copy the service configuration file to all the head nodes (replace HEADNODE with the name of your primary head node):> clusrun /scheduler:HEADNODE /nodegroup:headnodes copy "\\path\to\share\directory\CustomHPCExcelService.config" "C:\Program Files\Microsoft HPC Pack 2008 R2\Data\ServiceRegistration"(That’s a single line – don’t enter any carriage returns or newlines).In any clusrun command, use the option /scheduler: HEADNODE to identify your primary cluster head node. Use the option /nodegroup to describe a set of nodes. For this command, the configuration file needs to be installed on all head nodes – primary and backup. We can use the built-in node group “headnodes” to run the command on all cluster head nodes.Create a directory on the compute nodesUse clusrun to create a service directory on all compute nodes:> clusrun /scheduler:HEADNODE /nodegroup:computenodes mkdir "C:\HPCServices"Here we use the built-in node group “computenodes” to select all compute nodes; then we create a directory on each compute node. The directory used here, C:\HPCServices, matches the directory specified in the service configuration file.Copy library files to the compute nodesUse clusrun to copy the service library and dependent libraries to the new directory on the compute nodes:> clusrun /scheduler:HEADNODE /nodegroup:computenodes copy "\\path\to\share\directory\CustomHPCExcelService.dll" "C:\HPCServices"> clusrun /scheduler:HEADNODE /nodegroup:computenodes copy "\\path\to\share\directory\Microsoft.Hpc.Excel.dll" "C:\HPCServices"(Again these are two commands, each a single line. Don’t enter carriage returns in the commands).Building an HPC/Excel Client Application for the New ServiceIn the last part, we built and deployed a custom service utilizing HPC Services for Excel. The service exposes a number of methods which can be used by client applications to execute Excel calculations on the HPC cluster.In this part, we’ll build a client application which can use that service. If you read the first parts of this article, describing custom applications which use the default HPC/Excel service, much of this may be familiar. In fact the process is exactly the same, with just a few changes reflecting the custom service.The first part of this article described several different application patterns for client programs utilizing Excel calculation services on the cluster. In this part, we’ll just address the most basic application pattern: running a fixed set of calculations in the “simple” application model.To build an HPC/Excel client application for the new service:Create a new projectIn Visual Studio, create a new project. You can create this project in the same solution as the service you created in the last part.Create a new project. In the project selection dialog, choose Visual C# > Windows > Console Application. Set the project name to “CustomServiceClientApplication” or something similar.Add project referencesRight-click the new project and select Add Reference… We’ll need to add references to some standard .NET components and to the HPC SDK components. (You may need to add these references in several steps; if it’s easier, just add one at a time until you have all of the required references).In the .NET tab of the reference dialog, add references to “System.ServiceModel” and “System.Runtime.Serialization”. In the Browse tab of the reference dialog, locate the Windows HPC Server 2008 R2 SDK directory. By default this will be installed into “C:\Program Files\Microsoft HPC Pack 2008 R2 SDK”. Enter the “Bin” directory and add references to “Microsoft.Hpc.Scheduler.dll”; “Microsoft.Hpc.Scheduler.Properties.dll”; and“Microsoft.Hpc.Scheduler.Session.dll”.You should now have the following references added to your project:Microsoft.Hpc.SchedulerMicrosoft.Hpc.Scheduler.PropertiesMicrosoft.Hpc.Scheduler.SessionSystem.Runtime.SerializationSystem.ServiceModelAdd the service proxy classIn the previous part, when you built the custom HPC/Excel service, you generated metadata and proxy class files for use in building client applications. We’ll now use the proxy class file to provide method and data types for use in connecting to the service.Right-click the new project and select Add > Existing Item… In the dialog, locate the build directory for the service library you created in the last part. Select the file “CustomHPCExcelServiceProxy.cs”, and in the dialog click Add as Link (from the drop-down box next to the Add button).Note: if you click Add to add the file, Visual Studio will copy the file to your new project. We use Add as Link instead, which leaves the file in its original location. This can be useful if you are doing ongoing development. If you make any changes to the service project, when you rebuild it the proxy file will be re-created by SvcUtil. Using Add as Link ensures that the client project will always have the most recent service definitions in the proxy file.Add directivesTo simplify the code structure, add directives at the top of the file for the HPC namespaces:using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;using Microsoft.Hpc.Scheduler;Write code to use the service “test” methodIn this example we’ll write a very simple client program that uses the new HPC/Excel service. In this code, the parameters (head node name, unit type, etc) are hard-coded in the file. You can use the models in the first parts of this article to implement more complex application patterns and use command-line arguments for the various parameters. In this part we want to illustrate only what you need to do differently to use the new service.Add the following code to the “main” method of the new project. We’ll enter the code in sections and describe each section in turn; at the end of this section we’ll provide the completed main method.First set some general parameters, and create the “SessionStartInfo” instance which contains the cluster parameters for this calculation: // change this value to match your cluster head node const string headNode = "HeadNodeName"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo(headNode, "CustomHPCExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true;Change the value for headNode to match your settings. We create the SessionStartInfo object with the name of the cluster head node and the name of the service. The service name must match the name of the configuration file you deployed to the cluster in the last section. The name we suggested was “CustomHPCExcelService.config”; if you used a different name, change the service name in the SessionStartInfo constructor.After creating the SessionStartInfo object we set some general parameters for the cluster calculation: unit type (core), minimum and maximum units, and security.Next create the HPC session and an instance of the client proxy object. Creating the session just requires the SessionStartInfo object. To create the client proxy object, use the generic type “BrokerClient” with the interface type of the service. (here, the interface is ICustomHPCExcelService). The interface type is provided by the client proxy code which we added to the project in step 3, above. // step two: initialize the session and create the client object Console.WriteLine("Creating session and client"); Session session = Session.CreateSession(info); BrokerClient<ICustomHPCExcelService> client = new BrokerClient<ICustomHPCExcelService>(session);The client proxy code provides the interface definition used to create the BrokerClient object, and some message class definitions which we can use to send requests to the service. Now create requests to call functions in the cluster service: // step three: send requests for calculation Console.WriteLine("Sending requests..."); for (int i = 0; i < 10; i++) { TestMethodRequest request = new TestMethodRequest(); request.inputValue = i; client.SendRequest<TestMethodRequest>(request, i); } client.EndRequests();This code first creates 10 service requests, using the TestMethodRequest type. TestMethodRequest is a class defined in the client proxy library, which represents a request against the “TestMethod” function in the service. The proxy code defines this object and changes the function parameters to class members; so we set the “inputValue” parameter by setting the class member variable in the TestMethodRequest object.Note: you can also set the inputValue parameter in the constructor for the TestMethodRequest object. Here we use the class member just for clarity. The next line sends this request to the cluster using the client object. Here, in addition to the TestMethodRequest class, we pass the “UserData” parameter (the second parameter to client.SendRequest). We’ll use that parameter to match requests and responses when responses come back from the service. The service running on the HPC cluster runs in parallel, and runs asynchronously. That means that one single request may be calculated on any available compute node. Depending on the speed of the compute nodes, requests may be returned in a different order than the order in which we sent them. Using the “UserData” parameter makes it easier to match requests and responses.After the loop, we call client.EndRequests, which tells the service that we are finished sending requests.Next we wait for responses from the service: // step four: retrieve responses from the cluster Console.WriteLine("Waiting for responses..."); BrokerResponseEnumerator<TestMethodResponse> responses = client.GetResponses<TestMethodResponse>(); foreach (BrokerResponse<TestMethodResponse> response in responses) { int userData = response.GetUserData<int>(); int result = response.Result.TestMethodResult; Console.WriteLine("Request: {0}, response: {1}", userData, result); }The BrokerClient object provides an enumerator of responses; we can loop over responses as they arrive. Each response is provided as an instance of the generic BrokerResponse class, with the type TestMethodResponse. Like the TestMethodRequest object, TestMethodResponse is created in the client proxy code to represent the results of a method calculation, and provides the function return value as the “Result” member.In the loop, we retrieve the “UserData” object from the response; this is the same “UserData” object that we used when we sent the request to the service.Again “UserData” can be used to match requests and responses. The test method implemented in this service simply squares the input value: so in the results, we will expect that the output (the result of the function) will be the square of the input value; and the input value is stored as the “UserData” field. The last section of code cleans up the session and client objects: // done - clean up Console.WriteLine("Cleaning up"); client.Close(); session.Close(); Console.WriteLine("Press return to exit"); Console.ReadKey(); }The complete main method, including all the code added above, should look like this:using System;using System.Collections.Generic;using System.Linq;using System.Text;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;namespace CustomServiceClientApplication{ class Program { static void Main(string[] args) { // change this value to match your cluster head node // and path to the test workbook (on a network share directory) const string headNode = "HeadNodeName"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo(headNode, "CustomHPCExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; // step two: initialize the session and create the client object Console.WriteLine("Creating session and client"); Session session = Session.CreateSession(info); BrokerClient<ICustomHPCExcelService> client = new BrokerClient<ICustomHPCExcelService>(session); // step three: send requests for calculation Console.WriteLine("Sending requests..."); for (int i = 0; i < 10; i++) { TestMethodRequest request = new TestMethodRequest(); request.inputValue = i; client.SendRequest<TestMethodRequest>(request, i); } client.EndRequests(); // step four: retrieve responses from the cluster Console.WriteLine("Waiting for responses..."); BrokerResponseEnumerator<TestMethodResponse> responses = client.GetResponses<TestMethodResponse>(); foreach (BrokerResponse<TestMethodResponse> response in responses) { int userData = response.GetUserData<int>(); int result = response.Result.TestMethodResult; Console.WriteLine("Request: {0}, response: {1}", userData, result); } // done - clean up Console.WriteLine("Cleaning up"); client.Close(); session.Close(); Console.WriteLine("Press return to exit"); Console.ReadKey(); } }}Running the client applicationYou should now be able to run the client application. Right-click the project and select Set as Startup Project; then run the application in the debugger from the menu Debug > Start Debugging. If everything works, you should see results similar toDepending on your cluster, the results may not be in the same order. In most cases the results will not be in numerical order, even though we sent the requests in numerical order. This is because different compute nodes may run the calculation faster or slower than others. We use the “UserData” parameter in the request and response objects to match requests to responses, and in the output of the application – whatever order the results are presented – the “response” value displayed should always be the square of the “request” value.Debugging the applicationIf the project fails to build, check the Visual Studio output window for errors. Make sure that your code matches the example code listed above in section 5. Also make sure that you have added the references listed in section 2, above, and that you have added the client proxy library as described in section 3. If the project builds but you see an exception in the output window or in the Visual Studio debugger, you may have a problem with the cluster configuration or with the deployed service. Open the HPC Cluster Manager application and click the Job Management tab in the lower-left to view cluster jobs. Select All Jobs in the tree view at the top-left. You should see an entry for the job you just ran, listed as “CustomHPCExcelService”. If you don’t see this entry, double-check that your cluster settings are correct (the head node name, at the top of the main method); and verify that you have permissions to run jobs on the cluster (in the HPC Cluster Manager, select the Configuration tab and click Add or Remove Users).If the job is listed but the state column shows “Failed”, double-click the job to look for any errors. The most likely error is that the service has not been deployed properly. Take the following steps to verify the service is deployed properly:Configuration file presentOn the cluster head node, verify that the file “CustomHPCExcelService.config” is correctly installed in the directory “C:\Program Files\Microsoft HPC Pack 2008 R2\Data\ServiceRegistration”. Verify the name of this file – it must match the code in the main file describing the service name when the SessionStartInfo object is created.Configuration file correctOn the cluster head node, open the configuration file in notepad. Locate the ServiceRegistration section and verify that it provides the correct values for assembly, contract, and type. See the previous part on building the service for the correct values.The “type” value must describe the class name, including any namespaces. The “contract” value must describe the interface name, including any namespaces. The “assembly” value must provide a path to the library file – the service .dll file – from the perspective of the compute nodes (not the head node). Service libraries installedOn the cluster compute nodes, open the directory where the service was deployed. If you followed the steps in the previous section on deploying the service, this will be the directory C:\HPCServices on each compute node.That directory must exist and should contain two files: the service library CustomHPCExcelService.dll, and the dependency file Microsoft.Hpc.Excel.dll.Verify that this directory matches the value of the “assembly” attribute in the configuration file.Note: if you encounter an exception and then stop the debugger, the service may continue to run on the cluster (because you stopped the program before it could close the service connection). Open the HPC Cluster Manager application and check the list of running jobs. If you see a running job for the service “CustomHPCExcelService”, right-click the job and select Cancel Job.Calling the Excel calculation methodIn the last step, we wrote the client application to call the service “test” method, which squared the input value. That function does not use Excel to run the calculation, but it’s useful to start with a simple method to verify that the service and the client application are working correctly. In this part we’ll modify the client application to use the Excel calculation function in the service.In the previous step we used the proxy class to create request and response objects matching the test method in the service. The proxy class provides similar classes for all methods in the interface, so we only need to change the request and response sections of the code. The sections which create and manage the cluster connection and client object can stay the same.Change the code for steps 3 and 4 so it reads: // step three: send requests for calculation Console.WriteLine("Sending requests..."); for (int i = 1; i <= 12; i++) { string workbookPath = @"\\path\to\share\directory\ExampleWorkbook2.xlsx"; CalculateWorkbookRequest request = new CalculateWorkbookRequest( workbookPath, 45, 50, i / 12.0, .30, .03, .02); client.SendRequest<CalculateWorkbookRequest>(request, i); } client.EndRequests(); // step four: retrieve responses from the cluster Console.WriteLine("Waiting for responses..."); BrokerResponseEnumerator<CalculateWorkbookResponse> responses = client.GetResponses<CalculateWorkbookResponse>(); foreach (BrokerResponse<CalculateWorkbookResponse> response in responses) { int userData = response.GetUserData<int>(); double[] results = response.Result.CalculateWorkbookResult; Console.WriteLine("Request: {0:00}, call: {1:00.00}, put: {2:00.00}", userData, results[0], results[1]); }Here we’re sending requests with the type CalculateWorkbookRequest. This represents the CalculateWorkbook method in the service interface.When we wrote the service interface, the CalculateWorkbook method included a number of parameters: first the workbook path, representing the Excel workbook used in calculation; and then the various parameters used in the pricing calculation.When we call a method on the service from this client application, we use the CalculateWorkbookRequest class to represent a method call. In the constructor to that class we pass in the parameters we would use if we were calling the method directly.Here we’re hard-coding the workbook path in the code file, but in most applications the workbook path would be a parameter. Make sure to change this path to match a share directory accessible by the compute nodes; if you created a share directory in the above section on deploying the cluster service, you can re-use that directory.To test the workbook, we’re going to run 12 individual calculations, varying the time to expiration. Again if you look at the workbook, you’ll see that it takes values for the various option parameters, one of which is time to expiration. Time to expiration is defined in years. Here in the application code, we’re sending 12 requests to the service: each one represents a different expiration time, from 1/12 year to 12/12 year (or in practical terms, one month, two months, and so on, up to one year). That’s the “time” parameter, which we set with the value “i / 12.0”.When responses arrive back from the cluster service, they’re represented by the class type CalculateWorkbookResponse. Because our service function returned two double values, as an array, we can get the result value as a type “double[]”. Then we write the results to the console, getting the two values out of that array.Everything else is the same as when we wrote code using the test method. The complete code listing for the updated application should look like this:using System;using System.Collections.Generic;using System.Linq;using System.Text;using Microsoft.Hpc.Scheduler.Properties;using Microsoft.Hpc.Scheduler.Session;namespace CustomServiceClientApplication{ class Program { static void Main(string[] args) { // change this value to match your cluster head node const string headNode = "HeadNodeName"; // step one: create the session start info with parameters for // the cluster session Console.WriteLine("Starting..."); SessionStartInfo info = new SessionStartInfo(headNode, "CustomHPCExcelService"); info.ResourceUnitType = JobUnitType.Core; info.MinimumUnits = 1; info.MaximumUnits = 128; info.Secure = true; // step two: initialize the session and create the client object Console.WriteLine("Creating session and client"); Session session = Session.CreateSession(info); BrokerClient<ICustomHPCExcelService> client = new BrokerClient<ICustomHPCExcelService>(session); // step three: send requests for calculation Console.WriteLine("Sending requests..."); for (int i = 1; i <= 12; i++) { string workbookPath = @"\\path\to\share\directory\ExampleWorkbook2.xlsx"; CalculateWorkbookRequest request = new CalculateWorkbookRequest( workbookPath, 45, 50, i / 12.0, .30, .03, .02); client.SendRequest<CalculateWorkbookRequest>(request, i); } client.EndRequests(); // step four: retrieve responses from the cluster Console.WriteLine("Waiting for responses..."); BrokerResponseEnumerator<CalculateWorkbookResponse> responses = client.GetResponses<CalculateWorkbookResponse>(); foreach (BrokerResponse<CalculateWorkbookResponse> response in responses) { int userData = response.GetUserData<int>(); double[] results = response.Result.CalculateWorkbookResult; Console.WriteLine("Request: {0:00}, call: {1:00.00}, put: {2:00.00}", userData, results[0], results[1]); } // done - clean up Console.WriteLine("Cleaning up"); client.Close(); session.Close(); Console.WriteLine("Press return to exit"); Console.ReadKey(); } }}Running the client applicationFirst copy the test workbook from the download files to the share directory, where cluster compute nodes can access it. Copy the workbook “ExampleWorkbook2.xlsb” to the share directory so it matches the path you entered in the code in the last pile the modified code and run the debugger. If everything works, you should see results similar to the following:Again these may come back in any order; we use the “UserData” object to match requests and results. The test workbook simply cubes the input value, so for each result the “response” value should be the cube of the “request” value.If you have any errors, you will see an exception in the Visual Studio debugger or in the application window. Check the exception information to figure out the error. If you were able to run the test method in step 6, above, any errors you encounter now most likely involve Excel. Ensure that the workbook is stored in a share directory matching the “workbookPath” value in the code, and that the workbook can be loaded by Excel on the compute nodes. If you have not already done so, double-check the Excel installation on the cluster using the diagnostic test described in the section “Before you Start”, above.Next stepsIn this section we’ve constructed a basic client application to utilize the custom HPC/Excel service. This uses the simplest application pattern, sending a batch of requests and waiting for responses. You can use any of the patterns described in the first sections of this article with a custom HPC/Excel service, including the interactive and “fire and recollect” models. Most of the application code will be the same as that presented in the earlier sections. To use your custom service, make the following changes to the original applications:Add the client proxy library you created when you generated the service library. Follow the instructions in step 3 (“Add the service proxy class”) of this section.Change the service name when you create the SessionStartInfo object to match your custom service. Remember that the name of the service must match the name of the service configuration file installed on the cluster head nodes. For the custom service, we used the service name “CustomHPCExcelService”. In the applications which used the default HPC/Excel service, we used the service name "".Change the interfaces used in the generic BrokerClient and BrokerRequest objects and methods. Modify the request and response objects to match your custom service methods and the class objects described in the proxy library.With those changes, you can use your custom HPC/Excel service with the models presented in the first sections of this article. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download