It's been a while since I posted anything on HPC-related stuff. This is broadly down to whatever I have going on at the moment.

But a couple of interesting things:

• WindowsHPC.Net has been revamped. Very nice.
• Via Savas - this is an interesting post from Ian Foster on the now, the soon and the later of Grid and Cloud computing and the things we need to think about.
• Talking of Grid computing - take a look at this Silverlight Grid computing article. Lovely.

You can find the details on the all new Windows HPC Server 2008 here.

It's exciting to see this move into a second iteration and build on some of the capability delivered in CCS Edition 2003. Main big features appear to be:

• New high-speed networking support
• Highly efficient and scalable cluster management tools
• Support for partners’ clustered file systems
• Advanced failover support
• Service-oriented architecture (SOA) job scheduler expanding the HPC market

There are 25 case studies on the site, and I've contributed to 2 of them here and here.

On the subject of HPC, Joe Hummel of Pluralsight (remember the CCS training?) has a great blog post here on using Mac Minis as a portable cluster. Brilliant.

I note with interest that Pluralsight are offering a 5-day training course in HPC (specifically with CCS, though I imagine that the MPI stuff is suitable across environments).

Details at http://www.pluralsight.com/courses/AppliedComputeClusterServer.aspx

Course Description:

Five intensive days developing parallel applications that take advantage of high-performance clusters and Microsoft Compute Cluster Server .

This course provides in-depth coverage of high-performance application development and design using Microsoft's Compute Cluster Server (CCS). Intended for both commercial developers and academic researchers, our goal is to quickly bring attendees up to speed on the design and development of 64-bit parallel codes. Attendees learn to take full advantage of modern high-performance clusters, from running simple parametric sweeps to developing multi-core, multi-node apps with OpenMP and MSMPI. Discussion includes numerous practical considerations, including data distribution and collection, debugging, porting, performance tuning, visualization, and customizing the client-side submission process. The course follows a lecture/lab format, providing every attendee with hands-on development experience using a CCS cluster. The programming language used is C++.

Highlights

· Introduction to Compute Cluster Server: setup, usage and customization

· Parallel programming — theory and practice

· Parallelizing existing applications for parametric sweep

· Using OpenMP to develop multi-threaded, shared-memory codes

· Using MPI to develop multi-process, distributed-memory codes

· Advanced MPI programming: I/O, processes, groups, and topologies

· Excel Services for CCS

· Integrating SQL Server databases into high-performance applications

· Compiling, tracing and debugging with Microsoft Visual Studio 2005 and Visual C++

· Porting considerations and performance tuning

· Patterns for parallel applications

· Hands-on lab exercises with your very own CCS cluster

I've been working on an article, along with Simon Cox, for the Architecture Journal which I'm pleased to say is now published in issue 11.

You can download here if you don't have the paper copy.

The article covers the architecture we've constructed for using WF (as previously described) as an application layer for controlling HPC tasks but also looks at the idea of High "Productivity" Computing through consideration of the overall process and value stream rather than simply the power of the algorithms or tin to run them on.

I'll be really happy to answer questions, or expand on the thoughts if I can, so please let me know what you think.

Technorati Tags: HPC - WF - Architecture

If you're attending IT Forum in Barcelona, I'll be speaking - along with my colleague Giovanni Marchetti - on High Performance Computing and our experience with Windows Workflow.

The session is INFCT04 - Notes from the field: deploying and managing Microsoft high-performance computing (HPC) clusters

The work has been mentioned elsewhere in this blog - here - although we've refined some aspects since then (I'm writing a paper on the overall architecture). As it's a Chalk n Talk style session - there'll be plenty of opportunity to ask questions. Please join us if you're interested in this area.

It's been a couple of months since I had a look at this, but I said I'd make some comment on some concept work with WF and CCS so here it is...

You can see a video at the Microsoft site which was used as part of the launch for CCS (if you look really closely you can see me waving my hands about in it...)

So, aside from the specifics of enabling computational fluid dynamics solves on CCS, some of the interesting technical architecture was around the use of WF to access the job scheduler to complete an overall solve workflow.

As it turned out, WF was very well suited to the purpose. So what did we do? Quite a lot.

The things we were trying to achieve were:

• Harnessing the job scheduler of CCS from a front-end application.
• Creating workflows of multiple CCS jobs of multiple types.
• Create an authoring and publishing experience for Workflow generation.
• Track and persist long running workflows (BIG computations)

To do this, we did the following:

• Wrapped the XML API for CCS with a WCF layer.
• Created an NT Service that would be used to host the WF runtime and enabled this service with WCF.
• Create a DB to hold XAML descriptions of workflow templates.
• Created a Winforms application that rehosted the WF designer from Visual Studio (with limited access) allowing Workflow 'users' to load a workflow, change parameters and execute the workflow.
• Used WCF and the XAML definitions to compile and execute the workflows on-the-fly at the NT Service instead of the client application
• Used and extended the out-of-the-box WF tracking and persistence service to provide a complete trail of all previous and current workflow states.
• Created a new 'source control provider' inside Visual Studio to be used with the WF design surface allowing WF XAML templates to be stored in the template database and provide a simple publishing experience.

So we have a straightforward tiered design, though handling a very specialised engineering process. The application (or framework really) allows a very simple experience across what is a complex workflow, and provides a number of strategies for introducing data visualisation, optimisation loops, and other business rules and features.

In this instance, we built our own host (the NT service) for WF in order to keep things relatively simple (our Proof of Concept build periods at the MTC are only 3 weeks), but we could have chosen Sharepoint Server which could be worthwhile in a production architecture.

The real strength was the flexibility of WF for hosting, tracking and persistence as aspects of its capability and then the great model for building hierarchies and libraries of custom activities for the construction of workflows to harness CCS.

The unsung hero was CCS itself, which worked well with pre-existing code, and we even recompiled this with the MS MPI stack. Treating CCS as a 'service gateway' - like a database - made a lot of sense as it could be called upon when needed rather than being embedded in the middle of the solution.

After a bit of CCS and a bit of WF, my latest project is on WPF giving me a chance to use the tools and start to understand how to architect real solutions around the libraries.

I don't think the flexibility of WPF can be underestimated - with all of the increasing possibility of templating, styles and binding and therefore how these all interact. There's also the split of XAML and C# to consider. So from a developer perspective, you can achieve all sorts of things - SeeWindowsVista - and from an architectural perspective there needs to be careful consideration of how to assemble a good WPF solution. I'll be pondering this over the next couple of months.

Given that I can't draw a straight line with a ruler, it's also very handy that I'm working with Martin Grayson, who is a resident expert and blogger on the subject of WPF.

There's a nice article on The Register on High Performance Computing mentioning Prof. Simon Cox at Microsoft HPC Institute at the University of Southampton.

I've just finished a Proof of Concept project working with Simon and his team in the engineering space, harnessing WF and Cluster Compute Server to enhance the value stream and capability of an engineering team (more on that later). It was great to give WF a real workout, and I guess the immediate impact was quite how powerful and feature-rich WF was from 'out of the box' which gave us a real head start.

I'll blog some more about this architecture when I'm able to.