Ant's Blog

Logging with Message Broker

2013-05-13T16:35:00.001+01:00

Message Broker has several logging options all of which are valuable in different circumstances. First of all there is the new kid on the block: record and replay. This feature is designed to capture messages as they flow through Message Broker and store them in a database. Key fields can be selected and pulled out for searching and filtering. The Message Broker web application provides a nice easy to use UI for accessing the stored messages and replaying them if required. This is the ideal tool for auditing and long term storage.

Activity log is another very useful facility. It provides a high level view of key activities as they occur in Message Broker (file opened, message written, database updated etc). The key with activity log is that it is left on all the time. It consumes very little overhead and ensures there is always some diagnostics information available.

User trace is text based and a much lower level view of what Message Broker is up to as it processes your messages. User trace supports several levels of verbosity. There is a trace node provided in Message Broker which adds your information to the Message Broker trace. Trace can be controlled with the mqsichangetrace command and siblings (or equally through the Message Broker Java API and Message Broker Explorer). There is also service trace but that is really only intended for IBM to diagnose product issues.

So there is something of a gap for a more flexible way to log output from message flows. With that in mind I've knocked up a logging node which you can use to instrument your message flows. The node is configurable and outputs its messages through the most excellent Log4J logging framework.

You can read about what the node does in this PDF. The node works on Message Broker v8 Fix Pack 2 and later releases. Essentially the node writes messages to Log4J. The messages can contain anything including pieces of the actual message body being processing (for example, from an HL7 message). The output messages are collected into files according to their message flow, execution group and broker. You can switch logging on or off at any level at any time using a simple configuration file which is automatically reloaded. The node itself packaged as a JAR file here along with the source code here.

Healthcare Analytics with Message Broker

2013-04-07T17:18:00.001+01:00

One feature I would like to call out in the new version of the Healthcare Connectivity Pack is our healthcare analytics capability. This feature makes it incredibly easy to find, search and extract meaningful data from a range of healthcare data sources (HL7 v2, DICOM and CDA). The tooling can do this because it understands these healthcare formats, not just at a syntactic level but the semantics as well. This knowledge of a standard is encapsulated in what we call a Data Analysis Profile (DAP).

For example, in the case of clinical documents, the CDA DAP understands the HL7 v3 RIM and structural attributes (such as templateID and root). With this understanding the tooling can pick apart clinical documents and show you the meaning of different sections (medications, procedures etc). This is just the tip of the iceberg, the tooling makes it a breeze to do really complex work with clinical documents.

Another example are HL7 v2 observation results, the HL7 DAP understands where the coding system is stored in the OBX segment. Armed with this information the tooling deciphers LOINC codes to present user friendly display names when you are working with your data. In fact you can plug in your own bespoke coding systems as well (a common requirement in providers).

Understanding and working with these healthcare standards is one big step forward, but the tooling goes way further. The tooling builds for you the Message Broker solution to validate inbound messages and extract the data you want. The data can be extracted into a simpler XML message format. This is the ideal approach if you want to send some extract of the healthcare data to a downstream application. Alternatively, the tooling can extract the data directly into database tables. In fact, the tooling will even design you the database table layout, key selection and everything else!

If you are interested to learn more, take a look through these slides I have posted.

Medical Imaging (DICOM) with Message Broker

2013-04-07T16:47:00.000+01:00

One of the key things we focus on with the Healthcare Connectivity Pack is to have a single healthcare ESB which can work with a wide range of protocols, standards, transports and formats. This focus is becoming ever more important as healthcare organisations seek to join up more sources of data whether they be clinical, financial or anything else for that matter.

In many healthcare organisations I talk to DICOM has traditionally been kept separate from the main line integration of clinical applications (often using HL7). This legacy is quickly disappearing though accelerated by many factors not least of which is work being done by IHE (for example, the scheduled workflow profile which requires integration between HL7 and DICOM modalities).

The latest version of the Healthcare Connectivity Pack includes support for medical imaging (DICOM). This enables a wide range of high value use cases. The diagram below shows an example walk through of how the Healthcare Connectivity Pack can route DICOM instances to external experts for a second opinion. This scenario comes in various flavours. For example, perhaps you have several radiologists and you want to work load balance between them. Alternatively, radiologists with specialist skills might be geographically dispersed and so DICOM images can be routed to the correct location.

Another use case is to pre-fetch DICOM instances when a patient is admitted to a hospital:

This scenario uses the Healthcare Connectivity Pack to pull down DICOM images to the local PACS so that the right information is in the right place for a patient's effective treatment. These scenarios uses a range of DICOM capabilities including C-FIND, C-STORE and C-MOVE.

I've posted some slides here which walk through all the technical details of the DICOM integration.

Healthcare Integration with Message Broker

2013-04-07T16:17:00.006+01:00

We have just released our latest and greatest healthcare integration product, the Message Broker Healthcare Connectivity Pack. In the last three years we have made huge steps forward in the healthcare integration space, indeed we have had four product releases in the last two years alone.

This new release comes packed with a host of new features:

DICOM, the digital medical imaging standard, is now natively supported. It enables the integration of image archive systems (PACS) and imaging applications and also enables a broad range of high-value integration use cases such as pre-fetch on admission.

Analytics, the ability to glean clinical insight, is now made easy with integrated tooling that quickly and easily extracts meaningful information from a range of medical sources. These sources include clinical documents (C-CDA/CDA/CCD/HITSP), DICOM images, and HL7 v2 messages. Meaningful data from these sources can be filtered, captured and fed directly into downstream applications and data warehouses for healthcare intelligence applications.

HL7 data models that use the industry standard Data Format Description Language (DFDL), are included in support of the HL7 messaging standard. HL7 v2.3 through to the most recently approved HL7 v2.7 are provided in the pack. The graphical modeling tools bring a modern feel to HL7 data processing, provide excellent performance characteristics, and allow healthcare integrators to customize and develop healthcare integrations quickly and easily.

The ability to send IHE compatible Audit Trail and Node Authentication (ATNA) messages to an audit repository enables logging events across a healthcare network.

The new capabilities augment the existing capabilities of the Healthcare Connectivity Pack:

Ready-to-use connectivity and time saving patterns for healthcare providers who seek to integrate clinical applications based on HL7 v2.x.

Direct connectivity to bedside medical devices that includes patient monitors, infusion pumps, and more (the full list of medical devices is here).

Pattern-based healthcare focused development tooling - rapid integration without detailed knowledge of integration engines or enterprise service bus (ESB) technologies.

Healthcare-oriented, operational monitoring for connected clinical applications and devices.

I've posted some slides here which walk through the high level features now available.

Message Broker Goes Mobile!

2012-09-26T16:23:00.000+01:00

Message Broker v8 FP1 shipped just a few weeks ago. One of the many new features in this release is close integration with mobile applications. The background to this work is an acquisition we made at the start of the year with Worklight. IBM Worklight is a mobile application development product that enables you to quickly and easily develop mobile applications for many devices (iOS, Android, Windows Phone, RIM Blackberry and more). Furthermore, mobile applications are written using well established web development skills (HTML, CSS and JavaScript).

Message Broker now has a host of patterns that address key integration challenges in the mobile space:

Mobile enablement for Microsoft .NET applications

In just a few easy clicks, convert any Microsoft .NET class (C#, VB.NET, F#) into a web service running in Message Broker. The pattern generates the integration logic for Worklight and a complete mobile application that can exercise the .NET service.

Mobile service enablement

Take any Message Broker web service and mobile enable it. The pattern adds an HTTP/JSON binding to the service so that mobile applications through Worklight can integrate with back-end systems. The pattern also does a really great job of documenting (in HTML) the JSON format that the mobile developer should send to Worklight.

Push notifications to mobile devices

There are a whole host of use cases where pushing data out to mobile devices is incredibly useful. For example, perhaps you might want to send a meeting reminder to a client. This pattern creates all the integration logic for pushing notifications out to users on their mobile devices. The notifications themselves are sent out through Worklight using vendor specific push notification servers (for example, Apple APNS and Google C2DM).

Resource handler

This pattern brings a fresh look at the world of CRUD (Create, Read, Update and Delete). The pattern builds a Message Broker application that supports the CRUD operations as well as adding caching and security to complete the picture. Caching uses the WebSphere Extreme Scale (WXS) cache now available in Message Broker. Security supports authorisation and authentication against an LDAP server (such as Microsoft ActiveDirectory).

The source code for all these patterns is available so you are free to take these patterns and extend them in any way that makes sense to you!

More information on all these patterns is available here on my SlideShare account. If you are new to Message Broker, you might find the following introduction slides helpful. And finally, if you are looking for some guidance on how to build integrations with Message Broker, these slides provide lots of information.

Pattern Communities

2010-12-17T09:40:00.003Z

Message Broker patterns are all about creating production ready connectivity solutions. The value of a pattern multiples hugely the more it is shared, and consequently the more it is used. In Message Broker fix pack 2 we added a significant capability to enable pattern communities. You can see some slides that walk through some of this work here.

Now it is worth saying that a community means different things to different people. For some organisations, a community might be private where patterns are created and shared within a development team. In this scenario, patterns offer consistency and time-to-value.

For other organisations, patterns are a way to deliver capability to the market in an easily consumable form. For example, you might want to develop a suite of patterns for a particular industry or technology area.

With this in mind you might be interested in a new global pattern community - mqseries.net has added a pattern community where you can find, download and share patterns. We will be putting some very cool example patterns up over the next few weeks to help get it started. Why mqseries.net? Well communities are all about people, and mqseries.net is where the Broker community go to find answers.

One particular thought to leave you with. We are very motivated to make your business succeed. So you might consider uploading a free starter version of your pattern to mqseries.net. In your pattern you can very easily include your own branding and links to your web site where the professional and enterprise versions can be purchased.

Pattern Authoring in Message Broker

2010-12-17T08:33:00.007Z

Almost a year to the day, Message Broker v7 was released. Well it has been a busy year in the development team. One area I particularly wanted to highlight was the work the team has done around patterns.

Broker v7 introduced the first generation of patterns technology to the connectivity world. The v7 release shipped with an extensive set of built-in connectivity patterns. For example putting a web service front end on an existing application is a problem that comes up again and again in the field. Adding a production-ready pattern to the product made perfect sense.

Back in July we shipped Broker fix pack 1. Along with many new features, pattern authoring made its debut. Pattern authoring allows anyone to design, test, debug, package and distribute their own patterns. The patterns are presented in the Pattern Explorer alongside the built-in patterns provided by the product.

This capability has proved to be a big hit with our customers. We have talked to many customers this year who are now using patterns to simplify their development process, get more consistent solutions and to reduce costs.

Yesterday, Broker fix pack 2 was released. Fix packs are the way we release new functionality between our biennial major product releases. In fix pack 2 we have released the third generation of patterns technology. This includes major enhancements to pattern authoring. You can read about the pattern authoring capabilities in this slide show.

Pattern authors can now plug in their own Java and PHP code to their patterns which is called whenever pattern instances are created. Included in this is a major new Java API to programmatically manipulate message flows. You can browse the infocenter here and the new Java and PHP additions are documented here.

A couple of other links to mention. First there is an introductory article on developerWorks for pattern authoring. We also have a web site called the IBM Education Assistant, here you will find lots of short videos showing you how to use many of the features in the product.

Patterns and Message Broker

2009-10-08T14:41:00.016+01:00

I see WebSphere Message Broker has just announced v7. The product has been brewing for considerably longer than my involvement and we are all working flat out to finish it off. I've been working on Broker for a little over six months now. My bit of the jigsaw is in a new feature called patterns.

Patterns solves a conundrum familiar to developers - it goes like this:

Broker is a product that can do, well, just about anything. It's purpose in life is to connect everything to everything. More than that, it routes, transforms, filters, and validates messages and data along the way. But this flexibility can easily be an Achilles heal.

How, as an inexperienced user of the technology, do I learn to use the product? How do I solve common problems? I don't want to make mistakes, so how do I avoid them? And of course how can I find out what best practices exist?

Samples is one important answer of course! Samples typically demonstrate a particular feature of a product. Very rarely are samples intended for production use. Indeed, many samples remove production requirements like error handling altogether! Another important answer is documentation. This includes concept, task and reference material.

Patterns is different. Patterns creates production ready solutions to problems. For example, perhaps I want to create a Web service front end to a WebSphere MQ application (a frequently asked about problem). No problem, simply create an instance of the Service Facade to WebSphere MQ: one-way pattern. Provide some configuration details, like the queue name, and click Generate. Lo and behold out pops a Message Broker project that implements the pattern! Need advanced logging and error handling? No problem, just enable those options in the pattern parameters.

In v7 we are providing a set of patterns that cover a surprisingly wide set of problems. Need to process messages stored in files? No problem, try the File Processing pattern. Need to put a proxy between a Web service client and server? Try the Service Proxy pattern. And so on.

If you would like to read more, try the Introduction to Patterns developerWorks article.

UTF-8 Problems and Subversion

2009-09-14T19:09:00.016+01:00

I was writing some unit tests recently and needed to store files containing UTF-8 text in Subversion. The file names also contained UTF-8 characters (see below). I was using Eclipse for the development work and ran into problems sharing the files between Linux and Windows.

The problem was that a file created and committed by a Windows machine, was not synchronising correctly to the Linux machine (and vica versa). The character encodings in the file names were being changed somewhere along the commit and synchronise path.

So here is what I needed to do to make it work. First I needed to make sure Eclipse marked the contents of the file as UTF-8. This is straightforward, simply right click the file name, select Properties and change the text encoding to UTF-8 (on Windows the default is Cp1252):

The next step involves configuring the CVS repository. Change to the CVS perspective, right click the repository and select Properties:

Change the Server Encoding to UTF-8 (on Linux the default Server Encoding is UTF-8, hence the problem sharing files):

The files should now commit and synchronise with the correct contents and names. When you commit the files you should probably also commit the org.eclipse.core.resources.prefs file that Eclipse generates in your project. This file stores the character encodings for the file contents.

Integrating SAP NetWeaver in Zero PHP

2009-03-16T14:13:00.012Z

I've been playing with the Big Daddy of enterprise systems, SAP.

I thought it would be interesting to hook up a Zero PHP application with a SAP NetWeaver system. So last weekend I summoned up the courage and, armed with a bottle of red wine, set about the task. Surprisingly enough it wasn't too difficult (setting aside finding 20GB of disk space needed by the installer).

The bit I was interested in was the SAP Java connector. SAP Java Connector (SAP JCo) is a middleware component that enables the development of SAP-compatible components and applications in Java. Since we have a PHP/Java Bridge in Zero I thought it should be possible to wire them up.

Here is an example that invokes a SAP remote function call (RFC):

<?php

try {

java_import("com.sap.mw.jco.JCO", NULL, FALSE, NULL, FALSE);

// Establish the client connection to SAP

$connection = JCO::createClient("000", "bcuser",

"minisap", null, "localhost", "00");

$connection->connect();

$repository = JCO::createRepository("ZeroRepository", $connection);

$template = $repository->getFunctionTemplate("RFC_SYSTEM_INFO");

$function = new Java("com.sap.mw.jco.JCO.Function", $template);

$connection->execute($function);

$list = $function->getExportParameterList();

$structure = $list->getStructure("RFCSI_EXPORT");

$iterator = $structure->fields();

while ($iterator->hasMoreElements()) {

$field = $iterator->nextField();

echo $field->getName()." : ".$field->getString()."\n";

}

$connection->disconnect();

} catch (Exception $exception) {

echo $exception->getMessage();

}

The code turns out to be pretty straightforward. In a production environment it would be better to use some of the connection pooling options to avoid the repetitive connect/disconnect overhead. The function RFC_SYSTEM_INFO doesn't require any arguments but those can be passed as well if required.

If anyone is interested in SAP there is an un-conference event being organised by Nigel James and Zoe Slattery on Saturday 4th April. More details here (beware the SAP/SDN sites do not work in Google Chrome). I am planning to be there to chat about Zero/Java/PHP with SAP NetWeaver.

developerWorks Article: Business Intelligence

2009-03-06T13:14:00.026Z

The developerWorks article on business intelligence I mentioned a while back has been published. It covers several use cases for integrating BIRT reports into a Zero PHP application. It uses the sample database and reports provided by BIRT and runs against a MySQL database.

From my perspective the most interesting example is the last one: scripted data sources. In this use case the report doesn't get its data from a database query. Instead the PHP script creates the data and pushes it into the BIRT runtime ready for the report to render. Here is the sample code:

<?php

require_once 'startup.php';

$engine = startup();

$root_directory = zget("/config/root");

$report_engine = $root_directory."/ReportEngine";

$design_file = $root_directory.

"/samples/data_source.rptdesign";

$images_directory = $root_directory.

"/public/images/logos";

java_import(

"org.eclipse.birt.report.engine.api.HTMLRenderOption");

java_import(

"org.eclipse.birt.report.engine.api.HTMLServerImageHandler");

$design = $engine->openReportDesign($design_file);

$task = $engine->createRunAndRenderTask($design);

$task->validateParameters();

$generated_report_file =

tempnam(sys_get_temp_dir(), 'report');

$models[] = array("ANG Resellers",

"1952 Alpine Renault 1300", "Red");

$models[] = array("AV Stores, Co.",

"1969 Harley Davidson Ultimate Chopper", "Blue");

$models[] = array("Alpha Cognac",

"1969 Ford Falcon", "Blue");

$models[] = array("Asian Shopping Network, Co",

"1969 Dodge Charger", "Plum Crazy Purple");

$models[] = array("Asian Treasures, Inc.",

"1969 Corvair Monza", "Red");

zput("/request/report", $models);

$options = new HTMLRenderOption();

$options->setOutputFileName(

$generated_report_file);

$options->setOutputFormat("html");

$options->setImageDirectory($images_directory);

$options->setBaseImageURL("/images/logos/");

$options->setImageHandler(new HTMLServerImageHandler());

$task->setRenderOption($options);

$task->run();

$task->close();

echo file_get_contents($generated_report_file);

unlink($generated_report_file);

The bold section hard codes some data for the report. A more realistic example might query an external system like a web service to build the data set.

The magic that makes this possible is the JavaScript engine that BIRT embeds. When BIRT renders the report it executes JavaScript embedded in the report with the Rhino scripting engine. Because the BIRT runtime and the PHP application are all running on the same JVM they can share data very easily and efficiently. In this case the PHP code puts the data collection into the Global Context, and the JavaScript in the report pulls it out.

Business Intelligence - The Code

2009-03-03T16:32:00.007Z

A little later than intended - here is the code we were using on the Zero stand at the (most excellent) PHP London conference last Friday. I had it running as a blog in Drupal but this will run just as well standalone. More detailed destructions for getting this running are up on YouTube here (the video is six minutes long).

<?php

require_once 'startup.php';

$engine = startup();

$root_directory = zget("/config/root");

$design_file = $root_directory."/samples/hello_world.rptdesign";

$design = $engine->openReportDesign($design_file);

$task = $engine->createRunAndRenderTask($design);

$task->validateParameters();

java_import("org.eclipse.birt.report.engine.api.HTMLRenderOption");

$generated_report_file = tempnam(sys_get_temp_dir(), 'report');

$options = new HTMLRenderOption();

$options->setOutputFileName($generated_report_file);

$options->setOutputFormat("html");

$task->setRenderOption($options);

$task->run();

$task->close();

echo file_get_contents($generated_report_file);

unlink($generated_report_file);

And the include file that configures the BIRT runtime is as follows:

<?php

function startup() {

zpost("/app#lock", "/engine");

$engine = zget("/tmp/factory");

if ($engine != NULL) {

zpost("/app#unlock", "/engine");

return $engine; // Did someone else get in first?

}

$root_directory = zget("/config/root");

$report_directory = $root_directory."/birt-runtime-2_3_1/ReportEngine";

$log_directory = zget("/config/appLogDir");

java_import("org.eclipse.birt.core.framework.Platform");

java_import("java.util.logging.Level");

java_import("org.eclipse.birt.report.engine.api.EngineConfig");

java_import("org.eclipse.birt.report.engine.api.IReportEngineFactory");

$config = new EngineConfig();

$config->setBIRTHome($report_directory);

$config->setLogConfig($log_directory, Level::$SEVERE);

Platform::startup($config);

$factory = Platform::createFactoryObject(

IReportEngineFactory::$EXTENSION_REPORT_ENGINE_FACTORY);

// Store in process wide non persistent zone

$engine = $factory->createReportEngine($config);

zput("/tmp/factory", $engine);

// Unlock when we are finished!

zpost("/app#unlock", "/engine");

return $engine;

}

PHP London Conference

2009-02-23T22:18:00.034Z

Just a few more days until this year's PHP London Conference. I think it is going to be a brilliant event given the outstanding speaker line up. The PHP London User Group is also hosting a free pre-conference social on Thursday night with guest speaker and PHP contributor Derick Rethans.

I know little about the new features in PHP 5.3 (shame on me) so the talks by Scott MacVicar and Sebastian Bergmann should be well worth taking in. Stuart Herbert's talk on PHP frameworks is sure to be interesting and no doubt will create plenty of audience discussion. There is lots more with talks about security, cloud computing and a term I had never heard before, database sharding.

My small contribution to the event is to put together a couple of demo applications for the Zero stand. Given my main thing is language interop I thought to start with some PHP/Java Bridge examples. The BIRT stuff I rambled about the other day fits nicely and produces some glossy charts just perfect for a demo [:o)

Thanks to Rob we also have a demo using Apache POI (the project name is a story in itself). POI is interesting because it provides a cross platform library for reading and writing Microsoft Office file formats. POI is really easy to use, just grab the download from the POI web site and extract the three JAR files. In Zero they need to go into the lib directory.

<?php

java_import("org.apache.poi.hslf.HSLFSlideShow");

java_import("org.apache.poi.hslf.usermodel.SlideShow");

$fs = new Java("java.io.FileInputStream", "/test.ppt");

$hsss = new HSLFSlideShow($fs);

$ss = new SlideShow($hsss);

foreach ($ss->getSlides() as $slide) {

echo $slide->getTitle();

}

The code simply iterates through the titles of each PowerPoint slide. All very straightforward and just the tip of the iceberg for what POI can do.

Singletons, BIRT, Threads and PHP

2009-02-17T21:13:00.026Z

PHP is designed to run in multithreaded web servers (such as Apache and IIS). What this means in practice is that a single PHP process concurrently processes multiple browser requests. It does this by having multiple threads, each of which handles one HTTP request at a time.

PHP can also be built for non multithreaded web servers. In this deployment model life is somewhat different because the web server spawns multiple PHP processes. When requests arrive in the web server it farms them out among its pool of PHP worker processes. Each worker process handles just one request at a time.

The pool of worker processes can grow and shrink over time to allow for busy periods. Zero is multithreaded so it handles multiple web requests concurrently in the same process.

I’ve been playing with the Eclipse Business Intelligence and Reporting Tools project (BIRT). BIRT is an open source Eclipse project that provides a whole bunch of stuff around designing and deploying reports and charts. It has a Java API and so I thought it would make a nice demonstration of the PHP/Java Bridge in Zero.

The problem is that BIRT needs to be started once, and only once, in any given process. Once it has been started up, then it is plain sailing to load reports and render them to HTML. The BIRT runtime should be kept around until the process shuts down.

This sounds suspiciously like a singleton pattern (or anti-pattern if you prefer). The trouble is that process wide singletons are hard to implement correctly in PHP. You might think the following snippet would work:

class Singleton {

private static $birt = NULL;

static function getInstance() {

if (self::$birt == NULL) {

self::$birt = new BIRT(...);

}

return self::$birt;

}

There are two problems with this. First, at the end of the current request, all statics are cleared away so $birt is reset to NULL. What we want is the BIRT runtime to be kept around forever (or at least until the process shuts down).

The second problem is that on a multithreaded PHP runtime each PHP request gets an independent set of static and global variables. If two requests are being processed, and each calls getInstance, then they will both create an instance of the BIRT class.

The design intent in PHP is that each request looks like it is running in a single threaded PHP process, even though it is actually running in a multithreaded PHP process. But what I want is one (yes really one!) object in a process which can be shared amongst all the threads. The reason this works is that the BIRT runtime is thread safe, meaning that it can be safely used by multiple threads at the same time.

The solution for me turned out to be the Zero Global Context. The Global Context is a storage area in Zero where just about anything can be put. Different bits of the Global Context have different life times. For example, the /request zone is only kept around for the current request. I opted to put the BIRT runtime in the /tmp zone so that it is kept around for the life time of the process. It is also available to all threads running in the same process.

The final piece of the jigsaw was to ensure only one thread actually created the BIRT runtime. Dealing with multiple threads can be something of a headache as race conditions are easily created.

The Global Context provides a solution (albeit slightly crufty) with locking commands. Each thread does a lock, checks to see if the BIRT runtime has already been created (if not then it creates it and puts it in the /tmp zone) and then unlocks.

With a bit of luck I'll get this written up in an developerWorks article soon as the reports look really sweet (cross tabs, charts, compound reports and so).

Charles Darwin and PHP

2009-01-29T19:12:00.010Z

This year is the 150th anniversary of the publication of On the Origin of Species (24 November 1859) and the 200th anniversary of Darwin's birth (12 February 1809). The BBC recently screened an absolutely fascinating program called "What Darwin Didn't Know". It is on iPlayer here although I'm not sure it is available outside of the UK - sorry if not!

What really struck me was not only his insight, but also how Darwin devoted an entire chapter to describe the difficulties of the theory! Remarkable as well how he accepted the theory was incomplete and asked for future generations to fill in the gaps.

All this got me thinking about applying the idea of natural selection to open source software. I was sure there must be an analogy for how open source projects grow and die that mimics the world of evolutionary biology.

I also think natural selection provides a great explanation for why PHP is so hugely successful. I don't often hear people describe PHP as a pure and elegant programming language. A criticism that purists in the OOP world sometimes direct at us.

But that is rather missing the point I think! PHP succeeds because it has evolved over a relatively long period of time to do things quickly, easily, and well.

Anyway, I set to task to find some prior art on this but my Googling didn't find any great matches. So my next thought was to ferret around and find some obsolete programming skills to show what has actually died out.

Alas, as before, nothing quite hit the mark. I did however dig up this web site where people are documenting obsolete skills. The problem is that most of the obsolete skills listed turn out to either still be used, or staging comebacks in one form or another (such as putting a needle on a vinyl record)!

So if you do know of somewhere that talks about natural selection and open source software, please let me know.

COM/.NET Interop in Zero PHP

2009-01-29T14:47:00.030Z

Zero doesn't currently support the COM/.NET extension in PHP.

No matter though, there is a handy open source project called JACOB that bridges between Java and COM/.NET. In fact, there is an easier way to do this using a Groovy library called Scriptom. This is really just a friendly wrapper around JACOB to provide a better syntax for calling methods and accessing properties.

The aim in this example is to list the processes on a Windows machine - here are the steps to do it in PHP. I'm using the latest Zero download of Sebring and Eclipse PDT as the IDE.

1. Download Scriptom and put the JAR and DLL in the Zero lib directory:

2. Run a resolve step in Eclipse so that the JAR file is added to the class path.

3. Add the following extension to php.ini (if not already there):

extension=com.ibm.p8.engine.xapi.groovy.GroovyExtensionLibrary

4. Copy and paste this code into a PHP script in the public folder:

<?php

define("wbemFlagForwardOnly", 32);

java_import("com.jacob.com.ComThread");

ComThread::InitMTA(true);

groovy_import("org.codehaus.groovy.scriptom.Scriptom");

$locator = new ActiveXObject("WbemScripting.SWbemLocator");

$services = $locator->ConnectServer('.');

$processes = $services->ExecQuery(

'SELECT * FROM Win32_Process',

'WQL', wbemFlagForwardOnly);

$total = 0;

// Iterate through the COM collection

foreach ($processes as $current) {

if (Scriptom::isNull($current->CreationDate) == FALSE) {

$date = new ActiveXObject("WbemScripting.SWbemDateTime");

$date->Value = $current->CreationDate;

$creation = $date->GetVarDate(TRUE);

$command = $current->CommandLine;

$command = (Scriptom::isNull($command) ? "None" : $command);

echo "Process found [".$current->Name.

" - ".$command."]"; $total++;

}

ComThread::Release();

echo "Found $total processes in all";

Before you say it, yes I agree, the code is really yuck but that's COM for you.

5. Run the application and point a browser at your script:

If you are curious about what all these crazy objects are, there is lots more information on the Microsoft web site. The example is using Windows Management Instrumentation (WMI) to enumerate the processes.

The calls to ComThread::InitMTA() and Release() are important to make sure Zero doesn't hang. COM has some weird stuff in it to do with apartments and threading.

The Long Road to 64 Bits

2009-01-27T16:23:00.013Z

There is an interesting story in January's Communications of the ACM called The Long Road to 64 Bits. The magazine is free online here. It charts the long and often twisted road from 32 to 64 bit architectures.

From a software perspective the challenge is around writing C code that compiles and runs cleanly on 32 and 64 bit platforms. The issue being that compilers on different hardware architectures adopt different sized data types for integers, longs and pointers.

So for example, on Windows 32 bit (and many Unix's from the 1990's) all three were 32 bits wide, hence the tag ILP32. On 64 bit Windows, integers and longs remained 32 bits wide while pointers necessarily increased to 64 bits - so that goes by the tag IL32P64, or just P64 for short. Meanwhile many 64 bit Unix systems adopted I32LP64 - meaning integers remained 32 bits while longs and pointers widened to 64 bits.

PHP has to deal with this issue since it is written in C and is ported to many platforms. This issue also comes up when mixing Java and C code because Java has a different approach. Java set out to simplify the programming model by fixing the sizes of all data types. An integer in Java is 32 bits, always 32 bits and never anything apart from 32 bits. Java also has a long data type which is always 64 bits and has no unsigned data types at all.

Problems arise when Java code (with a fixed size data type) interacts with C code (that varies the data type size depending on the platform). Java has no pre-processor like C so it's not easy to write Java code that would, for example, use a Java integer on 32 bit machines, and magically use a Java long on 64 bit machines (and could therefore match the C code). So instead we generally use the wider data type (a Java long) on both 32 and 64 bit machines (and down cast where necessary).

At this point I should also mention z/OS which for it's own honourable reasons supported 24, 31 and 64 bit architectures...!

Extending PHP in Project Zero

2009-01-21T13:26:00.022Z

The de facto open source implementation of PHP is written almost entirely in C. Looking under the hood shows that it is really two fairly distinct parts. First there is the Zend Engine. This is the core language engine that parses PHP scripts and interprets the resulting opcodes.

The Zend Engine also contains the implementation of PHP types like variables (zval) and arrays (HashTable). So where are all the PHP functions and classes implemented then?

The answer is that the Zend Engine provides an extension API. This allows anyone to write classes, functions, constants and much more that plug in to the php.net runtime. Taking a peek at the php.net source code in CVS reveals just how many extensions there are.

This C API is well documented in Sara Goleman's book. It talks in detail about how to extend PHP and indeed how to embed PHP in your own applications.

Zero has a very similar architecture that separates engine and extension. The main difference is that the core PHP language engine is written in Java. Zero also has a comprehensive API called XAPI-J that allows Java extensions to be written for the Zero PHP runtime.

We use XAPI-J to implement virtually all the PHP extension functions supported in Zero (listed here). But there is a bit more to this story than meets the eye. The slides Rob Nicholson presented at last year's PHP Quebec cast some light on this.

Zero actually ships a pretty large number of the php.net C extensions. We do this by re-compiling the php.net source code against a different implementation of the php.net C API. The Zero implementation of the C API (called XAPI-C) knows how to talk to the Zero PHP runtime through JNI.

There are several benefits to this solution. First and foremost we maintain compatibility with the open source PHP. It is really important to PHP developers on Zero that we are as close to 100% compatible with php.net as possible, and using the php.net code certainly helps!

Secondly, we avoid re-implementing all those extension functions from scratch. Why bother re-writing everything in Java when there is a perfectly good implementation already available! There are in fact a few performance critical functions that we do implement in Java, the array functions are one good example.

If you are interested in writing extensions for Zero you will find the documentation and examples for XAPI-J on the Zero web site.

PHP and Java Collections: More PHP, Less Java

2009-01-21T11:32:00.029Z

We've added some nice integration with the Java collection classes in Zero.

Java has a rich set of collection classes. The three really important ones are Set, Map and List. We use the Java collections quite a bit when integrating Java libraries in PHP applications. So it got us thinking whether we could provide some good PHP syntax for these classes. It turns out there are quite a few things we could do to improve working with these classes in PHP.

For example, Java lists now support the following:

<?php

java_import("java.util.ArrayList");

$list = new ArrayList();

$list->add("Hello World!");

$list[0] = "Hello Again!";

$value = $list[0];

$array = (array) $list;

$check = isset($list[0]);

unset($list[0]);

// Append some items to list

$list[] = "Hello Again!";

$list[] = TRUE;

$list[] = "Updated!";

Java maps and sets also get the full treatment. The reason why I like this is that it avoids lots of Java API calls. Some example code for hacking around with a Map using the Java APIs is shown below. Notice all the method calls, just so much more long winded than PHP's nice expressive syntax!

<?php

$map = new Java("java.util.HashMap");

$map->put("title", "Java Bridge!");

$value = $map->get("title");

$check = $map->containsKey("title");

$map->remove("stuff");

More information is on the Project Zero PHP/Java Bridge reference page here.

Creative Commons picture attribute to MicMac.

PHP/Groovy Bridge

2009-01-12T08:53:00.001Z

Ah, something new and shiny in Project Zero to talk about.

Zero supports three languages: Java, PHP and Groovy. It also supports a workflow runtime called Assemble and since that stores its flows as XML, and workflows are kind of like code, I suppose it could be considered an unofficial fourth language.

Anyway I digress. Up until recently the PHP runtime was treating Groovy code just the same as Java. Anyone could compile a Groovy class to Java class files using the Groovy compiler. Those class files could then be dropped into a Zero application and used in PHP through the PHP/Java Bridge.

The problem is that Groovy isn’t the same as Java. Groovy adds lots of scripting features over and above Java’s current capabilities. Closures being one obvious highlight. This shines through if Groovy classes are used through a PHP/Java Bridge, it all just looks like Java. Take for example the following Groovy class that makes use of Groovy’s interceptors:

class Dynamic {

def storage = [:]

def invokeMethod(String name, args) {

return "Hello World!";

}

def getProperty(String name) {

storage[name]

}

void setProperty(String name, value) {

storage[name] = value

}

Another piece of Groovy can use this as follows:

foo = new Dynamic();

foo.bar();

foo.guff = "Hello World!";

print foo.guff;

Now see how this looks from a Java client:

public class Test {

public static void main(String[] args) {

Dynamic foo = new Dynamic();

String result = (String)

foo.invokeMethod("sayHello", null);

foo.setProperty("guff", "Hello World!");

result = (String) foo.getProperty("guff");

}

Yuck! And the same applies if we use this class in PHP through the Java Bridge. Well the PHP/Groovy Bridge solves this problem by integrating PHP and Groovy as first class citizens - see how it looks with the Groovy Bridge:

<?php

groovy_import("dynamic.groovy");

$foo = new Dynamic();

$foo->bar();

$foo->guff = "Hello World!";

echo $foo->guff;

That’s better! Intuitive use of the Groovy class as intended. The groovy_import function is just one aspect of the PHP/Groovy Bridge. It also supports a range of other nice Groovy interop features including closures, currying and range objects. With this in mind I’ve put a slide cast up here where I ramble my way through an explanation of the PHP/Groovy Bridge.

The product documentation for the PHP/Groovy Bridge is also available here.

2009-01-07T15:23:00.001Z

I've put some slides up here that Andy Coleman presented at the WebSphere Technical Conference in November last. They discuss how sMash PHP fits into Message Broker and some of the really sweet integration points with the PHP language. Our inspiration for this was the SimpleXML extension which provides a fantastically easy way to navigate XML documents.

Message Broker has an existing Java API which we made heavy use of for this work. Much of the work involved was to catch method and field accesses in the PHP script and turn them into Java API calls on Message Broker. For example, take a look at the following message tree:

Here's a first attempt at navigating through this message tree to the title:

$xml = $input->getLastChild();

$doc = $xml->getFirstChild();

$ch1 = $doc->getFirstChild();

$ch2 = $ch1->getNextSibling()

$title = $ch2->getFirstChild();

Yuck! This is really horrible, lots of method calls and very little fluency. Ok we can do better, let's try again:

$xml = $input->getChild(‘XMLNSC’);

$doc = $xml->getChild(‘document’);

$ch2 = $doc->getChild(‘chapter’, 1);

$title = $ch2->getAttribute(‘title’);

Well it's a bit better but still far too much like hard work.

$title =

$input->XMLNSC->document->chapter[1][‘title’]

That's better! A really expressive way of navigating through messages. This is only scratching the surface of the PHP integration and the slides cover many other examples such as array integration, message shredding and content based routing.

Static Fields and Methods

2009-01-06T10:36:00.000Z

One nice feature of PHP is its support for magic methods. These allow a class or object to catch a range of events such as method calls and property accesses. This kind of interception is really powerful and allows classes to implement all sorts of dynamic behaviour. For example, a database wrapper could intercept property access to get a property value from a database row. Many scripting languages offer something similar, for example Groovy has its own set of interceptors.

php.net is implemented in C and under the covers it implements magic methods by attaching a table of function pointers to each object (called the object handlers). In most cases the default implementation of an object handler is to simply call the appropriate PHP method on the class.

For example, there is a __call object handler which calls the __call PHP magic method if it exists. Built in classes like SimpleXML and MySQLi use object handlers to implement whatever logic they need. All of this and a lot more besides is documented in Sara Golemans book.

In Project Zero we added magic methods like __call, __get and __set a long time ago. Over the Christmas break we finally got round to adding their static equivalents: __callStatic, __getStatic, __setStatic and __issetStatic.

Here's an example of how __callStatic can be used:

<?php

class Foo {

static public function __callStatic($a, $b) {

print "__callStatic [$a]\n";

}

foo::Test();

We have also surfaced this feature into the Java Bridge:

<?php

java_import("java.lang.Integer", NULL, FALSE);

var_dump(Integer::$MIN_VALUE);

var_dump(Integer::$MAX_VALUE);

var_dump(Integer::toHexString("1234567890"));

var_dump(Integer::toOctalString("1234567890"));

// Signatures also work as normal!

$signature = new JavaSignature(JAVA_STRING);

var_dump(Integer::parseInt($signature, "1234567890"));

This makes for a much cleaner syntax than using JavaClass. JavaClass was always something of an anomaly (er, hack) as it required the script to create an instance of an object just to access static class methods and properties.

More information about this feature will be available just as soon as the Project Zero documentation has rebuilt tonight [:o)

Thoughts on Bridging PHP and Java

2009-01-01T09:49:00.000Z

I thought it might be interesting to talk a little about how we bridge between Java and PHP in Project Zero. So to start with I'm going to explain some background to the problem and then in a follow up post talk more about how the Java Bridge works in practice.

Java and PHP

I'm not going to enumerate every difference between the languages but I will talk a little about the main issues that effect the Java Bridge. First of all I should say that Java and PHP share much common ground in their object oriented design: objects, interfaces, classes, methods, fields, static members, exceptions, visibility (public, private etc) and so on.

They also have some differences and this is where the problems often arise when integrating the two languages. For example, PHP methods, classes and function names are case insensitive, while in Java they are case sensitive. So it is possible, although unlikely, that a PHP script could invoke a method foo() which has several possible target methods on a Java object (Foo, FOO, foo etc).

Overloaded Methods

PHP does not support overloading of constructors nor methods. Java does support overloaded methods and constructors. Overloading in Java means two or more methods or constructors on the same class that have the same name but different argument types. Overloading on the return type alone is not allowed.

The most difficult issue with overloading is selecting the right overload. This has to be done at runtime for reasons explained later.

Static and Instance Methods

PHP also allows instance methods on an object to be called statically, consider the following code:


 class Foo {
    function bar() { }
}

Foo::bar();

This runs fine albeit with the following strict coding notice to highlight the potential problem:

Strict Standards: Non-static method Foo::bar() should not be called statically in foo.php on line 7

In case you are wondering, if the method trys to access an instance field as follows:

class Foo {

function bar() {

$this->a = "Hello World!";

}

Foo::bar();

Then a fatal error terminates the script with this output:

Fatal error: Using $this when not in object context in foo.php on line 4

In the context of the Java Bridge, we cannot call an instance method on a Java object in a static way because this is simply not possible in Java. We can however do the reverse, which is to allow a PHP script to call a static method through an instance of the object. As the Java API documentation for a dynamic invoke shows, the obj argument is ignored if the method is static.

Dynamic Typing (Duck Typing)

From my experience, the biggest difficulty by far is with data typing. In PHP, variables have no declared data type. One minute a variable might be an integer and the next a string. In Java all variables have a fixed type, albeit with some latitude granted by polymorphism. Furthermore PHP does an amazing job of coercing types to match the circumstances:

echo strlen(array(1,2,3));

In this snippet I'm passing an array to a function clearly expecting a string. PHP politely informs me via a notice that it had to do something strange:

Notice: Array to string conversion in foo.php on line 3

And outputs 5 (why 5? Because it converted the array to the string Array).

Here is another simple example of type coercion:

echo abs("123456789");

In this snippet the abs function converts the string to an integer.

So what does this mean?

It means selecting the Java method to invoke is often the most difficult (and time consuming) part of the Java Bridge. This is especially true for overloaded constructors and methods. Perhaps the data type of an argument is a string and there is no candidate Java method that accepts a string argument. But instead, what if the string can be coerced to an integer like the abs snippet above? The permutations are many and varied and that's what makes bridging languages such an interesting problem.

The key thing is that we always follow PHP's example of being incredibly tolerant and wherever possible do the best thing we can. In some cases we simply cannot be absolutely sure we have chosen the right method or constructor. In which case we pick what we think is best and output a message saying the script needs to give us a better clue (for example, by using the JavaSignature class).

In many regards the Java Bridge acts much like a compiler does in selecting methods and fields. The big difference is that the Java Bridge does the selection at runtime based on the data types of arguments. A compiler does the selection at compile time based on the declared data types of arguments. And of course the Java Bridge has to be quick, really quick. Java's reflection API is much faster than it used to be but it is still relatively slow to lookup classes, methods and parameter types.

Java Bridge

2008-12-31T13:07:00.001Z

I work in the team developing the PHP runtime for WebSphere sMash (Project Zero). The PHP engine is implemented in Java. One very powerful feature of the PHP runtime is its Java Bridge. The aim is to allow completely seemless integration between Java, PHP and Groovy. We released WebSphere sMash 1.1 just a few weeks ago and are hard at work on the next version already. With this in mind I've posted some slides that show what the Java Bridge can do here.

There is also a developerWorks article here which shows how to use the Apache Lucene libraries in sMash.

Because we compile PHP scripts down to Java byte code, the Java Bridge is also extremely fast. This has made it possible to do some interesting integration with sMash PHP into products other than web server technology. For example, we shipped a new feature in WebSphere Message Broker at the end of November so that custom transformation and routing logic can be written in PHP.

Message Broker is the brains behind many seriously high traffic systems (from various stock exchanges to the Wimbledon Tennis Championships). This kind of business logic is performance sensitive and the fast PHP/Java interop really helps. There is more information about the new PHP Compute Node here.

If you are interested in PHP you might also be interested in Zoe's blog - she recently posted about some related work we have been doing to integrate sMash into CICS. All of this goes to show how PHP's flexibility makes it a really good general purpose programming language beyond just the web application space.