Thursday, October 8, 2009

Patterns and Message Broker

I see WebSphere Message Broker has just announced v7. The product has been brewing for considerably longer than my involvement and we are all working flat out to finish it off. I've been working on Broker for a little over six months now. My bit of the jigsaw is in a new feature called patterns.

Patterns solves a conundrum familiar to developers - it goes like this:

Broker is a product that can do, well, just about anything. It's purpose in life is to connect everything to everything. More than that, it routes, transforms, filters, and validates messages and data along the way. But this flexibility can easily be an Achilles heal.

How, as an inexperienced user of the technology, do I learn to use the product? How do I solve common problems? I don't want to make mistakes, so how do I avoid them? And of course how can I find out what best practices exist?

Samples is one important answer of course! Samples typically demonstrate a particular feature of a product. Very rarely are samples intended for production use. Indeed, many samples remove production requirements like error handling altogether! Another important answer is documentation. This includes concept, task and reference material.

Patterns is different. Patterns creates production ready solutions to problems. For example, perhaps I want to create a Web service front end to a WebSphere MQ application (a frequently asked about problem). No problem, simply create an instance of the Service Facade to WebSphere MQ: one-way pattern. Provide some configuration details, like the queue name, and click Generate. Lo and behold out pops a Message Broker project that implements the pattern! Need advanced logging and error handling? No problem, just enable those options in the pattern parameters.

In v7 we are providing a set of patterns that cover a surprisingly wide set of problems. Need to process messages stored in files? No problem, try the File Processing pattern. Need to put a proxy between a Web service client and server? Try the Service Proxy pattern. And so on.

If you would like to read more, try the Introduction to Patterns developerWorks article.

Monday, September 14, 2009

UTF-8 Problems and Subversion

I was writing some unit tests recently and needed to store files containing UTF-8 text in Subversion. The file names also contained UTF-8 characters (see below). I was using Eclipse for the development work and ran into problems sharing the files between Linux and Windows.

The problem was that a file created and committed by a Windows machine, was not synchronising correctly to the Linux machine (and vica versa). The character encodings in the file names were being changed somewhere along the commit and synchronise path.

So here is what I needed to do to make it work. First I needed to make sure Eclipse marked the contents of the file as UTF-8. This is straightforward, simply right click the file name, select Properties and change the text encoding to UTF-8 (on Windows the default is Cp1252):

The next step involves configuring the CVS repository. Change to the CVS perspective, right click the repository and select Properties:

Change the Server Encoding to UTF-8 (on Linux the default Server Encoding is UTF-8, hence the problem sharing files):
The files should now commit and synchronise with the correct contents and names. When you commit the files you should probably also commit the org.eclipse.core.resources.prefs file that Eclipse generates in your project. This file stores the character encodings for the file contents.

Monday, March 16, 2009

Integrating SAP NetWeaver in Zero PHP

I've been playing with the Big Daddy of enterprise systems, SAP

I thought it would be interesting to hook up a Zero PHP application with a SAP NetWeaver system. So last weekend I summoned up the courage and, armed with a bottle of red wine, set about the task. Surprisingly enough it wasn't too difficult (setting aside finding 20GB of disk space needed by the installer).

The bit I was interested in was the SAP Java connector. SAP Java Connector (SAP JCo) is a middleware component that enables the development of SAP-compatible components and applications in Java. Since we have a PHP/Java Bridge in Zero I thought it should be possible to wire them up.

Here is an example that invokes a SAP remote function call (RFC):

try {

java_import("", NULL, FALSE, NULL, FALSE);
// Establish the client connection to SAP
  $connection = JCO::createClient("000", "bcuser", 
  "minisap", null, "localhost", "00");
$repository = JCO::createRepository("ZeroRepository", $connection);
  $template = $repository->getFunctionTemplate("RFC_SYSTEM_INFO");
  $function = new Java("", $template);
$list = $function->getExportParameterList();
  $structure = $list->getStructure("RFCSI_EXPORT");

$iterator = $structure->fields();
  while ($iterator->hasMoreElements()) {
  $field = $iterator->nextField();
  echo $field->getName()." : ".$field->getString()."\n";

} catch (Exception $exception) {
echo $exception->getMessage();

The code turns out to be pretty straightforward. In a production environment it would be better to use some of the connection pooling options to avoid the repetitive connect/disconnect overhead. The function RFC_SYSTEM_INFO doesn't require any arguments but those can be passed as well if required.

If anyone is interested in SAP there is an un-conference event being organised by Nigel James and Zoe Slattery on Saturday 4th April. More details here (beware the SAP/SDN sites do not work in Google Chrome). I am planning to be there to chat about Zero/Java/PHP with SAP NetWeaver.

Friday, March 6, 2009

developerWorks Article: Business Intelligence

The developerWorks article on business intelligence I mentioned a while back has been published. It covers several use cases for integrating BIRT reports into a Zero PHP application. It uses the sample database and reports provided by BIRT and runs against a MySQL database.

From my perspective the most interesting example is the last one: scripted data sources. In this use case the report doesn't get its data from a database query. Instead the PHP script creates the data and pushes it into the BIRT runtime ready for the report to render. Here is the sample code:

require_once 'startup.php';
$engine = startup();
$root_directory = zget("/config/root");
$report_engine = $root_directory."/ReportEngine";
$design_file = $root_directory.
$images_directory = $root_directory.


$design = $engine->openReportDesign($design_file);
$task = $engine->createRunAndRenderTask($design);
$generated_report_file = 
    tempnam(sys_get_temp_dir(), 'report');

$models[] = array("ANG Resellers", 
"1952 Alpine Renault 1300", "Red");
$models[] = array("AV Stores, Co.", 
"1969 Harley Davidson Ultimate Chopper", "Blue");
$models[] = array("Alpha Cognac", 
"1969 Ford Falcon", "Blue");
$models[] = array("Asian Shopping Network, Co", 
"1969 Dodge Charger", "Plum Crazy Purple");
$models[] = array("Asian Treasures, Inc.", 
"1969 Corvair Monza", "Red");

zput("/request/report", $models);

$options = new HTMLRenderOption();
$options->setImageHandler(new HTMLServerImageHandler());


echo file_get_contents($generated_report_file);

The bold section hard codes some data for the report. A more realistic example might query an external system like a web service to build the data set. 

The magic that makes this possible is the JavaScript engine that BIRT embeds. When BIRT renders the report it executes JavaScript embedded in the report with the Rhino scripting engine. Because the BIRT runtime and the PHP application are all running on the same JVM they can share data very easily and efficiently. In this case the PHP code puts the data collection into the Global Context, and the JavaScript in the report pulls it out.

Tuesday, March 3, 2009

Business Intelligence - The Code

A little later than intended - here is the code we were using on the Zero stand at the (most excellent) PHP London conference last Friday. I had it running as a blog in Drupal but this will run just as well standalone. More detailed destructions for getting this running are up on YouTube here (the video is six minutes long).

require_once 'startup.php';
$engine = startup();

$root_directory = zget("/config/root");
$design_file = $root_directory."/samples/hello_world.rptdesign";
$design = $engine->openReportDesign($design_file);
$task = $engine->createRunAndRenderTask($design);
$generated_report_file = tempnam(sys_get_temp_dir(), 'report');

$options = new HTMLRenderOption();

echo file_get_contents($generated_report_file);

And the include file that configures the BIRT runtime is as follows:

function startup() {
  zpost("/app#lock", "/engine"); 
  $engine = zget("/tmp/factory");
  if ($engine != NULL) {
zpost("/app#unlock", "/engine"); 
return $engine; // Did someone else get in first?

  $root_directory = zget("/config/root");
$report_directory = $root_directory."/birt-runtime-2_3_1/ReportEngine";
  $log_directory = zget("/config/appLogDir");


  $config = new EngineConfig();
  $config->setLogConfig($log_directory, Level::$SEVERE);

$factory = Platform::createFactoryObject(

  // Store in process wide non persistent zone
$engine = $factory->createReportEngine($config);
  zput("/tmp/factory", $engine);

  // Unlock when we are finished!
  zpost("/app#unlock", "/engine"); 

return $engine;

Monday, February 23, 2009

PHP London Conference

Just a few more days until this year's PHP London Conference. I think it is going to be a brilliant event given the outstanding speaker line up. The PHP London User Group is also hosting a free pre-conference social on Thursday night with guest speaker and PHP contributor Derick Rethans.

I know little about the new features in PHP 5.3 (shame on me) so the talks by Scott MacVicar and Sebastian Bergmann should be well worth taking in. Stuart Herbert's talk on PHP frameworks is sure to be interesting and no doubt will create plenty of audience discussion. There is lots more with talks about security, cloud computing and a term I had never heard before, database sharding.

My small contribution to the event is to put together a couple of demo applications for the Zero stand. Given my main thing is language interop I thought to start with some PHP/Java Bridge examples. The BIRT stuff I rambled about the other day fits nicely and produces some glossy charts just perfect for a demo [:o)

Thanks to Rob we also have a demo using Apache POI (the project name is a story in itself). POI is interesting because it provides a cross platform library for reading and writing Microsoft Office file formats. POI is really easy to use, just grab the download from the POI web site and extract the three JAR files. In Zero they need to go into the lib directory.

$fs = new Java("", "/test.ppt");

$hsss = new HSLFSlideShow($fs);
$ss = new SlideShow($hsss);
foreach ($ss->getSlides() as $slide) {  
    echo $slide->getTitle();

The code simply iterates through the titles of each PowerPoint slide. All very straightforward and just the tip of the iceberg for what POI can do.

Tuesday, February 17, 2009

Singletons, BIRT, Threads and PHP

PHP is designed to run in multithreaded web servers (such as Apache and IIS). What this means in practice is that a single PHP process concurrently processes multiple browser requests. It does this by having multiple threads, each of which handles one HTTP request at a time.

PHP can also be built for non multithreaded web servers. In this deployment model life is somewhat different because the web server spawns multiple PHP processes. When requests arrive in the web server it farms them out among its pool of PHP worker processes. Each worker process handles just one request at a time. 

The pool of worker processes can grow and shrink over time to allow for busy periods. Zero is multithreaded so it handles multiple web requests concurrently in the same process.

I’ve been playing with the Eclipse Business Intelligence and Reporting Tools project (BIRT). BIRT is an open source Eclipse project that provides a whole bunch of stuff around designing and deploying reports and charts. It has a Java API and so I thought it would make a nice demonstration of the PHP/Java Bridge in Zero.

The problem is that BIRT needs to be started once, and only once, in any given process. Once it has been started up, then it is plain sailing to load reports and render them to HTML. The BIRT runtime should be kept around until the process shuts down. 

This sounds suspiciously like a singleton pattern (or anti-pattern if you prefer). The trouble is that process wide singletons are hard to implement correctly in PHP. You might think the following snippet would work:

class Singleton {
private static $birt = NULL;
    static function getInstance() {
  if (self::$birt == NULL) {
    self::$birt = new BIRT(...);
  return self::$birt;

There are two problems with this. First, at the end of the current request, all statics are cleared away so $birt is reset to NULL. What we want is the BIRT runtime to be kept around forever (or at least until the process shuts down). 

The second problem is that on a multithreaded PHP runtime each PHP request gets an independent set of static and global variables. If two requests are being processed, and each calls getInstance, then they will both create an instance of the BIRT class. 

The design intent in PHP is that each request looks like it is running in a single threaded PHP process, even though it is actually running in a multithreaded PHP process. But what I want is one (yes really one!) object in a process which can be shared amongst all the threads. The reason this works is that the BIRT runtime is thread safe, meaning that it can be safely used by multiple threads at the same time.

The solution for me turned out to be the Zero Global Context. The Global Context is a storage area in Zero where just about anything can be put. Different bits of the Global Context have different life times. For example, the /request zone is only kept around for the current request. I opted to put the BIRT runtime in the /tmp zone so that it is kept around for the life time of the process. It is also available to all threads running in the same process.

The final piece of the jigsaw was to ensure only one thread actually created the BIRT runtime. Dealing with multiple threads can be something of a headache as race conditions are easily created.

The Global Context provides a solution (albeit slightly crufty) with locking commands. Each thread does a lock, checks to see if the BIRT runtime has already been created (if not then it creates it and puts it in the /tmp zone) and then unlocks.

With a bit of luck I'll get this written up in an developerWorks article soon as the reports look really sweet (cross tabs, charts, compound reports and so).

Thursday, January 29, 2009

Charles Darwin and PHP

This year is the 150th anniversary of the publication of On the Origin of Species (24 November 1859) and the 200th anniversary of Darwin's birth (12 February 1809). The BBC recently screened an absolutely fascinating program called "What Darwin Didn't Know". It is on iPlayer here although I'm not sure it is available outside of the UK - sorry if not!

What really struck me was not only his insight, but also how Darwin devoted an entire chapter to describe the difficulties of the theory! Remarkable as well how he accepted the theory was incomplete and asked for future generations to fill in the gaps.

All this got me thinking about applying the idea of natural selection to open source software. I was sure there must be an analogy for how open source projects grow and die that mimics the world of evolutionary biology. 

I also think natural selection provides a great explanation for why PHP is so hugely successful. I don't often hear people describe PHP as a pure and elegant programming language. A criticism that purists in the OOP world sometimes direct at us. 

But that is rather missing the point I think! PHP succeeds because it has evolved over a relatively long period of time to do things quickly, easily, and well.

Anyway, I set to task to find some prior art on this but my Googling didn't find any great matches. So my next thought was to ferret around and find some obsolete programming skills to show what has actually died out. 

Alas, as before, nothing quite hit the mark. I did however dig up this web site where people are documenting obsolete skills. The problem is that most of the obsolete skills listed turn out to either still be used, or staging comebacks in one form or another (such as putting a needle on a vinyl record)!

So if you do know of somewhere that talks about natural selection and open source software, please let me know.

COM/.NET Interop in Zero PHP

Zero doesn't currently support the COM/.NET extension in PHP. 

No matter though, there is a handy open source project called JACOB that bridges between Java and COM/.NET. In fact, there is an easier way to do this using a Groovy library called Scriptom. This is really just a friendly wrapper around JACOB to provide a better syntax for calling methods and accessing properties. 

The aim in this example is to list the processes on a Windows machine - here are the steps to do it in PHP. I'm using the latest Zero download of Sebring and Eclipse PDT as the IDE.

1. Download Scriptom and put the JAR and DLL in the Zero lib directory:

2. Run a resolve step in Eclipse so that the JAR file is added to the class path.

3. Add the following extension to php.ini (if not already there):

4. Copy and paste this code into a PHP script in the public folder:

define("wbemFlagForwardOnly", 32);

$locator = new ActiveXObject("WbemScripting.SWbemLocator");
$services = $locator->ConnectServer('.');

$processes = $services->ExecQuery(
  'SELECT * FROM Win32_Process',
  'WQL', wbemFlagForwardOnly);

$total = 0;

// Iterate through the COM collection
foreach ($processes as $current) {
  if (Scriptom::isNull($current->CreationDate) == FALSE) {
$date = new ActiveXObject("WbemScripting.SWbemDateTime");
$date->Value = $current->CreationDate;
$creation = $date->GetVarDate(TRUE);
$command = $current->CommandLine;
$command = (Scriptom::isNull($command) ? "None" : $command);
echo "Process found [".$current->Name.
        " - ".$command."]"; $total++;

echo "Found $total processes in all";

Before you say it, yes I agree, the code is really yuck but that's COM for you. 

5. Run the application and point a browser at your script:

If you are curious about what all these crazy objects are, there is lots more information on the Microsoft web site. The example is using Windows Management Instrumentation (WMI) to enumerate the processes.

The calls to ComThread::InitMTA() and Release() are important to make sure Zero doesn't hang. COM has some weird stuff in it to do with apartments and threading.

Tuesday, January 27, 2009

The Long Road to 64 Bits

There is an interesting story in January's Communications of the ACM called The Long Road to 64 Bits. The magazine is free online here. It charts the long and often twisted road from 32 to 64 bit architectures. 

From a software perspective the challenge is around writing C code that compiles and runs cleanly on 32 and 64 bit platforms. The issue being that compilers on different hardware architectures adopt different sized data types for integers, longs and pointers. 

So for example, on Windows 32 bit (and many Unix's from the 1990's) all three were 32 bits wide, hence the tag ILP32. On 64 bit Windows, integers and longs remained 32 bits wide while pointers necessarily increased to 64 bits - so that goes by the tag IL32P64, or just P64 for short. Meanwhile many 64 bit Unix systems adopted I32LP64 - meaning integers remained 32 bits while longs and pointers widened to 64 bits. 

PHP has to deal with this issue since it is written in C and is ported to many platforms. This issue also comes up when mixing Java and C code because Java has a different approach. Java set out to simplify the programming model by fixing the sizes of all data types. An integer in Java is 32 bits, always 32 bits and never anything apart from 32 bits. Java also has a long data type which is always 64 bits and has no unsigned data types at all.

Problems arise when Java code (with a fixed size data type) interacts with C code (that varies the data type size depending on the platform). Java has no pre-processor like C so it's not easy to write Java code that would, for example, use a Java integer on 32 bit machines, and magically use a Java long on 64 bit machines (and could therefore match the C code). So instead we generally use the wider data type (a Java long) on both 32 and 64 bit machines (and down cast where necessary).

At this point I should also mention z/OS which for it's own honourable reasons supported 24, 31 and 64 bit architectures...!

Wednesday, January 21, 2009

Extending PHP in Project Zero

The de facto open source implementation of PHP is written almost entirely in C. Looking under the hood shows that it is really two fairly distinct parts. First there is the Zend Engine. This is the core language engine that parses PHP scripts and interprets the resulting opcodes. 

The Zend Engine also contains the implementation of PHP types like variables (zval) and arrays (HashTable). So where are all the PHP functions and classes implemented then?

The answer is that the Zend Engine provides an extension API. This allows anyone to write classes, functions, constants and much more that plug in to the runtime. Taking a peek at the source code in CVS reveals just how many extensions there are. 

This C API is well documented in Sara Goleman's book. It talks in detail about how to extend PHP and indeed how to embed PHP in your own applications.

Zero has a very similar architecture that separates engine and extension. The main difference is that the core PHP language engine is written in Java. Zero also has a comprehensive API called XAPI-J that allows Java extensions to be written for the Zero PHP runtime. 

We use XAPI-J to implement virtually all the PHP extension functions supported in Zero (listed here). But there is a bit more to this story than meets the eye. The slides Rob Nicholson presented at last year's PHP Quebec cast some light on this. 

Zero actually ships a pretty large number of the C extensions. We do this by re-compiling the source code against a different implementation of the C API. The Zero implementation of the C API (called XAPI-C) knows how to talk to the Zero PHP runtime through JNI

There are several benefits to this solution. First and foremost we maintain compatibility with the open source PHP. It is really important to PHP developers on Zero that we are as close to 100% compatible with as possible, and using the code certainly helps! 

Secondly, we avoid re-implementing all those extension functions from scratch. Why bother re-writing everything in Java when there is a perfectly good implementation already available! There are in fact a few performance critical functions that we do implement in Java, the array functions are one good example.

If you are interested in writing extensions for Zero you will find the documentation and examples for XAPI-J on the Zero web site.

PHP and Java Collections: More PHP, Less Java

We've added some nice integration with the Java collection classes in Zero.

Java has a rich set of collection classes. The three really important ones are Set, Map and List. We use the Java collections quite a bit when integrating Java libraries in PHP applications. So it got us thinking whether we could provide some good PHP syntax for these classes. It turns out there are quite a few things we could do to improve working with these classes in PHP.

For example, Java lists now support the following:

$list = new ArrayList();
$list->add("Hello World!");

$list[0] = "Hello Again!"; 
$value = $list[0]; 
$array = (array) $list;
$check = isset($list[0]);

// Append some items to list
$list[] = "Hello Again!";
$list[] = TRUE;
$list[] = "Updated!";

Java maps and sets also get the full treatment. The reason why I like this is that it avoids lots of Java API calls. Some example code for hacking around with a Map using the Java APIs is shown below. Notice all the method calls, just so much more long winded than PHP's nice expressive syntax!

$map = new Java("java.util.HashMap");
$map->put("title", "Java Bridge!");
$value = $map->get("title");
$check = $map->containsKey("title");

More information is on the Project Zero PHP/Java Bridge reference page here.

Creative Commons picture attribute to MicMac.

Monday, January 12, 2009

PHP/Groovy Bridge

Ah, something new and shiny in Project Zero to talk about.

Zero supports three languages: Java, PHP and Groovy. It also supports a workflow runtime called Assemble and since that stores its flows as XML, and workflows are kind of like code, I suppose it could be considered an unofficial fourth language. 

Anyway I digress. Up until recently the PHP runtime was treating Groovy code just the same as Java. Anyone could compile a Groovy class to Java class files using the Groovy compiler. Those class files could then be dropped into a Zero application and used in PHP through the PHP/Java Bridge.

The problem is that Groovy isn’t the same as Java. Groovy adds lots of scripting features over and above Java’s current capabilities. Closures being one obvious highlight. This shines through if Groovy classes are used through a PHP/Java Bridge, it all just looks like Java. Take for example the following Groovy class that makes use of Groovy’s interceptors:

class Dynamic {
    def storage = [:]

  def invokeMethod(String name, args) {
  return "Hello World!";

  def getProperty(String name) { 

  void setProperty(String name, value) { 
storage[name] = value 

Another piece of Groovy can use this as follows:

foo = new Dynamic();;
foo.guff = "Hello World!";
print foo.guff;

Now see how this looks from a Java client:

public class Test {
  public static void main(String[] args) {
  Dynamic foo = new Dynamic();
  String result = (String) 
  foo.invokeMethod("sayHello", null);

        foo.setProperty("guff", "Hello World!");
        result = (String) foo.getProperty("guff");

Yuck! And the same applies if we use this class in PHP through the Java Bridge. Well the PHP/Groovy Bridge solves this problem by integrating PHP and Groovy as first class citizens - see how it looks with the Groovy Bridge:



$foo = new Dynamic();
$foo->guff = "Hello World!";
echo $foo->guff;


That’s better! Intuitive use of the Groovy class as intended. The groovy_import function is just one aspect of the PHP/Groovy Bridge. It also supports a range of other nice Groovy interop features including closures, currying and range objects. With this in mind I’ve put a slide cast up here where I ramble my way through an explanation of the PHP/Groovy Bridge.

The product documentation for the PHP/Groovy Bridge is also available here.

Wednesday, January 7, 2009

I've put some slides up here that Andy Coleman presented at the WebSphere Technical Conference in November last. They discuss how sMash PHP fits into Message Broker and some of the really sweet integration points with the PHP language. Our inspiration for this was the SimpleXML extension which provides a fantastically easy way to navigate XML documents. 

Message Broker has an existing Java API which we made heavy use of for this work. Much of the work involved was to catch method and field accesses in the PHP script and turn them into Java API calls on Message Broker. For example, take a look at the following message tree:

Here's a first attempt at navigating through this message tree to the title:

$xml = $input->getLastChild();
$doc = $xml->getFirstChild();
$ch1 = $doc->getFirstChild();
$ch2 = $ch1->getNextSibling()
$title = $ch2->getFirstChild();

Yuck! This is really horrible, lots of method calls and very little fluency. Ok we can do better, let's try again:

$xml = $input->getChild(‘XMLNSC’);
$doc = $xml->getChild(‘document’);
$ch2 = $doc->getChild(‘chapter’, 1);
$title = $ch2->getAttribute(‘title’);

Well it's a bit better but still far too much like hard work.

$title = 

That's better! A really expressive way of navigating through messages. This is only scratching the surface of the PHP integration and the slides cover many other examples such as array integration, message shredding and content based routing.