On Monday the green light was given for the diploma thesis. The morning started off with different information meetings passing on some general information on the part of the head of department and the project coach. Later on in the morning a more concrete meeting between the bexee team members and the project coach took place. Although this was basically the kick-off meeting for the project, a number of pretty interesting concrete aspects about the project and architecture were discussed already. At the end everybody was at the same level what concerned the project. The rest of the first day was dedicated to get everything up and running. This meant to set up the work place and to start building an infrastructure for the project. In the first place this was building the project structure and setting up Maven to build the project.
Tuesday was devoted to once again further set up the project infrastructure and integrating it into Maven. It took quite a while to get everything running, but at the end of the day we had set up Maven in such a way basically upon one key stroke the whole project gets build, the project documentation generated and the web site is automatically updated.
On Wednesday I finished writing a document which I started the previous evening. It was about recording how to set up and configure the development environment the way we had done it the previous two days. Also we had a rather extensive discussion about the project documents that had to be drawn up until the end of the week.
The last two days of the first week were devoted to bringing to paper what we discussed the day before. For me this meant to work out a detailed project schedule for the duration of the project. Although time-consuming the formal part was the easy one. The harder part was to develop a detailed project procedure. I started with the results we wanted to deliver at the end of the project and then tried to put the activities to reach them on a timeline in MS Project.
At the end of the week the project documentation was finished and we were ready for the next phase of the project that started after the weekend.
During the first week of the project I've concentrated mainly on preparing the infrastructure for the project and writing the requirements specification. Sometimes when I had an idea for the realization of bexee I only wrote it down or used the SourceForge tracker system to leave a trace of the idea. This because it would take me too long to evaluate the idea and I wouldn't have finished with the specifications by the end of the week.
Having gathered some experience with the DependencyTool project, preparing the infrastructure of the bexee project, i.e. the project site at SourceForge and the maven build system didn't take me too long. But in any case, it takes some time to create a Maven project. Although Maven is a great project tool, it always is necessary to spend some time on it, read the maven plugin sources, configure some properties and see finally what is to be done to have the expected results. In addition, Maven always does some things by mere magic and it's up to the user to guess that magic.
It was very interesting for me to write the requirements specifications. I didn't have any lectures on project management during the exchange year in France, so I had to gather important information about how the requirements specifications have to look like from my colleagues. Finally I've had templates for the requirements specifications document and information about the Hermes project management system.
Having finished the project definition phase, it will be possible to concentrate on the bexee system architecture during next two weeks. I'm looking forward to the next phase, because we'll be able to begin with the design of bexee.
The second week of the project was entirely devoted to working out the main concepts and the core architecture of the bexee engine. Thus the main activities during that week consisted mainly in trying to come up with ideas regarding the core architecture.
We discussed a lot of different solutions and carefully analyzed their strengths and drawbacks. After hours of pondering over various solutions and ideas, and two extensive meetings with our project coach we proposed a solution that will allow us to approach the program in an elegant way. We decided to use a tree structure to represent a BPEL process and to separate the execution logic from the structure by implementing it into a separate process controller. Using a tree structure will make it easier for us to represent the process graphically or to reconstruct a terminated process. Instance specific state and data will be saved on a per instance- basis in a separate process context object.
At the same time I was also studying the BPEL specification in greater detail which allowed me to gain a deeper insight into the BPEL concepts.
There is still much that needs to be done to work out the details, but I think the solution has now been defined in broad outlines.
At the end of the week we reached a point, where we were able to write down the key ideas of our concept and draw up some UML class diagrams identifying the most important components.
The second week of the project was entirely dedicated to investigation of the core bexee concepts. It was the first week of the concept phase. We've spent all the time discussing different possibilities for the realisation of the core model, the execution and the basic concepts. Most of the times we've had many solution proposals, but it happened quite often, that after further investigation of a solution proposal we discovered some kind of lack. So it happened, we've had many incomplete solutions often covering different problems. Fortunately we've had the possibility to discuss our proposals and problems with our coach. Dr. Dubuis. By the end of the week we could finally decide on the core model to be used in bexee. We've decided to represent the BPEL language elements as interfaces without polluting them with implementation specific items. For the base implementation of this model we've chosen a tree model representing the BPEL language structure but with the possibilities to traverse it as a flow. Finally this core model implementation is a mix of the different initial proposals. The products of this week were not really tangible; we've used a lot of paper and made much delineation. During the next week we'll investigate how to execute processes on this model and we'll write down and formalize the core model structure.
In the second and last week of the concept phase we were mainly occupied on further working out the details on how our engine works. After the data structure we are planning to use had been defined in the first week of the concept phase, the principal focus during the second concept week was to define the flow of a process. This included lots of thinking about what happens exactly from the time a message arrives at the system until the response leaves the system again. These thoughts were finally captured in a number of sequence diagrams and according documentation.
We implemented some small examples of key parts in our concept to see if everything is going to work as we believed. Among other things I was involved in investigating the possibilities of use multiple threads inside a servlet container and how to synchronize them.
At the end of the week we had broadened our understanding on how the engine should work to an extent that we felt ready to tackle the problem and have a first shot at implementing.
We are aware of the fact that we might need to go back to revise our concept and make some (hopefully) minor changes to our solution outline. But we are also sure that we have reached a point where talking and thinking about the problem wouldn't have led us much further. So we are going to close the concept phase in time and are looking forward to start trying to put our ideas of the last two weeks to work.
Having defined the core data structure of bexee during the second week of the project, during the third week we've concentrated our work on how the engine will work. We've defined the core workings of the engine and the engine core model. Before starting the implementation of the engine at the end of the week we've drawn a couple of sequence diagrams for the main and most difficult scenarios inside the engine. In addition we've verbalized our ideas and we've started documenting the core of the engine. This way we've started creating a discussion base for the minutes with our professor and those interested in bexee.
As we've had many proposals for the realization of out bexee engine we finally decided to start implementing the solution in order to validate our ideas. Not being very experienced with server and engine design and implementation we started the realization at the end of the week
This was the first week of the implementation phase in our project and so we started producing some productive code for the first time. We started with defining all the interfaces that we needed. Most of if was a simple mapping from the BPEL XML schema to Java interfaces and classes.
We also started implementing the process controller for the simplest and most important activities and were already able to test if the controller works as we had expected. And it did! :)
Besides this, I was occupied with working out the system architecture. This concerned mainly how we want to use the engine from the outside and what parameters we need to pass in. We had already decided in the concept week that we were going to use Apache Axis as a framework for handling SOAP stuff. The tricky part was figuring out how we could extend the framework to suit our purposes in a clean way. This involved a lot of searching through the Axis source code as the documentation isn't too bad but not complete enough for doing the things we wanted to do.
At the end of the week I had a quite deeper understanding of how the whole Axis engine works and I was able to plug in our own classes for invoking the bexee engine in the right places inside the Axis framework without having to modify the Axis source code.
At the same time I also integrated Axis into the bexee project. I created a custom engine factory for Axis, so we can configure Axis for bexee exactly the way we want it from the beginning. Maven will now create a war file that contains everything necessary to run bexee. All you have to do is deploy the war to the Tomcat server.
Having worked on the core concepts of bexee during two weeks it became obvious, that it's necessary to start implementing this core. It was a good decision to spend two weeks just thinking and discussing how bexee should work, but continuing this for one more week would certainly be a lost of time in the scope of a eight weeks project. It was therefore necessary to implement the core functionality and the bexee core scenarios. Like that we'd be able to proof that what we've imagined for bexee really works and that the direction we've chosen is right.
Like that I've spent the fourth week on transforming the BPEL xml schema descriptions of BPEL actions and elements into java interfaces and also providing default implementations for those interfaces.
The most interesting part came after the definition and implementation of the most important BPEL activities: the ProcessController. As this is the very core of the bexee core, we've done this part in real pair programming. I liked this part really most! So we've implemented the controller in a way it'd be able to execute a very basic scenario, a simple receive, invoke and reply process. We'd create such a simple process programmatically in order to be able to execute it. On Friday, just before going home our controller was able to execute this process that was a great award and a really big motivation for the continuation!
We didn't have to change the core structure and the workings of bexee in order to run this process. There has been only a minor change in the execution process but this had no repercussions on the core ideas and the existing implementation. This change was that it wasn't a good idea to execute the processes recursively; we'd have to do it in a sequential order. It'd be only necessary to change the documentation of the core process.
After this week it was obvious what'd have to be done next week: XML-OO mapping and the integration of bexee with Axis.
The second week of the implementation phase is over and I have the impression that it was rather an unproductive one when it comes down to generating code :(
I spent a lot more time than expected crawling through the Axis source code to figure out how to integrate Axis into bexee.
I completed the Axis engine configuration factory so that it will start up with a customized configuration. This happens by reading in a modified wsdd when the Axis server is created. We used this technique for making Web Services available at bootstrapping-time.
One service that uses this technique is the Manager Web Service which I implemented at the beginning of the week. The Manager Web Service takes a BPEL and WSDL document as input parameter and deploys them to the bexee and Axis engine respectively. The new processes are deployed into the Axis engine at run- time and made available as new Web Services.
I also spent some more time thinking about how to connect Axis with our Engine and how the communication between the two systems should look like. This also included defining responsibilities in a more detailed way for the various system components (e.g. Provider, Dispatcher, etc.).
After this I started implementing the bexee provider which is the main component for connecting the two systems. It turned out to be very time- consuming and more difficult than we assumed. I think this was due to the fact that there are still quite a lot of unpleasant issues on the Axis side but certainly also due to a lack of very profound knowledge of the Web Services technology on our side.
During the last two days of the weeks I updated the documentation on my part and recorded the establishments we've made so far. This included a detailed description of the system architecture and Axis integration.
Having devoted the whole fourth week for the definition of interfaces, representing BPEL activities and elements I've headed for the xml to oo mapping problem. This is a quite central problematic in bexee, because it is not possible to solve the problem in an other way, i.e. BPEL is a XML language and it's thus necessary to map the BPEL documents to Object structures. I've had some experience with Jakarta Digester, but after a few trials it occurred not to be possible to solve the problem with this otherwise great tool. BPEL has complex nested structures, and it's not possible to predict, where the activities are embedded, e.g. a sequence might be the root activity, but it might be as well an activity within a Flow, etc.
I was already afraid that I lost a whole day invested for the definition of parsing rules with Digester, without the real possibility of accomplishing the task with it. After a brief evaluation of other XML-OO technologies and tools I've however decided to investigate throughout the Digester solution and found that it's possible to parse even complex, recursive xml documents with digester. You can find the results in the XML-OO mapping documentation.
At the end of week five I've started documenting the XML-OO mapping solution and also after some necessary refactoring of bexee documentation I've started writing a developer's guide. The main purpose of this guide is to provide a technical documentation which would make it easy to extend and complete the bexee system. This is very important for bexee because it will not be possible to implement a complete engine and other people might want to continue working on bexee after the end of the diploma project.
I'd be very glad to provide not only the bexee system but also a good documentation that'd make it easy to continue the work on bexee. Even if the bexee project will not be followed up, I hope it will be possible for other people to reuse the core ideas. Therefore the documentation is as important as the bexee system.
During the second last of the implementation phase I continued working on integrating Apache Axis into bexee. I started the week with defining how information is passed from Axis to bexee and thus defined what information we need in a BexeeMessage.
We also created a small sample BPEL process and its corresponding WSDL from "scratch", which led to further insights on how everything is supposed to work.
Because we had quite a number of issues with Axis version 1.1 we decided to give it a try with the latest RC which was released just a few days before. Hoping for the best we installed Axis 1.2 RC2. After fixing a few compilation errors in our code that resulted from incompatibilities between the versions everything looked good. Too our astonishment we got a weird IncompatibleClassChangeError when executing the code. After carefully analyzing the versions of the Axis dependencies we confirmed that everything was in order. After revising the dependencies again, we established that the JAX-RPC and SAAJ libraries between Axis version 1.1 and 1.2 RC-2 didn't change the version, but their size! After copying the version from the Axis 1.2 RC-2 distribution into our project, it worked.
To our light disappointment the newly installed version didn't result in any improvements for which we were hoping for.
I continued implementing the deployment of a new BPEL process in the Manager class. The Manager now uses IBM's WSDL4J to generate an object representation of a WSDL file and then uses this object to dynamically create an Axis service description.
At the end of the week I also implemented two simple DAO classes that use memory rather than persistent storage for storing data. As we are running out of time, we decided that memory-only storage has to be enough for right now.
We finished the week off by giving a first try at implementing the event mechanism in the ProcessController.
This week filled us a little bit with consternation and made us realize quite a lot of things. I think it was the first time that we really realized that we are not going to be able to finish what we were secretly hoping for. We realized that there is just not enough time and that there are a lot of issues we didn't think about in the first place. We realized that much more deep knowledge in the context of web services including XML, SOAP, WSDL etc. is needed for a fluent proceeding of the project.
Of course this might be a little bit of a bummer right now, but we certainly also have to look at the bright side, as we undoubtedly learned an enormously lot by attacking such a demanding problem.
The next major task after the completion of the XML to OO mapping, its documentation and the documentation of the development process for unimplemented activities is the deployment. In bexee we decided to implement a hot-deployment mechanism. This kind of deployment for business processes will be very comfortable for the user. The user only has to provide the business process description in BPEL, the Web Services descriptions in WSDL for own and foreign Web Service interfaces and to feed this information using a Web Service interface of the bexee system.
But what does this imply on the bexee side? It is necessary to provide an own registration mechanism for Web Services in bexee. The standard mechanism of the integrated Axis Web Services framework works fine for deployment and registration when the Axis Admin client is used, but doesn't work so good for programmatical registration. Then the user provides Web Service descriptions for the processes' own Web Service interfaces, but without the concrete binding information, which depends on the internals of bexee and must therefore be generated by bexee. It also is necessary to provide the mentioned deployment and management interface for bexee (a Web Service). Because of the complexity of this task, it took us much longer to do this and it was not possible to concentrate on the workings of the bexee core.
Another main task during this week was to work on the generation of the concrete bindings for the Web Services. We've found an interesting framework and a specification proposal: WSDL4J and JWSDL. The purpose of the JWSDL specification proposal is to provide a standardized way to manipulate WSDL documents as java objects, to write programmatically created WSDL descriptions and also to parse WSDL documents and provide their object oriented representation.
This specification proposal and its reference implementation, WSDL4J are easy to use and on the first sight I was convinced that it'd take me no longer that a half a day to implement the generation of the concrete bindings and services. But so it happened, it took me at least two days to really understand how this framework works, a day to really implement what we wanted and another day to debug and integrate the solution into bexee. So it happened, it took us much longer to integrate external software into bexee than it was supposed to.
This week was therefore a difficult one, because we had to notice, that it would never be possible to do what we wanted to. We wouldn't have reached our goals by the end of the project. We've learned that integrating software is a difficult and complex task.
The last week of the implementation phase is over and we are reaching the end of our diploma thesis. The weekend gave us some time to think about a number of problems, which persuaded us to take out all the XML stuff from the BexeeMessage and replace it with Java Objects. Of course this implicated that, without generating classes for complex XML type to Java transformations, there would be no support for such.
On Tuesday, we took a short break to think of a very simple process that could be used for a small demo. Because the engine is still far away from being a complete implementation of the BPEL specification, this ment that we had quite a number of restrictions on how the demo process should look like. This included having only one port type per process, only XML simple types, no correlation data and only a limited set of activities. With this restrictions in mind, we build a process that was - although very primitive and probably not very fancy - a complete BPEL process (including necessary WSDL).
We spent the rest of the week implementing the remaining parts that were necessary to make this little process work. At the end of the week we were able to deploy and run the small demo process :)
We continued working on the code and fixing some bugs and also wrote some custom Ant task, in preparation for the demo.
All in all not a too exciting week, but also one with no bigger unexpected coincidences. Apart from fixing small bugs, we now try not to spent more time on coding but rather focus on the documentation and a clean termination of the project.
