3weeks2build

Time's up, put down your pastry bags!

2008-01-21T01:10:00.000-05:00

I'm having high school flashbacks right now. The sabbatical ends tomorrow morning, and I'm up at quarter past 1 "cramming". So much for "coming back refreshed".

So I've got things mostly done. Some small functional holes still exist, but I really need some sleep. I imagine I'll be sneaking in the remaining features when i have pockets of time.

There was a lot to this GUI - and the things I'd pegged as being easy were the things that bit me the hardest. I have so much appreciation for the Information Architects (or Information Architecters) of the world.

No longer making sense. . must get sleep.

I had Raisin Bran for dinner. Not even real Raisin Bran, I had
the Malt-O-Meal knock off.

Duty Calls. . .

2008-01-18T01:27:00.000-05:00

As it turns out, the Process Designer thing I'm building is missing an important feature. That feature is what we in "the biz" call "Save". Its used to take important data that's teetering on the edge of existence, sustained only by the good fortune that electricity has managed to flow constantly through the millions of circuits that compose a stick of RAM - and securely deliver it to a much safer place, like a hard drive, whose promise of data integrity can only be defeated by bumps, magnets, static electricity, liquid, dust, above-normal temperatures or any other condition that will probably happen soon.

So anyway - Nitram's been waiting for me to get this feature finished so that he can use the tool to help him test and exercise the engine. Dragging and dropping sure beats hand-editing XML, right? Fair enough. So, like a good partner, I fired up Photoshop and worked on the UI.

I cut some new graphics for the different node types, tweaked some CSS, and upgraded the look a bit, and added some bells and whistles. I also discovered that my home-brewed method for detecting the center of a Bezier curve wasn't always working. If you remember, I'm adding a button to the middle of each of the connecting lines - well, in some cases, the button was rendering nowhere near the line. So, i had to roll up the sleeves and do this right. I found de Casteljau's algorithm which is perfect for doing just this. I'm not a heavy math guy, but i was able to come up with a cheap implementation in js:


Transition.midpoint = function(e1,cp1,cp2,e2){
   var top = [_mp_line(e1,cp1),_mp_line(cp1,cp2),_mp_line(cp2,e2)];
   var mid = [_mp_line(top[0],top[1]),_mp_line(top[1],top[2])];
   return _mp_line(mid[0],mid[1]);
 
   function _mp_line(p1,p2){
       return {
       x: (Math.abs(p1.x-p2.x)/2) +Math.min(p1.x,p2.x),
       y: (Math.abs(p1.y-p2.y)/2) +Math.min(p1.y,p2.y)}
   };
};

All this really does is take the end points and control points that define a curve, and find the midpoints between each adjacent pair. Then it finds the midpoints between each of those midpoints, then again, until there is only one point left. Thats our guy. I think this could've been implemented more slickly using recursion, but i think it would actually make it a longer piece of code. Any way here's the jing of the algorithm, and some of the new UI in action. (Please don't mind the pimpley-looking drag circles and the misplaced text)

In all honesty, I spent a good bit of time today working out the details of the "save" feature. There actually are some interesting challenges in it. Since I've decided not to auto-save or make calls to the server at each event in the gui, There are potentially many changes that will need to be carried out by the time the user saves their work. Within one save request there may be tens of actions to perform, some nodes may get updated, others deleted, others created - some will have new Actions associated with them, some Transitions will be deleted, created, or moved. So, to get ready for this I wrote it out in pseudo-code to make sure that i'll be doing things in the correct order. I'm really almost done, Nitram, I swear. Maybe if I didn't blog so windily, I'd be done with it.

Oh yeah, I had Chix (hippie chicken) patty burgers, sauteed kale and baked beans for dinner whoooooo hoooo!

Rails forms...not so hot

2008-01-18T00:23:00.000-05:00

Before running through the activities from the last two days, a word on rails forms. While this is not my first time using rails, I still fall into a state of confusion and bewilderment whenever using the forms on anything more complicated than the ideal one-object-to-one-form mapping. So far I am really displeased with form handling in the rails template language, it just doesn't seem to scale well to more complex objects.

That said, over the course of this project I've developed a pretty good approach for myself to keep my sanity. It basically amounts to two guidelines: keep it simple by not overusing helpers, think of everything as it will exist when passed into the params hash. In retrospect, these rules are fairly obvious, but they still work well as a mantra. I love the params hash, when used in conjunction the model helpers it is a very clean, JSON-like, solution. It feels intuitive to me. The rendered html, with its bracket notation (...input name="sibling[126067293][take]"...) is also very intuitive, you can "see" how it translates into the params hash. It's just the code that generates this html that sucks...hey, maybe I could just code straight html? Or make my own helpers? Need to think more about this. I wish I had more of an argument, but it's more of a feeling than anything else right now. More to come on this topic...

Yesterday and today:

Solved JSON issue (from_json was not robust enough) with dave
Played some foos!
Process engine now passes first 8 of 20 workflow patterns
Fixed process definition XML serialization bug
Process engine now defaults all spawned tokens to wait-state on any fork, event code must "continue" the tokens, helper method created to facilitate this
Controller integration

Added wait_for_user_input helper method to process_engine

Now supports custom view callbacks for controller interaction
Default view handlers are in place for forks and end-of-process
Created routing to handle process engine requests seamlessly and globally

Hey, JSON, WTF?

2008-01-16T00:53:00.000-05:00

Ok, figured out the annoying JSON bug I was having yesterday. . . thanks to my use of virtual properties to traverse the model relationships and produce the json - my sub-arrays were getting double-JSON encoded. So instead of:


{name:'test node',
id: 234323,
type: 'decision'
transitions: [
   {to_id: 432343, arrival:"top", departure:"left"},
   {to_id: 343233, arrival:"right", departure:"bottom"}
   ]
}

I was getting:


{name:'test node',
id: 234323,
type: 'decision'
transitions: [
   "{to_id: 432343, arrival:\"top\", departure:\"left\"},"
   "{to_id: 343233, arrival:\"right\", departure:\"bottom\"}"
   ]
}

Notice the difference? Neither did I.

I actually got a number of items off of my list today. One of them, "Multiple hotspot/dragpoints per node" means that creating a transition between two nodes is more than defining a "from" and a "to". . . You might want to connect the bottom of one node to the left side of the other. Of course there's no functional difference, but users would become annoyed and diagrams messy without this kind of control. jQuery and my caveman-level CSS knowledge made it pretty trivial to dynamically add to each node, North, South, East, and West drag zones and drop zones. Now why am i calling them top, bottom, left, and right in the code? North, South, East and West sounds so much cooler. I wish i had thought of this earlier. Perhaps we can save these changes and promote version 2.0 with "Improved Variable Names! Upgrade Now!" or "Code base features even more consistent indentation!"

It hit me (while taking a power drill to my laptop (in order to justify an upgrade to the MacBook Air (Thanks Jared!) ) ) that users of our system will need to be able to select transitions. Up to this point, transitions have been a purely visual thing, simply reacting to the location and size of their related nodes. Selecting could be complex being that 1) nobody wants to try to click on a line that's curvy and only 2 pixels wide. While it is possible to detect whether or not the click was along the curve, this type of user experience is likely to make all the Davie recipients weep. 2) these Canvas elements (1 per curve, remember) overlap each other like the f'endas on a waitress - so the highest z-index canvas steals all the clicks. I quickly IM'ed Nitram to see if he had any ideas. He said, "how about a circle in the middle of each transition? -And you click it"
"Well Mr. Smarty-Pants, How's it feel to always have things like 'Answers' and 'Ideas'? I hope that works out for you!" So, i took his weak stump of an idea and reworked it tirelessly and I came up with this: Each transition will have a circle in the middle of it. And you can click it. This action will bring up the properties for that transition. This didn't solve the overlapping canvas problem - so what i've done is this: Mid-points for all transitions are determined and mapped to absolute page coordinates. Clickable DIVs are placed at these locations at the highest z-index. Seems to work so far. I can't wait to put a real skin on this thing. I hope i have time for that.

Unfortunately, each line I strike off, has me adding a line or two. My list currently looks like this. it may not make sense to you but, i mean, its MY list.

build/integrate graphics
finish: make transitions selectable/visually selected
finish: building property editing
finish: saving nodes.
panning / zoom
add more quality
self-transitioning nodes.
stop scratching it

Ok. I've gotta go, my laptop still works.

PS - I inadvertently got "fancy" tonight and ate a green curry w/ chicken out of a real coconut @ a local restaurant tonight. It wasn't what I ordered, but its what they brought me and what i had to pay for.

Another day...

2008-01-15T20:17:00.000-05:00

Today was mostly research along with several disturbances. I looked through the rails controller api, messed with routes.rb, and learned more about the ruby language, which was great. I was sidetracked by watching the Macworld 2008 keynote. Then, I saw this article in the nytimes which led to this paper which led to me wondering who in physics is actually performing scientific experimentation anymore...and on and on until I came to the fundamental meta-conclusion that the chief danger of our information saturated culture is tangential thought...and that I probably have severe ADD.

Update:

Shaved nascent sabbatical beard this weekend. Fiancee Melissa gave it a thumbs down for our anniversary date. Not that I blame her, it was looking pretty bad and was itchy as hell.
Cleaned up the model by adding a few polymorphic associations.
Started work on process-engine + controller interaction.

getting there.

2008-01-15T01:10:00.000-05:00

Well, I've been chugging through my side of things albeit somewhat slowly. Spent most of the day working with Nitram talking architecture and planning our last steps.

We made one particular change that took a little time. For each Node in our model, we have a corresponding Layout object. It contains the visual information needed to represent the node in the designer. The Node and the Layout have a one-to-one relationship. Each of the transitions in our model have a corresponding TransitionLayout object. I have been generating JSON to represent the Layout portion of the PDL for transmission to the client. Layouts used to have a direct relationship with TransitionLayout objects so building the json for this portion of the model was simple. "to_json" allows me to recursively traverse the object model pulling exactly what I want along the way. But now, We've updated the model like this: Instead of having direct relationship, A Layout and its TransitionLayouts are now only related through their corresponding Nodes/Transitions. In other words, a Layout belongs to a Node which "has_many" Transitions, and that transition, "has_one" LayoutTransition. This I believe is a better model, but does not work as nicely with the rails to_json method. Eat butter in moderation. So now I'm using "virtual properties" to do my traversing and having to_json pick them up as methods. This seems to be working out alright, although I've got to work out some oddities in the resulting JSON thats killing my JS.

The front end is coming along nicely and is pulling up processes from the DB and displaying them. A couple of small steps until we're able to update and save. Then, I think I'll be giving it some real graphics and get it looking smoov.

Had leftover fajitas - wasn't in the mood, so i wrapped 'em up and pretended they were burritos.

Time for bed.

Pier process engine is alpha...and an assortment of prophetic remarks

2008-01-12T17:48:00.000-05:00

I am excited to announce that the Pier process engine for Ruby on Rails is alpha. I am defining alpha as a "functionally whole" system, in OS jargon: we have a kernel. Now that the process engine is alpha, we can begin end-to-end testing using the process patterns outlined here, and start building feature enhancements. In its primordial alpha state the rails process engine can (and you may want to sit down for this):

Read and instantiate a serialized process definition.
Create and load a token with contextual application data.
Launch the token into a process flow
Execute the flow: creating forks and merges, executing embedded logic

Wow, pretty amazing stuff, no? Well it is! The great part about reaching this stage is despite any the future headaches, strange bugs, and the inevitable its-just-too-slow refactoring, we have a foundation to work off of and a number of proven concepts that are past the point of no return. It's alive.

Great, so what's next? The goals for next week are to build out the engine and integrate more closely with the rails framework. First, in keeping with the test driven approach, I will build and execute the process pattern tests mentioned at the beginning of this post. Running through these detailed tests and exercising the system will likely reveal missing features and a host of bugs. Fun!

Next, one major requirement that I already know the tests will reveal is the engine's ability to collect user input at runtime. In order to facilitate this, I will need to extend the rails controller with process engine power. In terms of MVC architecture, and in the context of a single application, the process engine enhances the controller. Certain processes may need user input to continue, so my goal is to create a seamless and intuitive connection between the process engine, rails controller, and application views.

As a final note, I'd like to set aside the process engine sub-project and take a look at the big picture. Pier is a BPM solution, not just a process engine. Dave is doing incredible work on the process designer side and I'm really excited to see the two sub-projects working together.

Once these sub-projects are fully integrated and tested, Pier will be capable of bringing agility to the enterprise application development process. Our goal is to enhance business software planning and development by providing Business Analysts, Information Architects and Techs an intuitive "living" picture of the system as it grows from a couple napkin doodles to a production ready application.

On a larger scale, I think Pier will be an essential piece of the emergent group of light weight, service oriented solutions built off the Rails framework. These solutions will promote and enable the development of multi-application systems. As everyone already knows, web-applications are talking to each-other. SOA was developed in the business world to define and encapsulate this phenomena. Public facing sites are quickly standardizing and catching up. One of the "third-arms" in the evolution of multi-application systems will be applications that do nothing more than orchestrate other applications ( Yahoo Pipes is an example of this concept). Pier is being built with these SOA concepts in mind.

Front End, meet Back End

2008-01-10T23:53:00.000-05:00

Yesterday and much of today I got the front end to the point where it's ready for me to start trying to integrate it with the server. As hard as I tried, i couldn't put off learning ruby/rails any longer. So starting this afternoon, I've been shoving migrations, YML fixtures, ROXML, and the like down my throat. I learned a lot today, but contributed very little.

I built up the object model that represents the "layout" portion of the PDL. Not much there, but its been a good way for me to start learning this. Finally by tonight I was generating the correct xml through Martin's badass plugin. Looks like sending JSON to the client will be a breeze with the highly-flexible to_json method that comes with Rails.

Tomorrow, I think i'll actually be able to get the Client and Server doing some interesting things.

If I hadn't waited 'till so late to blog, I'd be a bit wordier.

Goodnight.

PS. right, umm I made chicken fajitas for dinner. They were "eh".

Process Engine...in progress

2008-01-09T21:45:00.000-05:00

Today was a lot of coding, and a few headaches. ...I'll have a more complete post tomorrow.

Finished writing process engine spec tests.
Started building process engine.
Discovered and resolved, after much confusion, a bug that killed active record lazy loading for the important Token object. Caused by accidentally overriding an ActiveRecord::Base method.
Refactored several Token methods to use the more Rails-y hash arguments.

Front End in Sight

2008-01-08T23:56:00.000-05:00

Today was another js day. I decided to try out the jQuery UI package to handle the drag and drop stuff I'll be dealing with. Battled an interesting bug (i think) but I'm simply too tired to go into it now. But if you kids are good, I'll tell you all about it tomorrow along with the story about the time I won 30 bucks arm wrestling your Aunt Janie.

Well, I've finally got a bit more to show. You know, something to ~~make you think~~ prove that I've been working so hard. I've just got some basic node manipulation happening in the UI. Although its pretty dumb (and ugly) right now, i think some of the hardest stuff is done. Have a look. Jiiiiiiiiing! Obviously, transitions need arrows, each node type needs to look different, and i need to add more hotspots to each node so that you can specify that the transition goes to /comes from the right/left/top/bottom of the node.

Today:

Started using jQuery UI
Battled and beat mystery bug
Got basic node manipulation working (along with a couple of rounds of optimizations)
I'd rather not get into dinner, thank you.

Tokens

2008-01-08T23:20:00.000-05:00

UPDATE: For an intuitive demonstration of Petri Nets: http://www.informatik.uni-hamburg.de/TGI/PetriNets/introductions/aalst/

I spent today growing the Token object and I think I've made it to a mostly-stable beta version. The token is the developer's gateway to interacting with runtime process state. The current version is capable of:

Answering: where am I? (position in the workflow map)
Answering: what state am I in? (waiting, continuing, dead, lost in the void)
Answering: what is my parent/sibling up to? (present, elsewhere, alive, how many, etc)
Spawning children (when tokens hit a forked process the process engine will create child tokens for all possible transitions)
Answering: do I have cargo? Add/Remove cargo. (application data passed through the workflow)
Making a normal application object conveyable (it can ride a token)

The token object was built off the token spec (token_test.rb). In the next version, I'd like to add another chunk of functionality that will allow the Token to keep a historical record of its movement.

ProcessEngine is next...

Today:

Sabbatical beard made it out of the "stubble phase". This is dangerous territory, not yet lumberjack beard, no longer j. crew stubble...no rationalization to provide fiance Melissa
Learned that my cat loves crispy cheddar squares
Found that my cat's stomach has not evolved to properly digest crispy cheddar squares
Finished Token spec
Created Token, TokenCargo, TokenPlace and a bunch of exception classes
Created test classes TeaBox, Tea (for making_tea process)
Began spec for ProcessEngine

Blog Title

2008-01-07T23:55:00.000-05:00

I wasted a good bit of time over the weekend trying to efficiently calculate the minimum bounding box for a given bezier curve. Since I'm drawing each curve (or transition) on its own canvas, I'd like to keep each canvas as small as possible while safely containing every point along its line. After reading through a couple of algorithms, it hit me: "I don't understand a damn thing I'm reading!" This hit me too: "Its the weekend!" Given this bit of news, it was time for a new game plan.

I think I'll just be sizing each canvas element to contain all of the control points used to describe the curve. The elements will in many cases be much bigger than they need to be, but it probably won't matter anyway. We'll see.

Today:

Got all Smikey and re-did my wireframes for the app.
Checked out Nitram's commit and got it running locally.
Started coding some the base js for the designer tool.
Had my "front-end wires" reviewed by Nitram. (I recommend it)
Had some leftover taco salad. Not enough ground turkey left - had to supplement it with ground beef. (this, not so much)

Initial Checkin

2008-01-07T20:03:00.001-05:00

Last Friday Dave, sorry...Divad and I sat down to finalize the PDL and designer plans. We were able to come to enough of an agreement about the initial PDL spec, designer elements and layout that we could begin coding this week.

My thoughts on planning: enough is enough. The process I will be taking going forward is going to start looking a little more agile and will generally follow the BDD paradigm. I'll be using unit tests to spec out object behaviors and move through a concrete set of iterations, each about a day long. The goal for this week is to have a prototype of the process engine ready so that I can spend the next week polishing and extending its core features.

The schedule this week will look something like:

Monday: Initial project setup and check-in with core PDL objects.
Tuesday: Build token - Tokens move through the process map, responsible for contextual data and describing process state (work).
Wednesday: ProcessEngine - Interface to controller layer, responsible for moving tokens through the process map (flow).
Thursday: Asynchronous process execution.
Friday: Exception handling and misc cleanup.

Today, start of week 2:

Started super hip sabbatical hermit beard. Contemplating the pros and cons of bathing...and clothes.
Upgraded to Rails 2.0.2
Tested acts_as_roxml in 2.0.2 environment. No problems.
Created PDL object model and unit tests (checked in under rails_sandbox/pier)
Started Token implementation
Reviewed Dave's front-end wireframes and posted feedback.

Day 2 - Rescue 'copter still not here. . .

2008-01-03T23:48:00.000-05:00

TODAY WAS CHILLY.

Speaking of today,

Today, I:

Researched client-side implementation ideas. Initially, I was thinking that I'd build the designer as one big Canvas element and do all the drawing and event handling through it. But I quickly learned that Canvas element doesn't support the drawing of text. There are, however some workaround libs floating around, but it feels to early in the process to be hacking in fixes. Instead, nodes will be represented by standard Divs and transitions will be dynamically resized canvas elements at a lower z-index. A quick firebug session tells me that Yahoo Pipes is doing something like this as well.

Came up with a little formula for drawing the bezierCurves that will represent transitions in the designer. Look here: http://screencast.com/t/MZue42IvfZ

Started working out a way to achieve some level of "resolution independence" with the layout tool. I'm thinking of using a percentage coordinate system to describe the locations (and sizes?) of nodes. These will be percentages of the minimum rectangle containing all of the nodes. This will potentially make printing or rendering to other devices a bit simpler, but mostly I don't want the layout data in the PDL to be pixel-centric.

I imagine that a complicated process will need more real estate than a 17" monitor will offer. So I'd like to offer a zoom/pan function to control the users's view of the process. But before I spend too much time on this, I'd like to publicly remind myself that this is what scrollbars are for. But the zoom, that's still a good idea.

Created initial wireframe for the designer

Had a bowl of Moroccan Stew for dinner. Didn't even need a side dish, it has rice built-in.

my first rails plugin: acts_as_roxml

2008-01-03T23:46:00.000-05:00

NOTE: Updated on 01-07-08 with Rails 2.0.2 info

In order for our little BPM to function we need to be able to marshal and unmarshal XML quickly and easily. I spent the day researching and coding a solution to this problem.

Rails Active Record models come baked in with fairly flexible to_xml and (as of 2.0) from_xml methods that allows you to serialize/deserialize model objects into transportable and editable xml documents. Unfortunately, the to_xml method (unlike to_json) does not support n-depth serialization. For example, you cannot serialize something like:

Books
- Philosophy
  - Plato: Parmenides
  - Nietzche: Beyond Good & Evil
- Fiction
  - Eliot: Middlemarch
  - Tolstoy: War and Peace

to_xml only allows the serialization of 1st and 2nd level, the individual books would be left out of the document. Additionally, I am not pleased with the method approach, I want to be able to add new attributes to my xml without having to change each instance of to_xml/from_xml. I also wanted more control over how my XML is structured.

The ROXML approach is closer to what I had in mind. I wanted the power and simplicity of ROXML baked right into my model objects. Simply including ROXML was not an option since it overrides the Active Record accessors. So I grabbed the ROXML source, made a few modifications and came up with acts_as_roxml.

acts_as_roxml allows you to add the annotations described here while providing a couple additional callbacks and helper functions that make it easy to make the following transformations: DB -> Object -> XML -> Object -> DB

The source can be found in our svn server under rails_sandbox/acts_as_roxml. To see it in action run: ruby test/unit/acts_as_roxml_test.rb I'll continue to tweak this while building the process engine. We may wind up with an extra plugin to contribute to the community.

Today:

Built acts_as_roxml plugin

Getting Started . . .

2008-01-02T21:54:00.000-05:00

I've never been one to blog, but I'm going to give it a shot.

In the interest of maintaining privacy in a public forum, I've created a "blog name" by running my "name name" through an intensive encryption algorithm not unlike those employed by the Department of Homeland Security, the FBI and formerly, the KGB.

As Nitram* mentioned, we're building a big businessey program based on the RoR framework. This decision made clear my priorities:

Learn Ruby
Learn Rails
Learn all about BPM
Try to code something before the sabbatical ends.

In order to cleanly build a visual web-based Process Designer, I will abstract much of the client-side code into two jQuery plugins.

The first of these will provide a low-level API into the drawing functions of the browser's Canvas element. Methods will be along the lines of "drawLine", "drawRect", and "drawCurve".

The second plugin will provide higher level functionality for drawing and positioning workflow symbols. Methods will include "createDecisionNode()" and "drawTransitionTo()".

Today:

Got rails running on my mac. (less fun than I'd expected)
Worked through some docs and a sample shopping cart app.
Reviewed/discussed PDL specification.
Started investigating the introduction of layout data to the PDL specification.
Had taco salad for dinner. Started off with too much hot sauce. Couldn't taste subsequent bites. Will refactor.

* also encrypted

Happy New Year

2008-01-02T18:56:00.000-05:00

...from the AI rBPM team. Our sabbatical proposal is to build a RoR BPM plugin that we can leverage for future process-intensive rails projects. Additionally the BPM will form the cornerstone of a larger service oriented framework for use alongside rails. While a few tools like this already exist, we believe competition and new ideas are good, and that the existing tools lack...to put it nicely...a certain rockstar quality.

Development will consist of two parts: a process designer (Dave) and a process engine (Martin). Our challenge for this week is to finish the Process Description Language (PDL) specification so that we can start coding.

The proposal research gave us a clear theoretical and practical direction to take. We are using Petri Nets as our underlying process theory and jBPM as our practical model. Additionally, we have several high level requirements including: no logical operators or conditional statements, documentation quality diagrams for business analysis, PDL is able to describe P4 patterns.

P4 or The Process Four (http://www.workflowpatterns.com/) is the process equivalent of the Gang of Four in Object Oriented software (Design Patterns). The P4 describe a set of process patterns that encompass anything a mature BPM would encounter. Most existing languages (BPMN, BPEL) do not support all patterns. In the spirit of TDD we are using these patterns to evaluate our PDL and test its elasticity by implementing them and modifying where needed. Eventually these implementations will form the foundational unit tests for the BPM plugin.

Today:

* Finished initial working draft of the PDL specification.
* Begin testing of ROXML + Active Record model object marshaling/unmarshling within the Rails environment. We will have a functional prototype posted tomorrow (if ROXML actually works) for any who are interested.
http://roxml.rubyforge.org/