Subscribe to our mailing list

* indicates required
Close

Thursday, April 16, 2009

Does workflow always have to suck?

I evaluate CMS and DAM systems for a living, and one thing I keep coming back to is the fact that so very few of these expensive systems "do workflow" well. I think part of this may be because there are no industry-accepted standards around the kind of workflow I'm talking about (thus, every vendor reinvents the wheel). The closest thing to a standard is BPEL4People, which is an extension of BPEL (and thus too heavy, IMHO). There needs to be a minimalist standard around this domain space, something dedicated to human-facing interactions, supporting process-facing tasks optionally, not the other way around.

I think the other reason so many Web CMS and DAM vendors fail to do a nice job with workflow is that it's just plain hard. Light-duty taskflow or workflow (or "pageflow," as we called it where I once worked) is deceptively difficult to implement, especially if there's a requirement for good UIs around administration, design, and (re)configuration of workflows. And especially if there's a requirement for hot failover (being able to deal with STONITH and other messinesses). And especially if you need to support acyclic (reentrant) flows. And especially if you want to offer good extensibility APIs. And, and, and.

Most systems that support approval workflows (of the type seen in web publishing scenarios) get the basics right, but after that, not. Typically, though, the customer hasn't really thought out his or her use-cases very well before buying a system. And so begins a long process of design, test, rollout, fail, back to the drawing board.

Setting up workflows typically means developers need to touch XML, code, properties files, templates, and/or miscellaneous artifacts, often editing them by hand (since it's unusual to get good tools for this). You may be able to draw a basic flow on a canvas (although even that isn't done well, by many vendors), but applying timeout and retry policies, and handlers for exceptional conditions, may involve a good bit of "dirty fingernails" work. When you're all done, the customer thinks "Okay, done. This should last us for all eternity. Glad we never have to do that again!" But very soon, it becomes clear that a number of corner-cases that were not anticipated at the design stage need to be handled better. So it's surgery time again. Back to messing with a bunch of artifacts and their cobweb of dependencies, then finding a way to test it all, etc.

Administratration is often not well supported. How do you run a report for all workflows of Type X in the past month that either finished abnormally, didn't finish at all, or took too long? (Don't tell me "look in the logs.") What UI tools do you have for simply finding an orphaned workflow, or killing an in-progress workflow instance? How do you know if bouncing one machine (in a cluster) left one or more workflows (of potentially hundreds in progress) in an inconsistent state?

And then there's rights administration. When Sally goes on vacation, how does she give her subordinate, Bob, her "rights" in the system for a workflow of Type A but not for workflows of Type B?

The issues get sticky in a hurry. But I do occasionally see workflow systems that combine decent functionality with a usable graphic designer, or with good administrative tools. But it's hard to get a robust engine, a good feature set, good visual design tools, and good administrative tools all in one package. There are always warts and holes.

So I think the right way to look at all this, if you're a vendor, is that this presents a ripe opportunity. If you're a CMS or DAM vendor looking to differentiate, provide a superior workflow solution.

But there's also an opportunity in the market, right now (IMHO), for someone to come up with a fully productized lightweight workflow product with decent design, development, and admin tools, and easy extensibility (of course), that can be bolted onto a Web CMS or DAM system with little effort, so that customers who are using bespoke WF systems (of the kind that are so common in this industry) can move over to "real" workflow. And get real work done.

I wonder why products like this aren't more common? Again: Probably because it's hard. But again: This is an opportunity . . . for someone.

12 comments:

  1. Anonymous8:21 AM

    Thanks Kas. Readability would improve with less jargon

    ReplyDelete
  2. Gartner, Forrester, IDC and others have run conferences on it – OMG have a working group that we are part of it – and the Wikipedia entry for it is almost 3 years old – but now Business Architecture is getting its own system

    Process Master is about to bring to market its Business Architecture system, a system that has clever ways of managing a Multinational Corporations Process, Organization and Strategy models, so that you can keep a core Best Practice Library, plus specialized variants of each model for all your different locations, with full requirements tracing, change management & version control in both directions.

    This includes the ability to
    - mark some parts of some Best Practice models as ‘required’, so that they cannot be altered in individual variants
    - compare and merge related models, and even integrate different model types
    - compare and merge different versions (releases) of the same model
    - publish models in PDF, CSV and HTML, with different styles where required
    - send the models straight through to execution engines for automation
    - send the models straight through to an eLearning (LMS) for employee education & certification

    It is a fantastic system, it is simple to use and easy to deploy and the video in the link above will show you how it can deliver immediate benefits to your colleagues, and your management efforts.

    The video does not cover Business Architecture itself, it just shows you a sample of how a normal business user might use a Business Architecture system through his browser

    http://www.processmaster.com/Training-Videos/ProcessPad-Quick-Start-Guide.html

    Please have a look, and feel free to take a download if it is of interest to your research

    To sum up - its not allways the workflow system hat needs to be simple - simple needs to happen alot earlier than the point you start to think about automation.

    Yours sincerely,

    Alan Crean
    CEO
    Process Master
    E: alan.crean at processmaster.com

    PS - good blog, picked it up from following you on twitter

    ReplyDelete
  3. I think there's a lot of truth in what you're saying, Kas. Although sometimes built-in/integrated tools are not necessary. You mentioned editing by hand, for example. All workflow engines will require some kind of scripting and most developers would rather use their preferred text editor/IDE over one embedded in a CMS. Additionally, many workflow engines are XML based so the design and visualization of the workflow can be done in external apps integrated only via an XSLT.

    But, perhaps, that was your point and vendors should go above and beyond.

    ReplyDelete
  4. Anonymous9:52 AM

    Webmethods workflow? I haven't used it in years, and you have to use it with webMethods (which is fine if you have oodles of money), but you can do any arbitrary workflow with it. The tools for it in 2004 were clunky but I imagine they've improved..

    ReplyDelete
  5. Anonymous3:05 PM

    Sometimes it takes an upgrade to spoken and written language before standards come forth. Perhaps we still yet don't agree how to communicate work flows without becoming confused in the particular situation.

    ReplyDelete
  6. Anonymous3:15 PM

    I personally like Microsoft.NET's WF as it is a de-facto standard now.

    ReplyDelete
  7. Excellent post and what we've been feeling for years. Standard workflow tools don't provide the complete feature set a company needs to run anything but a basic process without getting cody and dirty.

    We built Iceberg (www.geticeberg.com) as an answer to the issues with workflow and software as a whole.

    So I'm sorry to self promote here but I reckon we're the vendor you're talking about and I'd love to give you a look at Iceberg some time to see if I'm right.

    Kind Regards,
    Wayne Byrne
    CEO
    Iceberg

    ReplyDelete
  8. Kas,

    Do you think Windows Workflow Foundation sucks too? Is the fact that WWF starts with a W enough to make it suck, in the same way the JCR starts with a J?

    Jon

    ReplyDelete
  9. Kas - I think you're spot on for the challenges around the difficulty of implementing workflows.

    I have had to (try to) do that in a number of systems using a number of different technologies over the years and it's ALWAYS very hard.

    In a nutshell, the issue always seems to be more than the technology, though - it's more that humans do not realize how complex their workflows are. When you interview users they'll explain their process to you in what seems like excrutiating (to them) detail but when you then codify that process, they realize that way too much of their process is exception-based and so easy to handle manually that they don't think of it as an "exception".

    In other words, any workflow is (in my experience) likely 25% the normal flow and 75% exceptional situations which are hard to grapple with in *any* workflow system.

    Layering on top of that the challenges with any workflow tool I've seen makes it a hard nut to crack.

    ReplyDelete
  10. I completely agree with you Kas. I've tried several solutions in mid-size companies, and we always found ourselves coding exceptions and tweaks shortly after the project launch.

    I also agree with you that there is a lot of complexity. Just creating reports of the activity is a nightmare! But I also think there is a **giant** market for this kind of solutions, probably bigger than the ERP market. I was involved in business case constructions, and the savings are measured in millions, tens of millions, hundreds of millions, depending of the size of the company and the years of the model.

    I guess a good solution may arise by designing a whole family of modular products, with a good architecture from the beginning.

    Regards,
    Pablo

    ReplyDelete
  11. Hi Kas, if a workflow product allowed its users to treat exceptional cases on the fly and to somehow "stack them" in the workflow dynamically at execution time (just like what happens in a real paper based settting), would that consitute a good starting point?

    ReplyDelete
  12. CMS Solutions could really give aide to those people that have more tasks than everybody else. availing this CMS

    Solutions could get your life run a little smoother.Thanks for this very informative review. This seems to be very

    interesting, and very helpful for the readers.
    Keep on posting!
    CMS Solutions

    ReplyDelete

Add a comment. Registration required because trolls.