a-soa.com

I’ve been working a lot lately with Semantic Web technologies. In particular I’ve been reflecting on the profound impact of basing everything on URIs. At one level it doesn’t look much different from primary keys or universal ids or GUIDs, but at a number of levels it is quite different.
I might talk about that difference in this article, but for now I want to talk about how systems look different after you’ve been marinating in these concepts for a while and, in particular, how we rely too much on location in our systems.
Let’s start with your typical desktop operating system. Let’s say you want to change a classpath variable so you can run a particular program (from anywhere on your computer!). For starters, the classpath variable is just a variable to tell the operating system where something is. (You still have to identify the program you want to run, but this makes its name slightly shorter.) Imagine that your computer was (or had) a big triple store instead of the anachronistic hierarchical file folder system we’ve come to know but not really love. We would identify our program by its URI, refer to it by the URI, and not really care where it was as long as we could get to it. (This is making your desktop look a lot more like the web rather than vice versa.)
“But wait!” you say, “What if I want two different versions of the same program?” This is one of those great abuses of the file system that we’ve gotten so used to that it doesn’t even seem perverse. (We do the same thing with our documents and spreadsheets: put them over here and they mean something different or are different versions even though they have the same name.)
If they really were two different versions they would have two different URIs. If you want to keep track of which one is newer, etc., this would be far easier with some triples than “knowing” which folder is more up to date.
The more amusing part of this location-based identity thing is the other half of the classpath problem I started with, not what it is (a pointer to a location where you can find something else) but where it is (where is the classpath variable itself). “Well, that’s easy. You just click on the ‘start’ icon, click on the Control Panel menu choice, select ‘System and Security’ then choose the ‘System” bringing up a new window (basic information about your computer). Over in the left hand menu, there it is: ‘Advanced System Setting.’ Another window pops up ‘System Properties.’ Just go to the ‘Advanced’ tab and push the ‘Environment Variables’ (another pops up), scroll down to find the variable you want to set, click the ‘Edit’ button, and there it is!“ (Why is it that all the variables you want to change are behind at least two and occasionally four layers of “advanced”?)
In the first place, I have no idea where it is. All I know is a navigation procedure through a bunch of UIs that will lead me to it. This is one of the simpler ones. I’ve been dealing with this kind of stuff a lot lately and have come to the conclusion that UI designers are frustrated game developers. The simplest thing involves a journey through several rooms (forms), down all kinds of hallways (buttons and menus), and picking up all kinds of magic feathers (click this check box in order to get the button enabled). Game developers (sorry, I mean application developers — I’m getting them more and more mixed up the more I use applications) compete on trying to make the navigation “more intuitive.” The enduring popularity of “classic” interfaces is a testament to how well this isn’t working. Maybe we need this multi-layered, multi-display type interaction to manage our way through the vast complexity. But maybe search and meaningful names would help. Maybe the UIs could be reduced to a tutorial for the first time through and then be done.
What if I had the URI of the classpath variable? I could just set it. No navigation required. Of the several thousand settable variables (think about it — there are over a hundred just on my print driver), I regularly use maybe a dozen. How cool it would be to have a docked window with those dozen variables I change a lot, and the ability to click and change them. And a search box to find the ones I don’t use as often.
Until that day, if you happen to see me wandering the halls of “Control Panel,” don’t forget to give me the sword that gives me three new lives. You never know when you might need them.

Posted in: Business, Software by mkumba /
2 Comments

Last week over lunch, my 18-year-old son Eli asked me, “Why are scanners so slow? They don’t even have as much to do as a copy machine. The copy machine has to move paper, put ink on the page. The scanner only has to scan.”

He was referring to flatbed desktop scanners; we have a couple at work and one at home. I’m not sure where this observation came from, but he was right.

My first reaction was to explain all the extra things the scanner is doing that a photocopier doesn’t do (allowing you to select the area you want scanned, de-skewing, scanning at different resolutions, optical character recognition, etc.). But as we talked I came to understand what’s wrong with these devices and why I don’t like them.

As Eli said, they are unnecessarily slow. But they are also unnecessarily complex. Each of the ones we have has three or four buttons on the front. Then there is the user interface. You can tell (just from the panels, menus, forms and widgets) that a lot of work went into these UIs, but that doesn’t make them useful. I like to think I’m some sort of power user for most things related to PCs but I won’t invest the time needed to master this software just to scan one or two documents per month. I’ve found some combination of buttons and sequence of choices on the multiple screens that give me an acceptable outcome. Most of the time. God forbid someone uses it in between my occasional use, because then invariably the settings are changed and it takes me about as long to find the scanned document in my file system as it would to type it.

So Eli and I started designing the desktop scanner of the future. Scanner manufacturers: you may have this design free of charge (send me a prototype if you’d like).

The scanner has one button: “Scan.” (We went through a few designs where you could pick resolution or color, but as you’ll see those distinctions aren’t worth making). This business of “warming up” is pretty lame; just have an on/off switch. When you turn it on, it should warm up.

So with the machine warm, you hit the “Scan” button and a screen pops up on the desktop with the image in full color in the highest resolution the machine is capable of. (I can hear the engineers saying, “It’s so wasteful to scan the whole flatbed if all you want is a photo or receipt,” or “It’s wasteful to scan in a higher resolution than you need.” Get over it. We’ll waste a few machine cycles to save some real time.)

The desktop application is just the image with a simple menu: you can save the image anywhere you’d like in any of dozens of well known formats. You can print it. And you can do some pretty standard image manipulation such as clipping, reducing resolution, or adjusting color. Then you could pipeline the image over to some other program (either included with the scanner or whatever you have installed; imagine, “send to…” Photoshop or an OCR program or your email as an attachment).

That’s it. This would be an incredibly more useful product, cheaper and easier to build and program.

This reminded me, I had the same reaction to the user interface for the digital dictation device I had. I know they spent a lot on the UI. It would have been more useful if at the moment the device was plugged into the USB it just appeared in the file explorer window.

You might be wondering why I’m writing this in our SOA blog.
The first reason is a desire to make the world a better place. I would buy another scanner like this even though I already have three, because of the ease of use I’d get from it.

But there is a broader message, one that echoes our credo at Semantic Arts: Let’s start taking complexity out of our systems. Most services and most applications get worse as more “features” are added. Our prescription for software: figure out the essential raison d’être of each service, get that right, and leave out all the other crap.

This will work for devices, too. Eli will thank you for it.

Posted in: Agility, Business, Software by mkumba /
No Comments

We’ve been promoting an idea lately that has been around for a while but doesn’t seem to get much press. We’re calling it “headless apps” but it likely has other names out in the wild.
We think it’s a major change in emphasis for how applications ought to be built. For the last, almost 20 years, the two prevailing modes for building applications have been either to use use cases, or for agile developers stories. We’re not suggesting abandoning either one, but recognize that both tend to subtly emphasize user interface lead design. Most use cases devote a lot of their spec to the way the user will interact with the application, which is primarily UI design. And the agile approach tends to focus on what the sponsors will see and how their experience will change.

Consider this, though: most of the code and most of the effort in application development goes into the UI. As a result the UI becomes harder to change, and you need to get consensus among the users if you’re going to mess with it. Further, the business logic tends to get marbled in with the UI making it also intractable. Finally, most developers aren’t all that good at UI design.

The headless app approach says: design the business logic (validation and side effects) needed to support a business interaction (and event or a request). But only provide an API (these days a RESTful or WSDL API). For testing and exercising you can build a “humble interface” (similar to Michael Feathers’ Humble Dialog ), but it’s really only a scaffolding interface. Then turn the UI / Web Developers loose on it. They can build mashups, they can make several different stylistic approaches to the same functionality and allow users to choose what would suit them best for the task at hand. The same user may use different UI’s for the same task in different contexts. The UI developers are free to experiment uncoupled from the business layer, as they can’t really break anything. In many ways we think this was what was intended but rarely achieved with the classic three tier architectures.

Posted in: Agile, Agility, EA, ROA, SOA by mkumba /
1 Comment

[The names have been changed to protect somebody.]

Once upon a time there was a firm. The firm had many, many employees and of course had payroll and personnel systems. One day the firm was visited by software vendors who convinced it that its systems were not “state of the art” and that if they were to change systems they would adopt “best practices.”

The clients decided that they didn’t want to have anything other than best practices so they decided to implement the new system. When the vendors told the firm how much this would cost, the firm decided that they would have to do a “rigorous cost benefit analysis” and that the project would have an “ROI.” This turned out to be a lot harder than anyone wanted to admit. No one was willing to sign up to head count or any other significant cost reductions. And frankly the existing systems weren’t all that bad.

So, instead they decided to rely on “gap analysis” and some “essential features” to sell the ever growing plan. When all was said and done the “essential feature” that everyone agreed to was “support for collective bargaining.” As it turned out, the existing systems just weren’t up to the task of supporting the upcoming major collective bargaining session. There were many other efficiencies touted in the very large feasibility study binders as well.

So the vendors toiled away. And the firm’s internal staff toiled away. Budgets were of course busted. But over the protests of many people involved, the system went live. Very few of the anticipated efficiencies were realized. And a surprising number of unexpected inefficiencies were suffered. Many of the divisions of the firm had to bring on additional staff to handle all the “efficiencies” of the new system.

And what of the collective bargaining? It turns out that while the new system was overrunning budgets and slipping schedules, the existing staff built the extra functionality needed to support collective bargaining onto the old system. Just in time for the collective bargaining (and also just in time to get replaced by the new system).

And nobody saw the irony in this. But they lived happily ever after anyway.

Posted in: Business, ROI by mkumba /
No Comments

I’m guessing that many of you puzzle over the same thing I do: “Why do large IT projects cost so much?” As we now know, it’s not the development costs (the development is done) nor the licensing costs (the software licenses are typically a small portion of the total cost).

There are many other factors, but the one that I think contributes the most is the cost of unneeded change. Why would people make “unneeded changes”? People make unnecessary changes when, in order to make a desired change, they introduce a number of other changes that weren’t really needed but “came along with the solution.”

It would be as if you wanted to paint your house blue, and someone told you they had blue walls, and in order to get your blue house you needed to replace your non-blue walls with the new blue walls. Most of us wouldn’t fall for this on our houses but it happens a lot with information systems.

Whenever a vendor says that you should implement “best practices,” they are often onto something. There is some area of your evolved systems that is far from optimum and could stand to be improved. What they don’t tell you is that the cost of this improvement is often replacing many other processes and procedures with others that are scarcely better, but disruptively different. And many of the existing procedures, while only marginally worse than the new ones, have been shaken down in use over decades to adopt to your peculiar set of circumstances. It’s this great number of unnecessary changes to existing processes, procedure and retaining people, plus the hidden cost of rediscovering the accommodations that the existing procedures have made that run up the cost of large implementations.

Posted in: Agility, Architecture, ROI, Software by mkumba /
No Comments

Most software projects start with an architecture. And most architectures are “stacks” as in “this is what our stack looks like.” This is where middleware, tools, languages and the like get decided. Two interesting things happen here.

The first is the “platform wars.” Vendors of middleware and tools are very interested in which stack gets adopted, since that is what generates their revenue. As a result they and their “ecology” (the other vendors who are dependent on the platform) promote the virtues of their “stack” over the other available options. By the way, there aren’t one or two “stacks” but there are some families of stacks, for instance Java and .NET have each engendered stacks, but there are variations on stacks due to databases, or functional components such as ESRI’s ARC GIS or Primavera’s Project Management Suite. The result of these wars is to be very exclusionary. Once you select J2EE you won’t be hiring a lot of .NET programmers.

The second interesting thing is “stack dependencies.” The stack looks so neat. You pretty much accept the whole thing. One of the things that caused it to win over the other stacks was “integrated functionality.” However, it’s this integration that causes lock-in and all kinds of other problems. Maybe you’ve selected .NET and ADO.NET as part of your stack. Now you probably have MS SQL, and initially it’s not a hard dependency, but you start to rely on the integration and pretty soon you’re locked in.

This tight integration within a stack actually gets in the way of integration across stacks (between your applications).

Now, I’m not suggesting you build applications without middleware tools or languages (that’s even too good for me to believe). What I am suggesting is in as many places as possible your stack says “Don’t Care.” As in “I don’t care what you do in this part of the stack.” If you follow through with this you’ll get flexibility at a pretty low premium.

Posted in: Agility, Architecture, Business, EA by mkumba /
No Comments

In the late 80’s, I was introduced to the “Don’t Care” Architecture by Sherman Woo of, what was then, US West (now Qwest). The Internet existed but the World Wide Web didn’t. Sherman was spearheading something he called the “Global Village.” I don’t remember a lot of the specifics of it, although I do remember the team room he set up in Denver was one of the most eclectic and creative places I’d ever seen (sort of anti-board room, no right angles, projectors, videos and white boards all over the place), wired to other US West locations.

What really struck me was how he described his technology stack. At each point in the stack, from the lowest layer (Token Ring v. Ethernet? “Don’t Care”) to the higher layers (Macintosh v. Windows v. OS/2 v. X Windows? “Don’t Care”) He would just repeat that mantra. And the more things you don’t care about the more flexibility you have.

It’s tempting and comforting to specify everything in your stack, but it’s empowering to not care.

Posted in: Agility, Architecture, Business, EA by mkumba /
No Comments

The way we build and deploy software is deplorable. The success rate of large projects is well under 50%. Even when successful the capital cost is hideous. In his famous “Mythical Man Month,” Frederick Brooks observed that complexity comes in two flavors: essential (the complexity that comes from the nature of the problem) and accidental (the complexity that we add in the act of attempting to solve the problem). He seemed to be suggesting that the essential portion was unavoidable (true) and the larger of the two. That may have been true in the 60’s but I would suggest that most of what we deal with now is complexity in the solution.

Let’s take a mundane payroll system as a case in point. The basic functionality of a payroll system has barely changed in 50 years. There have always been different categories of employees (exempt and non), different types of time (regular, overtime, hazardous, etc.), different types of deductions (before and after tax), different types and jurisdictions of taxes (federal, state, local), various employer contributions (pension, 401K, etc.), and all kinds of weird formulas for calculating vacation accrual. There have always been dozens of required government reports, and the need to print checks or make electronic deposits. But 50 years ago, with tools that today look as crude a flint axe, we were able in a few dozen man months to build payroll systems that could pay thousands of employees. Nowadays our main options are either to pay a service (hundreds of thousands per year in transaction costs for thousands of employees) or implement a package (typically many millions of dollars and dozens to hundreds of person years to get it implemented).

I’m not a luddite. I’m not pining for the punched cards. But I really do want an answer to the question: what went wrong? Why have we made so little progress over the last 50 years?

Posted in: Software by mkumba /
No Comments

A while back, I was working for a large consulting firm. When I was returning to the US from an overseas assignment, I was allowed to select the city I would return to. I told my boss, who was on the board of this firm, my choice. He counseled against it as apparently the office was being hollowed out, having just hosted the largest project writeoff in the history of the firm. (This was a while ago so these numbers will seem like rounding errors to today’s consultants, but I think the lessons remain the same.)

When I found out that we had written off something like $30 million, I asked how anyone could possibly miss a project estimate by that much. He said, it’s pretty hard, but you have to follow this specific playbook:

• The consulting firm starts the project, creates an estimate, staffs up the project and goes to work (so far this is pretty much like any other project).
• At about the time they’ve burned through most of the budget, they realize they’re not done, and not likely to finish anytime soon. At this point they declare a change in scope and convince the client to accept most of the projected overrun. Typically at this point it’s projected to be about 50%.
• As they near the end of the extension it becomes obvious that they won’t hit the extended budget either. Senior management in the consulting firm recognizes this as well and sacrifices the project manager, and brings in Project Manager #2.
• PM #2 has a very standard script (I don’t know if there is a school for this or if they all work it out on their own): “This is way worse than we thought. It’s not 90% complete (as the outgoing Project Manager had said). It’s not even 50% complete.” New estimates and budgets are drawn up, the client is apprised of the situation. The client has a lot into the project at this point, but also is very reluctant to pay for all this mismanagement. Eventually both parties agree to split the cost of the overrun. The total budget is now between 250% -%300 of the original.
• In order to spend all this extra budget, and to get some new much-needed talent on the team, PM #2 brings in more staff. If the project completes in the new (3^rd) budget (and sometimes they do) you have a reluctantly satisfied client (at least they got their system) and consultants (even at half price for the last portion they were making money).
• Alas, sometimes even that doesn’t work. And when it doesn’t, back to the playbook. Bring in PM #3. PM #3 has to be very senior. This has to work. PM #3, in order to maintain his or her reputation, has to make this troubled project succeed.
• PM #3 doesn’t miss a beat. “This is way worse than we thought…” (almost any number can be inserted at this point, but 400 – 500% of the original is not out of range. ) At this point there is no more going back to the client. They consulting firm will eat the rest of the overrun. PM #3 will make sure the new number will absolutely assure success. The consulting firm accepts the write off and finishes the project.

That is pretty much the playbook for how to run over a project by that amount. You might well ask, how did they manage to run over in the first place?

Tolstoy said, “Happy families are all alike; every unhappy family is unhappy in its own way.” And so it is with software projects. Each seems to go bad for a different reason. And if you do enough, the odds will catch up to you. But that will be a subject for another post.

Posted in: ROI, Software by mkumba /
1 Comment

The December 2005 issue of the Harvard Business Review has excellent articles by two of my favorite business authors, Geoffrey Moore (”Strategy and Your Stronger Hand”) and Clayton Christiansen (”Marketing Malpractice: The Cause and the Cure,” which is applicable as we start looking at commercializing Semantic Technology).

Moore’s article has many fresh insights; chief among them is that companies have a dominant business model. The model does not depend on the industry they are in, nor their age or size. He likens this to our dominant “handed-ness” and as the editor pointed out on the editorial page, “It’s easier to convert a shortstop into an outfielder than it is to change a southpaw into a righty.”

Some firms’ dominant model is “volume operations” and for others it is “complex systems.” The first relies on many customers, brands, advertising, channels and compelling offers. The latter relies on targeted customers and the integration of third party products into total solutions. For each the grass often looks greener in the other model, but almost no business succeeds when they attempt to change models.

The rhythm of most high tech sectors is that the complex sale companies forge new territories and solve unique customer problems. The volume companies come in later and try to commoditize the solution. To survive, the complex sale companies need to do two things simultaneously: defend, for as long as possible, the position they have already won, and move up the solution chain and incorporate the newly commoditized components into an even more interesting solution. The one thing they need to avoid is trying to convert their own early wins into volume opportunities.

What does this have to do with semantics? We are just beginning the commercial roll out of this technology. We are going to have all the fits and starts of any new high tech sector. We have an opportunity to be a bit more self aware. Those of us in the complex sale sector need to be aware that volume operations from adjacent marketplaces will soon enter ours. We need to be continually vigilant about incorporating rather than competing, and moving on up the solution chain. Consumers of this technology have the opposite challenge: how to recognize which aspects of their problems require “complex” solutions and which aspects are ripe to be solved with “volume” solutions.

Dave McComb February 2006

Posted in: Semantics, Strategy by mkumba /
No Comments