Haystack design

Now I have a newer, faster, bigger (and less broken, which is why I have it) hard drive, I got to downloading the whole archive (searchable archives, anyone?) of emails to the design mailing list of the Open Source Applications Foundation (which has in the past hosted discussion of the design of Chandler, but lately has become a channel through which barely-relevant press releases are forwarded) and searching it for references to Haystack, which clearly shares some goals and even some underlying "technology" (in particular the use of RDF).

There are a few mentions, starting with Aaron Swartz in October last year, then Dmitry Erkin two months later, and finally Mitch Kapor himself noted that a presentation explaining the project was available.

This final email was the first to get any response at all, strangely. The only comment it drew was one dig at its speed (seriously slow), and one comment that it was impressive, but slow.

Why is it so slow? Haystack appears to be written in a custom language, designed expressly for the purpose. Said language is a hybrid between a language for the declarative specification of RDF triples, and a language in which small scripts can be written. It is implemented in Java, and substantive functionality seems to be written in Java and called out to from the custom language.

It is clear that this approach allows rapid development of powerful software; the demo is exciting, as I have mentioned before. In fact, I suspect that the approach taken will score well on platform surface area. Perhaps the Haystack team will be able to optimise the performance problem away. I hope so. I wonder though whether writing a custom language, clearly designed to be a highly dynamic, configurable layer, on a very static underlay such as Java isn't at the root of the problem. I also wonder whether the mixture of declarative and imperative elements in the custom language, Adenine, doesn't create an increase in the cognitive load which a would be Haystack developer must deal with.

If you want dynamic, why not start dynamic?

If you want specialised, why mix two jobs in one language?

OK, grumbles over. The exciting bit is that Haystack includes innovation in two areas: data management and storage; and user interface design.

In the data management section there is little more or less than what I've been hoping someone would develop for ages. That's not to belittle it -- it looks very well thought out and consistent, and it actually exists, which is more than can be said of Chandler.

In the user interface area however there are some marvellous ideas which I haven't come across anywhere else, and certainly would never have thought of myself.

Chief among the user interface ideas is that of UI continuations. The idea is that when you start a task which requires interaction with the user, instead of opening a dialog box as is normal, a panel is added to a column at the left or right of the window (which I can imagine being my full screen most of the time). You can then provide the needed information through this panel, much as you would with a dialog except that you can move around the rest of your database freely while doing so, use drag and drop to fill fields, and so on. You can also leave the task unfinished and start something else, without losing what work you have done on it.

A simpler, but also quite far reaching innovation is that everywhere a piece of information appears on the screen, a right click will bring up the same actions. Finally we might have some consistency of behaviour in user interfaces, consistency which any number of user interface guidline efforts have failed to impose.

My gut feeling is that this should completely replace the "desktop metaphor". Away with icons, start menus, overlapping windows. I have tasks in progress, agents (eg scheduled alarms), and interconnected information. It would also make a cracking interface on a tablet.

Oh, here's something that's missing: secure, ad hoc, peer to peer collaboration, a la Groove. Groove itself exposes a difficult issue, in that by making shared spaces the primary unit of organisation, it creates disjointedness in an individual's data. This is very similar to the problem with clevercactus' use of "spaces" -- in this case private spaces -- as a primary data organisation structure. Project related spaces are just one way to view my information, and should not take priority over any other. So, could a notion of secure shared spaces be added to the Haystack metamodel, without making them too overarching?