The computer spreadsheet doesn’t get enough credit among computer programmers. I think that more than any other one concept, VisiCalc, 1-2-3, and Excel were the killer app for the personal computer. As a programmer, I have tended first to think of formulae and calculation mechanisms when I think of spreadsheets, but the UI and development style are perhaps more significant. For each individual cell, you can look at the value, the formula, or the formatting, and change each through a menu. You can incrementally build up quite a complex application all on your own, never leaving the very environment you use to view the results. Why doesn’t all software work this way, only better? That’s what I’m working on.
[For fun off subject, compare Croquet architect Alan Kay’s document-oriented fonts, overlapping windows, laser printer (plus workstation/laptop and Ethernet) versus Croquet architect David Reed’s spreadsheet community (plus TCP/IP).]
Of course, the spreadsheet UI is limited to grids, and the only views of each cell are the three that are built in (value, format, and formula). Even within the spreadsheet paradigm, I could imagine a chart view of an individual cell (not just of a whole table), in which the value is viewed as a color-coded 3D translucent cube with height off (or into) the grid proportional to the magnitude. You look at the chart by flying around to different angles. (OK, it’s not my imagination. We demo this in Croquet.) Or suppose the value for a cell is a reference to some other external data source. Why not have a data source view that lets you look at and even interact with the original source program, right there in the cell?
In Croquet’s immersive 3D environment, you can see and drive around and between objects. There’s no reason not to be able to get a menu of things to do with each object, wherever they are. The grid-ness of spreadsheet cells isn’t really necessary for the user-interface aspects I’m describing here. And since they’re not fixed to a grid, I’d like to be able to drag them around to wherever I want to keep them, much like icons or windows on a desktop, but in 3D. (Hmmm. In Croquet you can make new spaces within each space. Is that 3-1/2D?) And if I can drag these objects around, I’d like to be able to drag them onto each other to form composites that can be interacted with as a group. I call these things components, and the idea is to be able to build up applications by interactively assembling the components you want.
OK, so far that’s just an incremental improvement to the spreadsheet’s setting of different cell behaviors: any object anywhere can be moved around, combined into others, and the different behaviors are reached through menus. Now, suppose the behaviors are components themselves.
!
You could change menus just by moving them around. You could change how an object behaves just by moving behaviors from one object to another.
!
Here’s how it plays out. As you go about your business, you see stuff that you’d like to use. Geometries, textures, buttons that do things, music and video, text, whole arrangements of things. Some will have a behavior that presents a menu when you right-click on the component. You open the menu of things you can do, and you choose “Copy to Pocket” for each. This puts a copy in your pocket, which is really just a space that travels around with you. Don’t worry about collecting too much stuff, because you can always create more spaces within your pocket. There are lots of variations. You could perform various kinds of searches to bring you to places that have the things you want. You could obtain a “Bring me back” button from an object so that you could go back to the space later. (A bookmark.) You could make a kind of copy that will still see changes to the original object. (A link.) But let’s keep it simple and say that you just grab a deep copy of the whole darn thing of interest.
There’s something subtle here: the component you have in your pocket is effectively a complete version of everything you need to use this component. It’s not just the “data” (e.g., an .mp3) for which you need a separate player. All the behavior needed, whether it be static behaviors (e.g., properties, or data), or more active abilities, or buttons & other controls, are right there. A consequence of this is that there are never any issues of “will it work on my computer?” or “oh, you need to upgrade your copy of Word!” or “But I don’t have Photoshop!” As long as you have the ability (e.g., the permission) to see and use and make a copy of a component, then you will, in general, have the ability to use it in your application. It just works. (Of course, someone could deliberately build a component that only works in one space, or which interacts with other components for which you need to be connected (on-line). But the point is that you can think in terms of making whole copies of Word that are always “open” to a specific hardwired document.)
To build your application, you find a nice corner of some place to set up shop. Maybe you create a new space all to yourself, by selecting the “Make New Space” item in the menu of behaviors associated with the space your start in. (Yes, the whole space can be a component.) Now you start pulling things out of your pocket and arranging them where you’d like them to be. Want to change how a component presents itself? Just right-click and change some parameters. Don’t like the arrangement of a menu? Well, the stuff in it are just components, so drag them to where you’d like them to be, or remove the items altogether, or right-click on them to change how they present themselves. Want component A to have the same menu choice you see in component B? Just open up the menu on B and drag the behavior to A’s menu. Want all components (which might be windows) to be dragged around together? Just put them all into one big component. Want to make it so that people can’t copy it? Just remove the Make Copy behavior, and perhaps the Add Behavior behavior. [Want different capabilities available for different users? Frankly, I’m not sure. I haven’t written that part yet. I think it has something to do with making a behavior that presents a specific view of the component, and controlling the behaviors in each view. Then you give the views to your friends, rather than the original component. This multiple view concept has a lot of other uses, including the ‘link’ that I mentioned above. But I’m still working on this part.]
A major inflection point in this is in creating new behaviors from whole cloth. For now, I think of this as being a fairly small task, but one for “real” programmers. They will use Smalltalk in a Squeak window. Because Squeak is late-binding, this can be in a window that appears within Croquet itself, and the results can be used immediately without having to restart. But still, it’s done with textual source code, not by direct manipulation of graphical objects like everything else in the component system. All this means that new Croquet users can have a shallow and continuous learning curve in operating applications and building their own – up to the point where they want to create some new behavior they can’t just steal from someplace else. For now, I’m keeping my sanity (!) by limiting the graphical language of components to just behaviors of no or one argument (equivalent in form to getting and setting properties, or to presenting values in a menu that can be changed). No sequencing. No conditionals. More general self-describing scripting languages are interesting, but I’d like to see how much can be done with (not much more than) what I have described here.
Your proposal reminds me of Morphic in several ways, although it aims above it in terms of manipulating and composing behaviors.
Using direct manipulation, I created a Morphic sticky tab that automatically resizes as you type, and generally behaves nicely. I immediately stash one of these in a flap in any new image that I use. However, I did this using the hard-wired morph menus which don’t allow behaviors to be added/subtracted/modified. It’s cool that you’re aiming for this.
This is a complicated thing to tackle, because there are so many questions to answer, and the answers interact with each other. Some questions:
When should components themselves be added to other components, and when should a copy be added? Adding a cube to a group of geometric components should probably use the cube itself (unless a copy is explicitly requested, perhaps analogously to the green halo handle in Morphic). What about adding a menu item from another menu? It should probably be copied. Should it share any state with the original menu item? How can the framework support these types of policies?
Spreadsheets have a simple conceptual model that allow you to approach a spreadsheet and figure out how it works. Given a complicated nested component, it might be much more difficult for an end-user to develop a mental model of how the whole functions. How can we make this more approachable? David’s filters might help; we could create a filter that would show a "matrix" view of the component that would show the interrelated behaviors (hopefully, our visualization will be better, if not cooler looking, than green symbols cascading from the top of the screen đŸ˜‰ Of course, the design of such a filter is a large task in itself.
Can any behavior be added to any component, or are some behaviors only compatible with certain types of components?
I like your approach of limiting the semantics of components/behaviors to see how far we can get before we get fancy.
Thanks for sharing!
Katamari Damacy
http://www.namco.com/games/…
I think a kind of type system will be required before turning this loose on users, but I’m not working on that yet. (I have prototyped all the stuff described up above, except for copy.) I think the basic idea will be that one of the behaviors of a behavior is a list of all the other behaviors that need to be in place for it to operate. This can be used in testing drop targets. There’s more to it, though. For example, I think its important that behaviors be allowed to be present, but unusable, and should indicate that they are not currently valid. (Imagine temporarilly removing a behavior that someone depends on, but fixing it later. The rest of the behaviors should still be usable, in order, for example, to fix the problem.) Anyway, we can just remove the behavior of a behavior that allows the behavior to be removed rather than copied.
I’m currently working on having multiple views of a component, of which the behaviors is just one view. Dropping one thing on to another will do a little dance between the view that the target wants and the views capable of being provided by the dropee. This will allow us to drop A onto B, even if B expects a particular kind of thing which A isn’t, but which A can produce. It sounds more complicated than it is, but making a copy when appropriate is just one example. The idea is stolen from CLIM presentation translators. I expect that users will only actually see and manipulate views, not the underlying component directly, though most people will not need to know the difference. They’ll think in terms of components. I just wanted to introduce a limited set of ideas, above.
I absolutely expect that different filters of David’s will be able to hold different views of a component.
I have a lot of ideas on UI, and I’m sure that 99.9% of them are stupid. My goal is to make the framework just useable enough so that better HCI people than I can design better UIs.
John, I love the Katamari game description. It reminds me of the comparison of Scheme as a diamond and Common Lisp as a big ball of mud. Being a Lisp guy, I love the game image of this huge ball of stuff rolling over everything its path.
Actually, another game is, I think relevent. The Sims guy is creating a new game (spore?), where the architecture is based on the idea that game content is too expensive to produce. Game users configure their own beasts, which implies certain behaviors, and these creatures are let loose on the other players. The difference is that component configuration includes the ability to directly configure reified behaviors of all kinds, not just geometry and structural stuff like the number of legs or tails.