Current efforts at interactive television (ITV) have been crippled by simple misunderstandings and failures of the imagination.

One of the key misunderstandings relates to what interactive TV is. The skeptics are partly right, that TV is largely a passive medium, and interactive TV is something of an oxymoron. But they are wrong in missing the point that many viewers often do want to interact with content that is related to what is on their TV. The term "coactive TV" directs our thinking to this experience of media multitasking, to how we can interact with Web-like hypermedia content in full coordination with what we watch on TV. The technology challenge of interacting with our TV is a red herring--a problem we need not solve. The problem we do need to solve is how we can create this coactive media experience. That is actually less daunting, and largely a matter of software.

Victims of this misunderstanding, TV and PC industry players have struggled to redefine and extend their offerings to add TV-related interactivity. This has failed to capture the imagination of viewers because they have battled over their boxes instead of thinking outside of them. TV and computers are converging into a common core technology, but we still need to live with the fact that they will not converge into a single presentation device, at least not any time soon.

To provide a powerful user-centric interactive TV experience, we must look outside our current boxes to "co-vergence" -- TV and PC device sets must work together in seamless coordination -- to provide a powerful and unified interactive experience (see Figure 1). When we shift to that perspective, it becomes apparent that a rich second-generation ITV service is achievable now, with simple software, using existing TV and PC systems. It will get even better and more flexible with more advanced hardware, but the rich coordination needed to make ITV available and useful can be done with the equipment that tens of millions of people have in their homes right now.

Fig. 1

This paper explores the ideas of coactivity and co-vergence, what they enable now, and how they will evolve. It centers on the perspective of the user, and shows how serving the user supports the profit and service objectives of the key industry players.

The key points are:

• The debate over whether viewers want to interact with their TVs misses the point that they often do want to interact with content that is related to what is on their TV. That is a very different matter. Much of what has been called interactive TV is better understood as coactive TV.
• ITV based on the TV and remote control, and driven by smart set-top boxes (STBs) has been stalled by the inability of cable carriers to justify the expense of deploying more advanced STBs. Even apart from that, it is doubtful that the TV and remote control can effectively support serious interactivity, however smart the STB.
• A variant form of ITV based on synchronizing TV viewing with related Web interactions on a nearby PC, has been hampered by the difficulty of making that coordination easy and effective for viewers. Some 50 million homes already have PCs near a TV, but few TV programs support synchronous Web-enhanced TV, and few users bother finding the Web sites for those programs.
• Human interactive tasks vary dynamically across a spectrum, from brief, simple interactions with small amounts of supplementary data, to extended, complex interactions with extensive data and complex transaction steps. Sometimes you want to lean back, sometimes you don’t. A TV and remote works for simple interactions, but not for serious interactivity—especially when there are multiple viewers.
• An ideal ITV system would let the user interact on a TV screen that is across-the-room when that is simplest, or on a high resolution PC-style device when that richer capability is desired. Richly interactive TV viewing should be a multi-modal experience. The user should be able to use any device he wants, whenever he wants.
• As wireless laptops and new form factor tablet PCs become widespread, an ideal device for rich ITV interaction will be at hand in many homes at no added cost. Wireless PDAs can also be used as an effective complement to the TV and remote.
• The CoTV™ technology described here can provide the coordination needed for this co-vergence™ . In the near term, seamless coordination of TV and PC interactivity can be achieved with novel ITV portals that link the user’s current STB and PC through a cable headend. Simple software extensions to current Internet browsers and STB software can be downloaded to current home devices to complete this linkage at low cost. Later, this can be done with wireless home neworks. The core of CoTV™ technology is the export and import of the current state of a user’s interactive session between the TV and PC.
• From the user’s viewpoint, this lets him decide which box he wants to interact through, and to change that when his needs change. Advanced versions can support an "inter-box™ " session in which interactive sessions may start on the TV and then be transferred mid-stream, on demand, to a PC (or PDA), while maintaining the context of the session. The user will see the corresponding screen, with no loss of position, history, or data entered, and can maintain full continuity in any remote server session (such as during shopping or gaming).
• For programmers and advertisers, this provides a flexible platform for compelling enhanced content and direct response marketing using TV and PC devices in a fully integrated manner. Markup coding can be used to selectively target content to either the TV or PC. Personalized, addressable ads, VOD programming, and other streaming can be fully supported.
• For cable providers, CoTV™ portals can provide a new service that generates fees from users and/or advertisers. For satellite TV providers, this capability can motivate linkage of the STB and PC in the home to enable not only inter-box™ ITV, but other interactive TV services that draw on PC relay to a broadband back-channel.

The first generation of ITV has been hobbled by misunderstanding, resulting in the inability of either TV or PC industry players to find a value proposition that serves both themselves and the consumer. CoTV™ provides a simple new context, for a win-win collaboration that allows all the players to cooperate in bringing value to the user, and to share in profiting from the powerful new services and related commerce opportunities that will create. It is a collaboration that can bear fruit now, and grow rapidly to embrace a larger market that will demand increasingly advanced products and services.

Ask three media people what interactive TV is and you will get at least four answers. The term has been a catch-all. Few have really been clear about the fact that there are three very different kinds of interactivity.

• Interactivity with a TV set is the one that is most successful. This got its first big jump with the use of the remote control to enable new channel surfing behaviors, and is evolving to include video-on-demand, VCR-like pause, rewind, and fast forward, commercial skipping and the like. It does not change any content, only how we view that content.
• Interactivity with TV program content is the one that is "interactive TV" in its deepest sense, but it is also the most challenging to produce. This is the idea that the program, itself, might change based on viewer input. Advanced forms, which still seem pretty far off, include dramas where viewers get to choose plot details and endings. Simpler forms, which are enjoying some success, include programs that directly incorporate polls, questions, comments, and other forms of (virtual) audience response back into the show. There is much debate as to how effective and popular this kind of truly interactive TV can be. It seems likely that some forms of it will become popular, but that viewing of pre-defined content will remain a major part of the TV experience indefinitely.
• Interactivity with content that is related to what is on TV ("coactivity") is the one that has been least understood, yet is most likely to radically alter TV viewing over the next decade. Examples include getting more information about what is on the TV, whether sports, movies, news, or the like – or, most likely to pay the bills, more information about what is being advertised, and the ability to buy it. Partial steps in this direction have been taken and shown valuable, but confusion with the first two has impeded recognition of just how valuable they could easily become.

The proponents of ITV and the nay-sayers are both partially correct, but both seem to miss the opportunity that coactivity offers. The rest of this paper is oriented to re-visioning the part of this misunderstood beast - coactive TV – what it can be, and how we can enable it.

Chickens, eggs, and set-top boxes

At the height of the digital media boom of the late ‘90s, it appeared that the long-heralded convergence of TV and computer-based media was within reach. Interactivity and "t-commerce" were seen as gold mines, and the TV industry was preparing to deploy a new generation of advanced digital STBs that were to enable this new age of interactive TV. Much of the cable plant had been upgraded for two-way digital services, and first generation digital STBs were deployed to some 15 million homes, but those boxes lacked the computer power for more than the most basic interactivity. Another round of investment was needed.

As the cold light of skepticism and tight capital replaced these rosy predictions, the cable industry pulled back on plans to deploy advanced STBs. We now face what is often called "the chicken and egg problem" -- which comes first?: how can we justify advanced STBs without demonstrable demand for advanced services, and how can there be demonstrable demand if there are no advanced STBs to enable consumers to try and buy services they are do not yet understand.

Given this current impasse, the industry is trying to find other services that can generate revenues with current digital STBs now, and begin to offer a taste of interactivity to prepare the market for more. These include interactive services that are structured to draw on intelligence at remote head-end servers to support simple t-commerce, games, and such, and other limited forms of interactivity such as video on demand (VOD). Without thinking in these terms, the industry came to understand that interactivity with the TV set was valuable and readily enabled, and that has become the primary focus for now.

One-box, Two-box, Inter-box, Any-box

But the call of interactivity with content still beckons. Some programmers have sought to avoid this problem of the STB by moving these interactions to the Web. This potentially serves the roughly 50 million households that have a PC in the same room as the TV. For selected shows such as Jeopardy or Monday Night Football, a viewer can go to a Web URL for the program. Upon logging in and indicating the time zone, Web pages can then be provided in synchrony with the TV program. This "two-box," "two-screen" ITV scheme, often called Enhanced or Extended TV (ETV), and also known as "synchronized TV" or "telewebbing" offers considerable flexibility by being able to provide any kind of Web content with the full interactive user interface of a PC browser, but the coordination with the TV is very limited and must be established by the viewer every time a program is tuned in. The Web enhancements are synchronized globally (by time zone) and thus not applicable to VOD or time-shifted programming, or other individual variations such as addressable ads.

The "one-box," "one-screen" solution of interactivity on the TV screen and driven by the STB is far more seamlessly coordinated, since the interactive elements are presented by a browser within the STB that is fully aware of the program content and any interactive elements are tightly associated with that. There is no URL to enter. If a program has interactive enhancements, they are at hand and ready to interact with. If the viewer changes channels, the STB knows it, sees the corresponding interactive elements, and interaction is directly coordinated. If a Digital Video Recorder is used to pause, rewind, or defer viewing, the interactive elements can retain their synchronization. Current two-box technology cannot accommodate any such variations. For reasons such as these, much of the industry regards "two-box" solutions as just a stopgap, one that may be more or less useful to get limited interactivity while we wait for an effective "one-box" solution. The industry seems to have settled into the camps of those cautiously exploiting the two-box stopgap and those who ignore it as a temporary distraction.

This is where the industry imagination has fallen short, and the user perspective has been missed. In a simplistic view, one-box ITV is convenient to the user, and two-box reliance on a separate PC is awkward. What this simplistic view forgets is that

• One-box ITV is not so convenient when one wants to interact with more than a small amount of information in simple ways, especially when other viewers want to watch the program in its full-screen glory without interfering interactive overlay menus and text.
• Two-box ITV can be fully coordinated by adding some simple software to both the STB and the PC that links the two systems for coordinated use.
• As wireless laptops and tablet devices become more common, whether full function PCs or simple "Webpads," or even wireless PDAs, the second box can be physically quite convenient, and the user’s desire to use it will grow strong.

The user does not care what comes from the cable or satellite or broadcast TV source or what comes from the Internet. The user does not care how the boxes might be connected and coordinated. All he cares about is the functionality of his viewing experience. Once the user becomes familiar with ITV and the various kinds of tasks he can do, he will want to be able to decide which box to interact with and when, and will want to be able to use any intelligent box he has and deems useful ("any-box"). He will want his boxes to be coordinated so that what he does at one box can be reflected at the other box. He will want to decide when to work on one box, such as for a few quick interactions, and when to shift the activity to another box, such as for more intensive interaction ("inter-box"). By failing to recognize this as coactive TV, the nature of this multi-tasking coordination has been missed.

This any-box/inter-box coordination of ITV, this coactive TV, may sound like idealistic futurism, but it is not difficult to do. It can be made available not in years, but in months. Much of this capability can be achieved with no new hardware in the home, just by downloading some software and providing some simple server support, at a relatively modest investment. More advanced coordination will come as home networks make the linkage even easier.

Before outlining the technical aspects of just how this coordination can be achieved, it may be helpful to look more deeply why this capability is so important to enabling really powerful and user-friendly ITV and other similarly advanced interactive hypermedia. The short-term chicken and egg problem with one-box ITV set-tops is well recognized in the industry, but the deeper limitations of one-box TV interactivity are not.

Most people became familiar with interactive media when the Web became popular in the mid- to late-‘90s. We learned the wide range of tasks, services, and communications that can be facilitated using a fairly simple and powerful Web browser user interface, combined with the open linking of content and services on the Internet. We became familiar with high-resolution computer monitors, keyboards, and mice or similar pointing devices. Few of us remember the early days of interactive services, starting decades ago with much cruder screens and pointing devices like cursors and arrow keys. Services like videotex, teletex, and the early Prodigy service. Clunky text, displayed fuzzily on TV screens using Commodore 64s and Apple II’s. Tabbing line-by-line through menus. The reason few remember that is that few were willing to use such awkward and restricted systems. Now we are being invited back to those glorious days of yesteryear. It is déjà vu on the interactive TV screen.

Forgetting the constraints technology may or may not put on how we do it, consider the essential human experience of ITV and related media browsing activities. We work with various input/output devices, having various form factors, that let us see and hear video, audio, text, and other kinds of data as outputs, and to point, select, command, and enter text or other data as inputs. Let’s call the group of input/output (I/O) devices we work with on any given task a device set (see Figure 1). What is important from this perspective is not the STB or the PC or any other controller system, but the fact that these device sets, however they are driven, serve as the user’s window into a virtual world of media content, wherever it may be.

• The typical device set for watching TV is a TV monitor, usually across a room, and a remote control keypad that can be held in the hand (with links to the TV or STB via wireless infrared transmissions). This configuration of I/O devices has a form factor that is very effective for relaxed viewing of video images and simple device control interactions such as channel changes and volume adjustments. It is also workable for somewhat more complex interactions such as using an electronic program guide (EPG) to review a TV listing grid on-screen, scroll through it, and select shows to watch or view as video on demand. This is well suited to the lean-back mode of activity of the "couch potato."
• The typical device set for browsing the Web or doing other information-intensive tasks on a PC is a high-resolution screen, used up-close, and a keyboard and mouse or touchpad or joystick. This form factor is very effective for reading, entering, and interacting with text, controlling complex activities using multiple windows and rich graphical controls, and controlling highly interactive multimedia experiences such as DVD and Internet video. It is widely recognized that the graphical user interface (GUI) first popularized on the MacIntosh is what changed the PC from a tool for analysts to an information tool for people of all kinds, and that the GUI of the Mosaic browser (commercialized as Netscape) is what changed the Web from a tool for physicists to a medium for everyone. This configuration works very well for the lean-forward mode of activity typical of most Web browsing and related e-commerce.

In thinking about user interfaces and the device sets we need to embody them, it becomes apparent that these differences in lean-back and lean-forward device sets are fundamental to the kind of user interface device technologies can expect to rely on for many years to come. Someday we may have heads-up displays, head mounted goggles, and data gloves or even bionic interfaces that can present images of any format and sense rich gestures from any posture, but that day is probably many years off, at least for most purposes. High definition (HDTV), big-screen TV monitors and wireless keyboards could make working with text across a room a bit more workable, but that will never beat a lean-forward screen for extended intensive use. Similarly, we can get HDTV on an up-close PC monitor (or a second such monitor), but that will rarely be the ideal way to kick back and relax with a movie.

ITV and the spectrum of interactivity

But wait, some would say – lean-back interactivity may be limiting, but isn’t ITV a matter of simple interactions that do work on the TV-type device set? Aren’t current ITV services carefully designed to use few short menus, minimize text entry, and allow useful tasks to be completed with a few clicks. Doesn’t that mean the one-box solution will be just fine?

Only in part. Current ITV services seem to be repeating the early evolution of computer interactive services. Simple, limited UIs favor simple, limited services. More complex services can be attempted, but do not work well. Some limited market develops for the simple services, however slowly. But as soon as a better UI is available, the floodgates open, and the latent demand for richer services causes a quantum leap in the richness and variety of services offered. (AOL and CompuServe WinCIM made one such step, and Netscape made another.) Simple services become more polished and complex services are no longer hobbled. Online environments that could not keep pace dropped away, as those that did flourished.

Current one-screen ITV functions much like the early ‘90s pre-Web version of Prodigy, with simple walled garden services not much richer than the simple videotex systems of the ‘70s (Prodigy used the same videotex presentation technology, called NAPLPS). We are so inured to the current limitations that few look ahead to what ITV could become with an effective suite of UI devices. Just as the Web quickly took on a depth and richness and sophistication far beyond Prodigy, ITV can evolve to have a similar richness and power.

WebTV unwittingly demonstrated the power of a good interactive user interface by doing the reverse adaptation, degrading the Web to work on a TV screen. The inherent limitations of that adaptation are recognized as a major cause of WebTV’s failure. But cable players still talk of similar Web and PC services driven by their STBs. When all you have is a hammer, you can use it to drive a screw, but that will not be very satisfying if you have ever used a screwdriver.

Similarly, this simplistic form of TV-related interaction limits the vast potential of rich and open enhancement content. The Web moved online interactivity beyond simple and limited sets of screens with a narrowly programmed content theme, to deliver flexible, user-controlled linkage to a vast variety of content that could follow tangent upon tangent, giving us the concept of Web surfing (including both its casual and serious forms). ITV can extend that evolution into coactivity with a similarly open (and far more rich) hypermedia space of content related to an evolving, personally shaped video experience.

As an aside on industry dynamics, TV distributors, programmers, and advertisers may be concerned that greater richness will cause a leakage of their viewers away from the content they control and derive revenue from. However, the upside of this can bring even greater potential to attract, keep, and profit from viewers who find the base TV program to form the center of a truly compelling multimedia experience. ITV will have limited appeal as long as it is confined to the trivial: simple trivia contests, play-alongs, and spoon-fed content. The TV experience becomes far richer and more valuable, and can build enthusiastic community bonds, when it integrates with deep and rich content and services. Similarly, advertising can go beyond simplistic responses to offer a fully coordinated suite of rich interactive transactions, demos, tools, configurators, and infomercials (VOD or streamed). Rich opportunities for cross promotion, product placement, and other forms of integrated marketing will emerge as such media develop. In fully open form, such co-vergent services could offer a richness of content greater than any programmer can provide alone. (In deference to the continuity of the viewing experience, such coactive, multi-tasking tangents could also be deferred and held, much like bookmarks, to be pursued after the main program completes.) Nevertheless, the expanded capabilities of the CoTV™ approach proposed here have compelling value even if restricted to a closed, walled-garden service (much as some current Web enhanced TV services such as Disney’s ETV are closed).

The right tool for the job, as the job changes

The right tool for the job depends on what the job is, at any given time. ITV and Web interactive experiences are not inherently different, but lean toward different ends of a single spectrum of interactivity. ITV (and other advanced hypermedia) marries video to the Web (or more limited Web-like walled gardens) to range over the full spectrum of activities from pure lean-back, passive viewing to pure lean-forward, intensive interaction. That interaction may include video (or text) across the room, or in a window on a PC screen.

The job, and thus the right tool, varies over time. TV viewing, browsing, and the combined ITV experience is best understood as an interactive "session" having a sequence of steps. The nature of a session changes over time. Tangential sub-sessions can fork off from an initial session. Complex patterns of multi-tasking can occur, especially as a user becomes more skilled and empowered by rich content presented with powerful viewing tools. Some viewers will remain content with simple sessions, but an increasingly large population of "heavy media users" can be expected to exploit whatever power tools are provided, if they are flexible and well designed.

• At one time I may be watching an absorbing movie and want no interaction except maybe to pause for a moment or replay a segment. I may recognize an actor and want to know who he is and what else he was in. While watching a game with friends I may want to check some statistics or see other current game scores. It may be easiest to do that all on the TV, and not bother reaching for a PC or PDA-type device set.
• At another time I may see an ad for Disneyworld that mentions a new attraction and special rates on hotel rooms. I could just hit a button to request that they email me some information (or send me a videotape), but maybe I want to do a bit more right now. Wouldn’t Disney prefer that I could do that? I might do that on the TV, but if I got to picking hotels and accommodation details and coordinating that with flight prices and schedules, doing that on a TV is not the kind of feat to try at home. For that I would want to be using a PC-style lean forward device set. It would be nice if I could go part way down such an interactive path on my TV, then transition to the PC after I decided this was something I wanted to pursue right now, and be able to do that without retracing my steps from the home page at Disney.com. It would be even nicer to have the full power of a Web browser. Better yet, it would be nice to have access to my Outlook calendar, my hotel and air travel site cookies, and my Intuit budget and payment system. Such capabilities would be valuable for any considered purchase item, such as cars, financial services, appliances, and travel.
• Similarly, if I was watching a game with friends, I might want to go deeper into stats for other games and other teams, or to update a fantasy league selection. Doing that on the TV might be difficult and annoying to my friends. Wouldn’t it be nice to just push some buttons on the remote to take the interactive screens I am viewing on the TV and transfer to the equivalent Web version on my PC tablet? And wouldn’t it be nicer still if after finding some interesting statistics on the PC that I wanted to share, I could transfer that page back to the TV?
• Back to watching the movie, what if the movie was not captivating, or was one that I had seen before, and I wanted to keep it playing, but dig deeper into the credits, cast, director, related films, and do some serious surfing. It would be nice to do that on a PC in my lap, without having to figure out what URL to go to. At some point I might get re-involved in the movie. Alternatively, I might decide to what another film with two of the same stars and want to view it as a VOD with just a click from my PC. Before I did that, I might want to view a trailer or other segments of it in a window on the PC screen—especially if others in the room were watching the first movie and not yet convinced that it should be changed.
• Layering on channel surfing, I may be flipping between two or three sports events, and viewing statistics and replays relating to each. It might be nice to have the relevant windows stay open, and have those relating to the current channel pop to the top every time I flip the TV channel (but at other times I might want to decouple that). This experience fully combines the dynamics of both channel surfing and Web surfing.
• Depending on the relative dynamics and urgency of the program and of the coactive sub-tasks, I might want to defer a link to enhancement content that can be held like a bookmark for activation later, perhaps after the program ends. This might be desirable for incidental details about "the actor in this scene" or "the player who just made that play" or "this news story," or for responding to commercials. Such deferral could be set as the standard default, if programmers insisted.
• Similar capabilities can be extended to multiple personal devices sets, each used by individual viewers of a single TV screen. The ability to work privately as desired, and to transfer activities back and forth between the shared TV screen and the private PC/PDA screens, could be enable a whole new dimension of group ITV.

What is important from these examples is that the determining factor is not whether the activity is "TV" or "Web" or "computer" activity, but where on the spectrum of interactivity it lies (and how tightly it is coupled to the primary program). ITV can be lightly interactive or intensively so. The same applies to Web-based activities and other computer-driven media activities.

Some limited recognition of the need for complementary device sets has been creeping into the visions of the TV and PC industries. Evolution of the home media gateway brings convergence a step closer to the user, moving toward a technology base of common networks, common controls, and shared content access for an entire suite of home media systems, including TV, home theater, music, digital photos, PCs, the Web. Microsoft has recognized that a lean-back user interface for a computer acting as a media system controller is desirable, and is adding a new lean-back "Freestyle" UI to WindowsXP. Motorola has announced a lean-forward Webpad-style device for use with TV, called the EVr Enhanced TV Viewer, and Philips, Universal Electronics, eRemote, and others have proposed screen-equipped remote controls, including versions based on a standard PDA. These devices do not include support for real coordination as described below. These dedicated devices present another chicken and egg problem: we can’t see the need for buying a dedicated device if we can’t really see what it could do for us.

User-centric Co-vergence and Any-box flexibility

The emergence of "two-box" ITV using a PC, and of specialized Webpad crossover devices reflects some basic recognition of the limitations of the "one-box" TV form factor, but none of these efforts has addressed the high level of flexibility required for a truly effective user-centric viewing experience.

• The fundamental issue is usage dynamics. A given interactive session might be desirably shifted mid-stream from one device set to another as the dynamics of the task change. From a user-centric perspective, a user should be able to move any session to any device set, at any time. Only the user knows what he wants to do now, what he expects to do next, what tools he has at hand, and which of them he wants to use at any given moment. (While the user will generally want to be the ultimate arbiter of what appears on what device set, content providers may also exert creative control by recommending default behaviors. This authoring control can be enabled by such methods as coding markup attributes, as discussed below.)

• A second issue that is nearly as critical is that of provisioning. This is the key business economics question of where these boxes come from and who pays for them. From a user-centric perspective, a user would like to have the option to use any UI device set he has available, and not need to buy a dedicated device for use with his TV. Some 50 million households have PCs in the same room as a TV, and an increasing number of these are wireless laptops. A new generation of tablet devices are to hit the market in late 2002, both full function tablet PCs, and cheaper slave tablets based on Microsoft "Mira" technology (with versions for both business and consumer use). A growing number of wireless PDAs are also in homes. Consumers will want to use these device sets for ITV services, if they can. They will not be eager to buy (or pay subscription fees) for dedicated ITV Webpad devices that do not also serve as general purpose PCs or PDAs.

Clearly one tool is not right for all ITV tasks. Multi-function devices are well known to have the common problem of doing none of their functions well. But even a set of specialized tools will be awkward to use if it is not easy to shift from one tool to another as the nature of the task changes during the course of a session. Adding more expensive devices to the cost of an ITV viewing system would just bring us back to the chicken and egg problem. What is needed to break the provisioning impasse is a way to use available devices that are at hand, and need not be obtained specifically for use with ITV. (Dedicated ITV devices might find a market later, when users know exactly what they want in such a device and how much they will use it, and can justify paying for a particularly well-suited model to keep by the couch.)

We can view this in terms of a concept that might be called a Multi-Machine User Interface (MMUI). The broad requirement is to be able to use multiple co-vergent device sets in concert to view a single integrated set of content resources. That capability is provided by a MMUI. Such an MMUI could be controlled by a single integrated box that drives a varied suite of lean-forward and lean-back device sets -- that could be a future convergent STB or computer. The problem is that such a convergent box is a long time off, and such a multi-function system may never be economically attractive. So a key technology question is how can simpler, less costly, independent controller boxes (STBs and PCs/PDAs) be made to work in cooperation to provide such an MMUI?

Before addressing the question of how this co-vergent solution can be provided in the near term, let us first look at the general requirements of a co-vergent solution. The basic idea is to transfer a session out of one box and into another. As shown in Figure 1, TV viewing or ITV browsing starts on one device set, such as on the TV. That device set obtains and presents content resources through a user interface that works more or less like the Web -- with enhancement content that can be navigated by hyperlinks or some similar menu-based control process. Whether at the first step of interaction with enhancement content, or after some initial portion of a session, it is determined that the session is to be transferred to a different device set, such as the PC. For this to happen, the PC must have basic information about the current state of the session (a "state record") so that it can pick up where the TV left off. This process of getting that information from the TV to the PC and activating the newly migrated session could be triggered from either device set -- as a push initiated from the TV or as a pull initiated from the PC.

Fig. 2

The key steps to accomplish this and the software components involved are shown in Figure 2. In the upper part is System 1, the TV. The user interacts through some UI controls to conduct a session 1a. This session may be as simple as selecting a channel and letting it play, or may involve browser-like interactions to begin ITV enhancement viewing or other advanced functions at the TV. At some point a session transfer request is made, either a push or a pull, to the other System 2, the PC, below.

• For a basic CoTV™ portal, it may be sufficient to simply transfer the identity of the channel being viewed, so that an appropriate Web page can be presented, thus achieving a simple but significant improvement over current synchronized "two-box" ITV systems.
• More advanced support of pure TV functions can address richer state, such as whether a VOD program was being viewed, and whether the program was paused or time-shifted (using a DVR or head-end services), or which of many addressable TV ads was appearing on this particular TV. In such cases state would include information on program ID and time-position.
• For more advanced support of full "inter-box" ITV browsing ( in which ITV enhancements could be viewed on either the TV or PC), the state would add information on the current enhancement resource (equivalent to a Web page), the history of prior resources (pages) viewed, and other richer details of state such as data entered in forms.

The exporter at the starting system creates this state record in cooperation with the TV player and/or ITV browser software in the STB, and the importer at the destination system uses it, in cooperation with the Web browser to establish a corresponding session (a "co-session") in the PC. Depending on the options chosen, this transferred session might then present a new page or resource, just as if a hyperlink had been actuated on the TV and traversed on the PC. Alternatively, it might present a cloned version of the same page or resource viewed at the TV. This simple transfer is shown by the set of arrows marked #1.

Advanced implementations can provide for transfers in either direction, so that after conducting the continued session on the PC, the user might transfer it back to the TV, using a similar state transfer in the reverse direction, as shown by the arrows marked #2. It may ordinarily be desirable that when a session is transferred, it forks so that activity continues on the two systems in a relatively independent fashion, such as watching the primary program on the TV and browsing enhancement content on the PC in forked, loosely-coupled co-sessions. However it may also be useful at times to maintain full synchrony, so that interactions at one system are tracked and mirrored at the other ("synchronized co-sessions"). This can be done by transferring state after every interaction, as shown in the further arrows marked #3.

The details of these importer/exporter components will depend on the internal details of the player or browser systems they work with.  For all but the most advanced implementations, these components can be quite small, and can be built as downloadable software extensions for current STBs and current PC browsers (like Internet Explorer or Netscape), or PDA-based browsers.

A challenge to achieving this coactive coordination with currently deployed systems is how to link current STBs to current PCs, given that home network connections are still uncommon, especially for STBs. Here again, a solution can be found outside the box--this time far outside the box. The STB and the PC are not linked within most homes, but if we look outside that box (the home) we can make the connection. Like the old song, the STB is connected to the head-end…The head-end can be connected to an Internet portal…The Internet portal is connected to the Internet…The Internet is connected to the PC. A bit roundabout, perhaps, but that is the beauty of the Internet.

This means that cable companies can offer a simple ITV portal service that links a consumer’s TV to their PC using their current basic digital STBs. Some 10 million homes already have such STBs as well as always-on cable modem service. And that number is growing rapidly. A cable company can create a "CoTV™ portal" that supports all of the channels that it offers (including VOD), giving viewers a single URL to access for all ITV viewing. Better access can be provided with a simple browser accessory that can be downloaded to provide an always-present toolbar that links to that base URL to obtain specific links for the currently viewed program. That can be presented as a browser page, or within the browser accessory toolbar as a basic menu of features related to the current program. Whenever the computer is on and connected, it can be synchronized to whatever show is on (see Figure 3).

Fig. 3

The CoTV™ portal can be a walled garden or a more open portal, and can have content generic to a channel or to a series or to a particular program episode. The portal can also provide content enhancements for TV advertisements that add rich supplementary content and direct response capability. Unlike current two-box Web ETV, a CoTV™ portal provides not just crude synchronization with a generic broadcast, but real coordination with what is actually happening on a particular user’s TV. Instead of dumb synchronization that requires frequent manual activation when it works at all, it provides smart synchronization that automatically tracks individual viewing.

• If the user changes channels, surfs, or flips between channels, the CoTV™ portal changes the Web enhancements to correspond. This could be done with multiple windows, so a viewer could have enhancements for several channels active concurrently.
• If the user does customized time-shift viewing with a DVR or VOD, the CoTV™ portal can coordinate and synchronize with that as well.
• If the cable system supports targeted ads to addressable STBs, so that one user sees one ad and his neighbors see another, the CoTV™ portal will know that, and can show the corresponding enhancements. Again, added windows could allow ad enhancements to pop on top while primary program enhancements remain open in another window.

This CoTV™ portal can also support other related applications that relate to TV viewing, such as an electronic program guide and specialized information tools relating to genres such as sports or films or news, etc. Such applications could be entirely server-based, such as typical Web portals, but local "thick client" applications might offer even better user interfaces. For example, a program guide could use spreadsheet-like controls to combine rapid scrolling with rich viewing of program details.

Some may be concerned that this transfer of current channel and other state information through the head-end and portal creates a privacy concern. Some potential for exposure does exist, but remember that this data is only for use by the consumer it belongs to. The portal can act as the user’s agent to prevent any other use of that data, and can require a sign-on with password and certification. Data monitoring can be enabled only for consumers who subscribe to the portal service, and can be limited to times when the user is actively signed in. Furthermore, lock-out controls could be provided at the STB to prevent any export of this state data, except when the viewer unlocks it. Thus a user could disable export at any time they wanted to avoid any risk of observation.  Still more complete assurance of privacy could be achieved by using encryption between the STB and the PC, so that all coactivity is controlled within the home, and the head-end and portal operator cannot read the state data at all.

The chicks are here; let them grow

Again, all of this inter-box coordination can be done with current digital STBs. The software additions are not overly complex or resource-intensive, and the head-end support is also not demanding. Current technologies for STB and PC-based ITV can be used. Selected content can be created at the channel, show, or episode level, as well as for selected ads, and it can be expanded incrementally as demand grows. The technology can also evolve gradually, as demand warrants.

In the simple case of basic digital STBs, all enhancement content must be viewed on the PC. That is because the STB is not capable of presenting such content (except for special content such as program guides). But if the user does have an advanced STB, then the more powerful inter-box function can be provided to support transfer on demand, again through the CoTV™ portal. The user can begin viewing enhancements on the TV, without any need to use the PC, but can transfer the session to the PC at any time that may be desired. Limited implementations might limit the granularity of session context that is preserved, but full continuity of session context can be maintained with a moderate level of added software support. All of this additional function can be done using currently available (but not widely deployed) advanced STBs.

Again, any PC can be used, so consumers need not get new TV Webpad devices, and cable operators need not fund such added hardware. As improved wireless laptop devices become more common, the potential CoTV™ user base will expand without any direct investment.

This means that a basic, but highly effective and appealing, ITV service can be put in place now, and begin to attract tens of millions of enabled users. As they and their friends see what coactivity can be and what this powerful system can do, they will form a base for rapid demand growth. Programmers and advertisers will see this large and growing target market, and develop increasingly extensive, compelling, and profitable enhancement content. As interest grows, early adopters and heavy media users, such as sports fans, movie fans, or news junkies, will become interested in the added flexibility of advanced STBs. These can be deployed selectively to customers who want them and want to pay for them. This can be done within the context of the larger base of an established two-box market, with established content that can be adapted to the TV format with only moderate incremental cost, and extended as demand warrants. The chicken-egg impasse can be broken, and a healthy cycle of demand and supply growth can gain momentum. As coactive TV becomes widely understood and used, truly interactive TV programming might also develop in the slipstream.

One of the issues relating to inter-box coactive viewing is the adaptation of content resources for viewing on either the TV or the PC. Since the presentation capabilities of the TV and PC device set form factors are so different, enhancements suited for one form factor will not be well suited for the other. That is the heart of flexible coactivity, and thus central to a full inter-box solution that supports a choice of enhancement viewing on either the TV or the PC (or on PDAs). Initial steps toward inter-box co-vergence can used fixed affinities of content resources to device form factors. This can include the basic CoTV™ portal service described above, in which all enhancements are viewed on the PC screen, or can apply some simple scheme of fixed allocations of selected resources to one device or the other. More advanced implementations will let the user decide whether a given enhancement screen is seen on TV, on PC, or on a PDA. This requires the ability to adapt resource presentation to suit these very different form factors.

This adaptation problem is one that has already drawn much attention in other contexts, notably for cell phone and PDA access to the Web, and a range of solutions have emerged.

• The simplest method of adaptation is to provide multiple parallel versions of a resource, one for each form factor to be supported. This is technically simple, but gets administratively complex as the extent of the content grows, and eventually becomes prohibitively costly and labor intensive. (The use of WAP Wireless Markup Language is an example of cell phone versions of Web content.)
• An alternative is to do adaptation automatically based on rules for converting one format to another. This is commonly done for PDAs that access the Web through clipping services such as supported by Palm or AvantGo, and very similar (possibly even the same) clipping could be used to adapt to presentation on TV. This is most often done by special gateway servers, but could be done at a client system as well. This provides a "write once, play anywhere" capability, but the limitations of rule-based adaptation may not give a desirable level of control over content style.
• The alternative that is emerging as preferred for adaptation of Web content to different form factors is to use style sheet codings within the Web page markup. This allows a single page to be authored with instructions on how it should be adapted to variant form factors. A number of current and emerging Web standards provide this capability, such as CSS, XHTML, and other ongoing work in the W3C and the MPEG-21 Digital Item Adaptation effort. This gives the content author full control of how the resource is presented on each class of device. With appropriate authoring tools, standard style sheets can be used with only minor tuning for specialized or demanding content needs.

Thus in the near-term, the simple method of multiple alternative formats may be effective for limited use. Increasingly, however, as flexible inter-box usage grows, the best solution is likely to be that of style sheet coding.

Programmer recommendations for creative control

Another key issue in making this flexibility work to its full potential is providing a balance of user control and content programmer or author control of what screens appear on what device set. A core focus of the co-vergent approach is the primacy of the user, and the need to give the user significant control over how his display set tools are used. However, too much freedom can create problems.

• Users may not want to have the full burden of thinking about when a transfer to another device set might be desirable. Sometimes they may not know where they are going, and sometimes they will want to leave some of that driving to an author.
• Authors/producers will want to be able to control, or at least suggest what class of device set best serves the presentation they envision, and the content that they are leading to.

The solution to this balance is much like that for the Web. The targeting of multiple device sets proposed here extends a similar concept used on the Web for targeting multiple windows in a browser. The default behavior of the Web is for link traversals to target the same window that they originate from, so that the page clicked to replaces the page from which a link was clicked. But author-coded Web markup can specify HTML link attributes that tell the browser to target another window, either new or existing. This gives the author a level of control over the use of multiple windows. CoTV™ provides for a similar kind of coding for targeting of multiple device sets.

A simpler, but less flexible method is to group enhancement content resource screens into categories by type of screen, and to define different types as having affinities to different device set. Thus certain kinds of screens would have affinity to the TV (if the STB was suitable), and certain kinds of screens would have affinity to the PC (if coordination were active and enabled). Such screen types could relate to simple structures like the "play, more, explore" structure used with the RealOne streaming video player, or with the content categories defined by the proposed ITV Production Standards.

Simple user controls and setup preferences

Also much like the Web, user control of device set targeting must be very simple. As noted, control techniques can be much like those used for window control on the Web. Experienced Web surfers know that they can cause a link to target a new window instead of replacing the contents of the current window by actuating the link with a shift-click instead of a simple click. (Another way is to right-click, and then select "open in new window" from the special drop-down menu control.)

• Working from the TV, an analogous control method might be as follows: instead of simply pushing the select button to actuate a link to an enhancement screen, an alternate key might be used, or a combination of some other key pressed with the select key.
• Working from the PC, the control might be to use a new combination, such as control-click to signal a transfer to the TV (or an added option on the right-click drop-down menu). Different commands could be provided to clone the current resource to the alternate device set, just as Web browser controls now provide for an "open new window" command that clones a new instance of a current window.

Another way to ease control is to let the user set preference behaviors, such as to define whether author coded targeting affinities should be followed or not. User preferences could also define logical device set groups, to be activated at different times, so that at some times the TV might be used alone, as one group, with the PC as another group, and with a PDA as a third group. Naturally the introduction of new "boxes" should be made easy, with plug-and-play discovery of new devices. Different profiles could be set up for different users, and for device sets in different rooms.

Lashing together a national ITV service based on current standards

A further aspect of the current chicken-egg problem is that current ITV services are very parochial, often restricted to specific cable operators or specific shows or networks. This limits the critical mass of viewers and thus weakens the motivation for programmers and advertisers to produce ITV content and ads. Web access to content with a PC (or PDA) device set offers a global content standard, but as noted, Web-based ITV has been very limited.

CoTV™ provides a new way to leverage Web standards in a way that co-exists with other ITV formats and standards, and provides a framework for harmonization of formats, based on the adaptation methods described above. Instead of competing with any deployed STB-based ITV content, CoTV™ enables both formats to co-exist and complement one another.

Programmers and advertisers can create content for Web access as the common denominator across all cable, satellite, and broadcast systems, and can supplement that with parallel content for STB-based use in whatever markets and to whatever extent demand warrants. Existing formats can be used, but style-sheet and XML-based Web-compatible formats will be increasingly desirable. The Web-based PC (and PDA) formats can have nationwide reach almost immediately.

Satellite TV, home networks, and the evolving variety of configurations

The cable TV-based CoTV™ portal described above is a configuration that has considerable and immediate appeal, but many other variations can be attractive as well. These include satellite TV, also in the near term, as well as various evolving forms of home networking.

Satellite TV systems are generally limited in interactivity because they lack an effective two way capability. Current satellite systems commonly have only a dial-up modem connection to use as a back-channel. Broadband Internet connections such as DSL (or cable) could provide such a back-channel, if the STB and PC were connected in the home. Satellite TV operators can provide CoTV™ portal services that work much like the service described for cable, but in this case the linkage would be added within the home. The ability to offer this superior ITV service to users could help drive adoption of such connections of satellite boxes to broadband modems, whether by direct cabling or wireless home networking. DirectTV’s DirectDSL service would seem to be a natural for this. (Some satellite systems do provide their own always-on back channel -- in that case, the state could be relayed through it to the head-end, just as described for the cable-style portal.)

In the longer term, wireless home networking will enable additional variations on the CoTV™ concept. Future STBs can be expected to add local WLAN network connections, so that state transfers can be done within the home, without need for relay through the head end.

Another variation is to use a wireless-equipped PDA to serve as a TV remote control, as some already do, and add software to enable the PDA to serve as the state exporter/importer for the TV. The PDA can monitor the state of the TV it controls, and relay that to a PC. Similarly, it can also import state changes (such as channel changes) from the PC back to the TV.

Conclusion: Technology should empower us, not confine us

Making coactive TV and other similar advanced hypermedia happen is a major step in the evolution of media, but in terms of technology, it is an incremental one.

• It does involve changes in user awareness and behavior, new ways of combining and relating content of diverse kinds, and new combinations of distribution, hardware, and software. But it is largely a matter of extension and convergence of current media and systems, not addition of unfamiliar new ones. This becomes clear when we distinguish the key variants that have been confused within the catch-all term "interactive TV."
• All of the pieces we need to make it happen are at hand, or readily provided, and most of the infrastructure is well known. The costs to begin the introduction of next generation coactive TV services are modest, and can expand incrementally with gradual build-out of infrastructure and content production as demand grows.

Technology convergence is necessary, but not sufficient. ITV is not a matter of boxes, whether TV industry boxes or computer industry boxes. ITV is based on interaction by users, and must deliver what the user wants from a hypermedia experience. Users, and content producers have not been fully aware of what they want, because they have seen so little of a real hypermedia experience, and have failed to see the importance of coactivity. But by stepping back and imagining the ideal hypermedia experience, it becomes clear that it mixes lean-back and lean-forward sub-tasks -- often with distinct, but related content elements -- that can be seamlessly interwoven to work with a rich blend of video, text, and other content. Supporting those sub-tasks is best done with different kinds of device sets working in coordination to present a unified media experience.

This coordination of lean-back and lean-forward device sets stops short of complete convergence, and takes a form we are calling co-vergence. Television and the Web are currently at different ends of a media spectrum. The bands of the spectrum should not be expected to converge into one thing, but to play out as a palette of colors, with different combinations at different times. We need to recognize the nature of multi-tasked coactivity and coactive content. We should not be confined within standalone boxes that can only deal with one band of the spectrum. Sometimes you feel like leaning back, sometimes you don’t.

Set-top boxes and computers are not in a winner-take-all battle of the boxes -- neither can be all things to all people. The sooner we start pulling in the same direction, the sooner a new age of interactivity can be realized and profit all of us.

When Ted Nelson coined the terms hypertext and hypermedia many years ago, he observed that, "everything is deeply intertwingled." He saw that the core task for information and media technology was to facilitate deep interconnection. It is time for us to take another step in that direction, and do what it takes to deal with coactive media.