User-centered Internet-TV Convergence    
Today and Tomorrow

1/4/12, revised 11/5/13

CoTV (Coactive TV) is a rich architectural framework for a variety of cross-screen services that coordinate Web services with TV viewing.  Such services provide screen-shifting and multi-screen user interfaces, including second-screen "enhancements" to the video, such as related content, social Web functions, ads, etc.  Coactivity of video and Web services can occur on a single screen, but is often far more powerful with multiple screens.

• Many advanced CoTV technologies are described on this Web site, and more fully in Reisman patent filings, including many that are still far beyond current offerings.
• The introduction of the iPad finally brought wide realization of the value of multi-screen CoTV, with the iPad (and/or iPhone) as a TV "companion device." Numerous CoTV "companion" apps emerged beginning in 2009, and are now coming from a full range of companies.  CoTV (and coactive media, more broadly) generalizes this functionality with extensive development of architectures for "Multi-Machine User Interfaces" (MMUIs).
• One key aspect of CoTV that is beginning to be recognized by some providers is the need for ubiquity of the multi-screen UI and content -- CoTV should work for any program on any channel (or Internet source), without need to open a program-specific app (and to first download it).
• Advanced features not yet widely recognized include flexible targeting of enhancement content as services to any screen, alternative enhancement channels from any independent source, link-and-pause features, and media concierge services (as described further below).
• An essential concept of CoTV is that we, as users, want to browse media of any kind, from any source, on any desired device, in any combination, all fully hyperlinked. (We may want to restrict this for convenience and simplicity at certain times, but this full functionality must be there when we want it.)
• CoTV also comes from a user-centric perspective of open services (as seen in the Web), and -- while supportive of the value provided by content providers, curators, and other industry stakeholders -- looks beyond proprietary boundaries and walled gardens to the full range of open hypermedia services to come.
• These services also have application beyond Web and TV, in games, augmented reality, wearable devices, etc.  And of course they apply to apps as well as Web services.

These features are summarized further below, and many are described in this Web site.  Much fuller descriptions are contained in extensive Reisman patent filings dating from 2002.

[Reisman's CoTV patents were licensed to RPX Corporation member companies as of 3/10/15 (over 200 companies). This includes many many major companies in the TV/video and smartphone/ tablet space. Sixteen CoTV patents have been awarded as of 2015.]

CoTV Yesterday

While many rich features of interactive TV date from decades ago, in many respects the vision has been limited by the barriers between the TV and Internet industries, and the blinders of industry thinking about TV.

• 1-Screen: Most concepts of ITV had been oriented to the single TV screen, the inherently limiting experience of interactivity with the "10 foot interface" and the idea that TV viewers always want a "lean-back" experience (not the "lean forward" "2-foot interface" experience of PC, phone, or tablet).
• 2-Screen: Early experiments in 2-screen ITV such as ABCs Enhanced TV (dating from 1999) were conceived as experiments to try ITV out when wide deployment of 1-screen ITV infrastructures were not available, and were expected to fade away when 1-screen became widespread.
• Trivia:  Essentially all early ITV trials provided very simplistic enhancement content, even on 2-screen PC services not constrained by the limitations of the single-screen. It seems TV producers were afraid of losing eyeballs, and saw little value in going beyond brain-dead enhancements like polls and trivia in small snippets. Conversely, some of the most richly produced offerings have been disappointing, partly because of walled-garden thinking.
• Commerce: The driving force for ITV was recognized to be advertising, and "T commerce," but in the narrow form of "buying Jennifer Anniston's sweater" or submitting a "Request For Information," not the full richness of Web marketing and shopping.

CoTV Today -- CoTV 1.0

The increasing prevalence of "media multitasking" (simultaneous use of TV and the Web) on laptops and smartphones began to change perceptions, and 2-screen ITV began to be seen as desirable in itself. Users were creating their own manual ITV experiences by finding relevant Web services on their own.  That set the stage for the emergence of CoTV 1.0, which was then kick-started by the iPad.  One indication of CoTV crossing the chasm into mainstream attention was the survey by Katherine Boehret of the influential Mossberg/Wall Street Journal/All Things D team on 12/20/11.

• Social TV: The emergence of the Social Web added to the growth of media mutitasking.  People were not just looking up actors, sports statistics, and other related content, but tweeting and posting on Facebook. (Spot411 was introduced in 2009 as an iPhone app to automatically synchronize Social Web activity with TV viewing.)
• ACR (Automated Content Recognition): Based on audio or video recognition, ACR was applied as a way to synchronize enhancements to TV or other video, without direct access to the video distribution infrastructure (closed cable or similar systems). ACR gained support from major players like Nielsen and high-profile services like Shazam. It is also gaining a presence in smart TVs, Yahoo Connected TV, and Watchwith (which made a nice demo video).
• iPad: The introduction of the iPad in January 2010 changed the game, as it became apparent that it was a nearly ideal second screen device. The term "companion app" came in vogue, and numerous CoTV companion apps began to appear, both from program providers and independents.  IntoNow offered a universal ACR-based companion app in early 2011, and was soon bought by Yahoo.
• System Operators:  The incumbent distributors also bought into it. Major cable services and other providers offered companion apps in 2010, and began work on 2nd-screen enhancements. They also began work on exposing EBIF triggers to enable content identification and synchronization via the TV distribution infrastructure.
• Basic screen-shifting:  Simple forms of screen shifting emerged with Apple AirPlay, Microsoft SmartGlass, and Google Chromecast. These services are gaining popularity and growing in power and ubiquity.

Many of the basic concepts of CoTV are now gaining wide acceptance, but the full power of this technology is still emerging, and many of the basic principles are still often ignored.

• Ubiquity:  One key limitation is that CoTV services must be ubiquitous and universally applicable to a critical mass of programs to have wide appeal. Most users cannot be expected to download and launch a different app for each program or network. The whole power of coactivity is to be automatic, and to work for any program with little or no setup. Programmers and networks may want special skins and content, but it is not sensible for them to own the basic platform for coactivity. Services like IntoNow and Zeebox point the way, and even program producers are beginning to see the light (such as Fox).
• Always-on: A related limitation is that synchronization should be automatic (unless turned off by the user), and not requite a manual synchronization action every time a user tunes to a new program. CoTV should follow the user as he channel surfs or navigates across DVR, VOD, or Internet TV content. Enhancements should be unobtrusively available, ready to be offered and seen, even when the user has not asked in advance for them. One primitive but broad-reaching step in this direction is Twitter TV Ad Targeting.
• The power of the Web:  A deeper failure of the imagination is that many of the offering continue the mind-set of simplistic enhancements. Rich CoTV services will provide the full power of the Web, in all dimensions of richness, depth, and variety. Web providers learned a decade ago to offer rich experiences, and to make their gardens hold users with their appeal, not with impenetrable walls. CoTV providers need to do the same. Zeebox is one of the few offerings to point to the broader potential. Another hint of this is in Related Content Database (RCDB), rebranded as Watchwith, a backend service for associating diverse content relating to any program. See this crude mock-up of a rich CoTV interface -- one that shows the richness that can be found in conventional Web services. The Game of Thrones SmartGlass offering demonstrates how much money can be spent cultivating a walled garden experience that is ultimately confining.
• Commerce: Commerce has hardly been forgotten, but coactive commerce experiences still remain very limited.
• Ratings: As a by-product of ubiquitous TV context tracking, ratings data on programs will be more complete and accurate than current ratings data (covering most viewers, not just sample panels), and will provide rich detail on engagement.

CoTV Tomorrow -- CoTV 2.0

The following are features described in the Reisman patents that remain advanced, and point to powerful opportunities yet to be realized.

• Screen targeting: Flexible targeting of enhancements to any screen (TV or companion...or wearable) and at varying levels of richness/intrusiveness – such as depending on enhancement type and user mood (lean forward/lean back), controllable by producers and/or users. Basic screen-shifting of video is emerging and gaining popularity, as noted above, but more advanced features would make it easy to shift any portion of the UI from one screen to another, at any time.
  • Emerging screen-shifting tools like AirPlay, SmartGlass, and Chromecast apply when starting a video from a companion device, and allow selection of a TV. A promising integration of this kind of function has been announced by Twitter and Comcast (a "See It" button) as a a potential standard that could apply to any Web service and any TV distributor using any browser or app.
  • Current CoTV enhancement services are generally locked-in to a specific screen, whether TV or companion, but that is a provider and technology-centered design, not compatible with the fact that users seek different modes of interaction at different times, and that there is no longer any real distinction of TV and Internet content.
  • Users should be given the capability to target any content to any screen. "TV Everywhere" is a step in this direction. It is now clear that sometimes we want our video on TV, sometimes on a tablet, sometimes on a phone, and sometimes on a wearable. The same is true for related Internet content, including HTML text, multimedia, Social Web services, transaction services, etc.
  • Targeting choices may depend on whether a user wants to lean back or forward (what level of interaction intensity), share enhancements with others in the room or let them watch without obstruction, etc.
  • Content authors and programmers might also use the same flexibility to author once, display anywhere, and to suggest desired targeting (which users may or may not be able to override).
  • In concept this is similar to the current HTML targeting capabilities that allow HTML pages resulting from a link to reuse a current window, open a new tab, open a new browser window, etc.  Just as a "target=" parameter can be used by an author to control that, or controlled by a user in their browser (such as with a right click or control-click), similar "target=" parameters can be provided for CoTV enhancement content to cause an enhancement to appear on the same screen as the video, or on a second (or nth) screen.
  • Such flexible targeting might also control alternative styles on any selected display, such as frames of text, reduced video images, tickers/zippers, text overlays, etc. on a video screen, or the usual range of sidebars, pop-ups, frames, etc. on a Web screen.
  • (Another important use case for flexible targeting is in wearable devices that are coactive with a phone, such as glasses or watches that present alerts and messages from a phone.)
• Flexible session-shifting: Related to screen targeting is the ability to move all or any portion of a session from one device to another. This may relate to the video, the enhancements or both.
  • The simplest case is video session-shifting. Such capabilities are now being offered by TV distributors, such as to shift a program being viewed on a TV to a mobile device, so the viewer can continue viewing when they must leave home. Desirable features include control of time-shifting, such as for live viewing, pause and resume (with or without a short repeat), etc.
  • Another common case is to shift all enhancements (but not the video). This can be useful to shift to or from lean-back to lean-forward mode, or to make enhancement interactions private on a personal screen or public on a shared TV screen.
  • (Advanced cases are to shift selected classes of enhancements but not others, as described under screen targeting, above.)
• Selectable, Alternative "Enhancement Channels:" Alternative/indie sync content suppliers should be empowered and exploited to offer rich CoTV experiences. 
  • These should be accessible in the form of alternative "enhancement channels"  (ECs) – user selectable channels of enhancements to a given program – such as director commentary, critics, editorial/political spin, humor, etc.  (For example users might select among one or more of liberal, conservative, or humorous spins on news. Film viewers might select film commentary from the director, various critics, friends, or other commentators. Sports viewers might select from commentaries spun neutrally or to either side, or to a fantasy league.)
  • Social TV can be thought of as a case of one or more ECs, each relating to specific social network services and/or subgroups.
  • ECs might be be sourced from (and branded by) the programmer/distributor of a given program and/or ad, or from any number of independent third parties (including user-generated content), and might be segregated on that basis.
  • EC selector/filter settings might be persistent across programs and sessions, or specific to a given viewing situation.
  • A basic sketch of a companion screen showing multiple enhancement channels is given in a figure from the patent disclosure.
  • ECs are related to (and might be supported by) standards now emerging for "open Web annotations."
• Link-and-pause (and sync bookmarks): Users should be able to enable sync to be frozen on either screen -- such as to pause video while linking to an enhancement (link-and-pause), or to hold enhancement links for later review (bookmarks).  This is explained more fully on the page linked above, and a detailed example is also provided.
• Full hypermedia browsing:  The full functionality of CoTV emerges in a context of full hypermedia browsing, where all media types can be fully interlinked, in a manner that is fully consistent. Hotspots can serve as link anchors whether in text, image or video, and targets can be of any media type -- rich combinations of hypermedia browsing and navigation across devices, including "Web as program guide," Web to video, and video to Web. The Twitter-Comcast "See It" button is a promising step in this direction.
• TVC (TV Context): To support the features described here, communication of TV context can be facilitated with a simple context parameter that identifies the program and the time position of the viewer within the program. Web services architectures can be based on this TVC parameter to allow any Web site to serve content that is synced to a program.
  • Any Web site could be programmed to check for a TVC parameter to decide if it wants to serve relevant content, or to ignore it and operate conventionally. Thus CoTV enhancements can be provided by any and all Web sites.  Commerce sites can provide a home page to some users, and a special landing page to viewers with a TVC that indicates they are watching a specific TV ad.
  • Such TVC parameters can be fully resolved formats that specifically identify a program (or ad) by Digital Object Identifiers or any other standard ID code, or unresolved forms that present a channel number, time zone, and service provider identifier, to allow reverse lookups in a program guide (or ad tracking service), or other methods to resolve the actual program identity, such as ACR.
  • TVC can also specify hot spots in video (such as to select Jennifer Anniston's sweater).
• Full Coactive Internet commerce and advertising: This is the elephant in the room. It is secondary in the consumer value proposition, but a major driver of the business case. Having Web services that ubiquitously sync to TV ads is the Holy Grail.
  • Anytime a consumer sees an ad, they can have access to a companion mini-site or other Web service (preferably on a second device) that knows the ad, the offer, and the viewer.
  • Not just personalized advertising (like Visible World), but advertising with a closed loop of interaction. TV ads meet direct marketing.
  • Not just the shotgun of siloed "trans-media," "multi-screen" advertising channels, any one or more of which a viewer might just happen to encounter or not, but full, automatic synchronization for the viewer, driven from the video.
  • And, when ads are well targeted and intelligently interactive, they become not a nuisance to the consumer, but a benefit.
  • Amazon is a natural to exploit this, and is moving in that direction with its expanding X-Ray and second-screen features.
  • Another easy step would be to combine Twitter Ad Targeting with the See It button, using See It to link to telescoping video extensions to a short TV ad spot.
  • For more on this, see an old, but still timely, discussion of CoTV ads.
• Third-party linking rights/fees:  Related to the above features are questions of rights to link from video content by independent parties -- and of fees relating to the value of such links, when provided as a service for use with relevant content/services/commerce. Methods can be applied to monitor such links and to effect charges in any desired direction.
  • The industry has been hung up on questions of who can enhance a program: whether 3rd party enhancement should be permitted (and is legal) and whether producer enhancements should be syndicated. 
  • Fees are one solution, but it seems some producers (such as Fox) are recognizing that it is to their advantage to get their enhancements out though all distributors.

Other opportunities and issues from the perspective of rich CoTV:

• "Over The Top" vs. Incumbents: As consumers mix Internet video sources ("Over The Top," OTT) with use of incumbent, closed TV services, they will demand consistent support for coactivity. Operation of EBIF triggers versus ACR should be transparent and consistent -- technology should support the user, not get in their way. Incumbents should use their power to provide superior services and compete on that basis -- trying to compete by blocking alternatives is likely to be self-defeating. Reasonable interoperability is essential to maximizing revenues in the long run, and to building the critical mass of usage that will give incumbents a healthy slice of a large pie. Recent talk of cable operators supporting Netflix on their set top boxes (or even ceding TV service to focus on broadband carriage) suggest a growing pragmatic flexibility, as the handwriting on the wall gets clearer.
• The dumb question of smart TV: The emergence of connected TVs has added to the opportunities to offer effective CoTV services, but the question of "smart TV" is also a distraction. Connected TVs need not be smart, as long as they are connected to smarts -- whether in a head-end, an Internet cloud, and/or another device. Some smarts in the TV is desirable, but what is smart today will be dumb and very limiting in a few years. Consumers will not want to buy a TV every few years, just to get current intelligence features that can be provided in better ways (and with much nicer UI).  Too smart is dumb.  The $35 Chromecast dongle is making this point very apparent.
• Media Concierge services ("Context is King"): Exploiting the Web as program guide, rich media concierge services can be built to cross media boundaries, and provide discovery services, recommenders, and guides, that apply knowledge of users, their social networks, their media providers, and other sources in an integrated, holistic service. Think of TV Guide + EPGs + Netflix Queue + iTunes + Genius/Pandora/Spotify + Social Web + MetaCritic/RottenTomatoes/IMDB, in an integrated, open, mashup. As others have observed, it may well be that providing such context will increasingly be more profitable than providing the content. This is more fully explained in a 2005 blog post.

Richard Reisman -- Bio
Consulting/About