Today and Tomorrow
1/4/12, revised 11/5/13
CoTV (Coactive TV) is a rich architectural
framework for a variety of cross-screen services that coordinate Web
services with TV viewing. Such
services provide screen-shifting and multi-screen user interfaces, including
second-screen "enhancements" to the video,
such as related content,
social Web functions, ads, etc. Coactivity of video and Web services can
occur on a single screen, but is often far more powerful with
- Many advanced CoTV technologies are described on
this Web site, and more fully in Reisman patent filings, including many that are still far beyond current offerings.
- The introduction of the iPad finally brought wide
realization of the value of multi-screen CoTV, with the iPad (and/or
iPhone) as a TV
"companion device." Numerous CoTV "companion" apps emerged beginning in
2009, and are now coming from a full range of companies. CoTV (and
coactive media, more broadly) generalizes this functionality with
extensive development of architectures for "Multi-Machine User
- One key aspect of CoTV that is beginning to be
recognized by some providers is the need for ubiquity of the
multi-screen UI and content -- CoTV should
work for any program on any channel (or Internet source), without need
to open a program-specific app (and to first download it).
- Advanced features not yet widely recognized
include flexible targeting of enhancement content as services to any
screen, alternative enhancement channels from any independent source,
link-and-pause features, and media concierge services (as described
- An essential concept of CoTV is that
we, as users, want to browse media of any kind, from any source, on any
desired device, in any combination, all fully hyperlinked. (We may want
to restrict this for convenience and simplicity at certain times, but
this full functionality must be there when we want it.)
- CoTV also comes from a user-centric
perspective of open services (as seen in the Web), and -- while supportive
of the value provided by content providers, curators, and other industry
stakeholders -- looks beyond proprietary boundaries and walled gardens to
the full range of open hypermedia services to come.
- These services also have application
beyond Web and TV, in games, augmented reality, wearable devices, etc.
And of course they apply to apps as well as Web services.
These features are summarized further below, and many
are described in this Web site. Much fuller descriptions are contained
Reisman patent filings dating from 2002.
licensed to RPX Corporation member
companies as of 3/10/15 (over 200 companies). This includes many many major
companies in the TV/video and smartphone/ tablet space.
Sixteen CoTV patents have been awarded as of 2015.]
While many rich features of interactive TV
date from decades ago, in many respects the vision has been limited by
the barriers between the TV and Internet industries, and the blinders of
industry thinking about TV.
- 1-Screen: Most concepts of ITV
had been oriented to the single TV screen, the inherently limiting
experience of interactivity with the "10 foot interface" and the idea
that TV viewers always want a "lean-back" experience (not the
"lean forward" "2-foot interface" experience of PC, phone, or tablet).
- 2-Screen: Early experiments in
2-screen ITV such as ABCs Enhanced TV (dating from 1999) were conceived
as experiments to try ITV out when wide deployment of 1-screen ITV
infrastructures were not available, and were expected to fade away when
1-screen became widespread.
- Trivia: Essentially all
early ITV trials provided very simplistic enhancement content, even on
2-screen PC services not constrained by the limitations of the
single-screen. It seems TV producers were afraid of losing eyeballs, and
saw little value in going beyond brain-dead enhancements like polls and
trivia in small snippets. Conversely, some of the most richly produced
offerings have been disappointing, partly because of walled-garden
- Commerce: The driving force for
ITV was recognized to be advertising, and "T commerce," but in the
narrow form of "buying Jennifer Anniston's sweater" or submitting a
"Request For Information," not the full richness of Web
marketing and shopping.
CoTV Today -- CoTV 1.0
The increasing prevalence of "media
multitasking" (simultaneous use of TV and the Web) on laptops and
smartphones began to change perceptions, and 2-screen ITV began to be seen
as desirable in itself. Users were creating their own manual ITV experiences
by finding relevant Web services on their own. That set the stage for
the emergence of CoTV 1.0, which was then kick-started by the iPad.
One indication of CoTV crossing the chasm into mainstream attention was the
survey by Katherine Boehret of the influential Mossberg/Wall Street
Journal/All Things D team on 12/20/11.
- Social TV: The emergence of the
Social Web added to the growth of media mutitasking. People were
not just looking up actors, sports statistics, and other related
content, but tweeting and posting on Facebook. (Spot411 was introduced in
2009 as an iPhone app to automatically synchronize Social Web activity
with TV viewing.)
- ACR (Automated Content
Recognition): Based on audio or video recognition, ACR was applied as a
way to synchronize enhancements to TV or other video, without direct
access to the video distribution infrastructure (closed cable or
similar systems). ACR gained support from major players like Nielsen and
high-profile services like Shazam. It is also gaining a presence in
smart TVs, Yahoo Connected TV, and Watchwith (which made a nice
- iPad: The introduction of the
iPad in January 2010 changed the game, as it became apparent that it was
a nearly ideal second screen device. The term "companion app" came in
vogue, and numerous CoTV companion apps began to appear, both from
program providers and independents. IntoNow offered a universal
ACR-based companion app in early 2011, and was soon bought by Yahoo.
- System Operators: The
incumbent distributors also bought into it. Major cable services and
other providers offered companion apps in 2010, and began work on
2nd-screen enhancements. They also began work on exposing EBIF triggers
to enable content identification and synchronization via the TV
- Basic screen-shifting:
Simple forms of screen shifting emerged with Apple AirPlay, Microsoft
SmartGlass, and Google Chromecast. These services are gaining popularity
and growing in power and ubiquity.
Many of the basic concepts of CoTV are now
gaining wide acceptance, but the full power of this technology is still
emerging, and many of the basic principles are still often ignored.
- Ubiquity: One key
limitation is that CoTV services must be ubiquitous and universally
applicable to a critical mass of programs to have wide appeal. Most
users cannot be expected to download and launch a different app for each
program or network. The whole power of coactivity is to be automatic,
and to work for any program with little or no setup. Programmers and
networks may want special skins and content, but it is not sensible for
them to own the basic platform for coactivity. Services like IntoNow and
point the way, and even program producers are beginning to see the light
- Always-on: A related limitation
is that synchronization should be automatic (unless turned off by the
user), and not
requite a manual synchronization action every time a user tunes to a new
program. CoTV should follow the user as he channel surfs or navigates
across DVR, VOD, or Internet TV content. Enhancements should be
unobtrusively available, ready to be offered and seen, even when the
user has not asked in advance for them. One primitive but broad-reaching
step in this direction is Twitter
TV Ad Targeting.
- The power of the Web: A
deeper failure of the imagination is that many of the offering continue
the mind-set of simplistic enhancements. Rich CoTV services will provide
the full power of the Web, in all dimensions of richness, depth, and
variety. Web providers learned a decade ago to offer rich experiences,
and to make their gardens hold users with their appeal, not with
impenetrable walls. CoTV providers need to do the same. Zeebox is one of
the few offerings to point to the broader potential. Another
hint of this is in Related Content Database (RCDB), rebranded as
Watchwith, a backend service
for associating diverse content relating to any program. See this crude
mock-up of a rich CoTV
interface -- one that shows the richness that can be found in
conventional Web services. The Game of Thrones SmartGlass offering
demonstrates how much money can be spent cultivating a walled garden
experience that is ultimately confining.
- Commerce: Commerce has hardly been
forgotten, but coactive commerce experiences still remain very
- Ratings: As a by-product of ubiquitous TV context
tracking, ratings data on programs will be more
complete and accurate than current ratings data (covering most viewers,
not just sample panels), and will provide rich detail on engagement.
CoTV Tomorrow -- CoTV 2.0
The following are features described in the
Reisman patents that remain advanced, and point to powerful opportunities
yet to be realized.
- Screen targeting: Flexible
targeting of enhancements to any screen (TV or companion...or wearable) and at varying
levels of richness/intrusiveness such as depending on enhancement type
and user mood (lean forward/lean back), controllable by producers and/or
users. Basic screen-shifting of video is emerging and gaining
popularity, as noted above, but more advanced features would make it
easy to shift any portion of the UI from one screen to another, at any
- Emerging screen-shifting tools like
AirPlay, SmartGlass, and Chromecast apply when starting a video from
a companion device, and allow selection of a TV. A promising
integration of this kind of function has been announced by Twitter
and Comcast (a "See
It" button) as a a potential standard that could apply to any
Web service and any TV distributor using any browser or app.
- Current CoTV enhancement services are
to a specific screen, whether TV or companion, but that is a
provider and technology-centered design, not compatible with the
fact that users seek different modes of interaction at different
times, and that there is no longer any real distinction of TV and
- Users should be given the capability
to target any content to any screen. "TV Everywhere" is a step in
this direction. It is now clear that sometimes
we want our video on TV, sometimes on a tablet, sometimes on a
phone, and sometimes on a wearable. The same is true for related Internet content, including HTML
text, multimedia, Social Web services, transaction services, etc.
- Targeting choices may depend on
whether a user wants to lean back or forward (what level of
interaction intensity), share enhancements with others in the room
or let them watch without obstruction, etc.
- Content authors and programmers
might also use the same flexibility to author once, display
anywhere, and to suggest desired targeting (which users may or may
not be able to override).
- In concept this is similar to the
current HTML targeting capabilities that allow HTML pages resulting
from a link to reuse a current window, open a new tab, open a new
browser window, etc. Just as a "target=" parameter can be used
by an author to control that, or controlled by a user in their
browser (such as with a right click or control-click), similar "target=" parameters
can be provided for CoTV enhancement content to cause an enhancement
to appear on the same screen as the video, or on a second (or nth)
- Such flexible targeting might also
control alternative styles on any selected display, such as frames
of text, reduced video images, tickers/zippers, text overlays, etc.
on a video screen, or the usual range of sidebars, pop-ups, frames,
etc. on a Web screen.
- (Another important use case for
flexible targeting is in wearable devices that are coactive with a
phone, such as glasses or watches that present alerts and messages
from a phone.)
- Flexible session-shifting:
Related to screen targeting is the ability to move all or any portion of
a session from one device to another. This may relate to the video, the
enhancements or both.
- The simplest case is video
session-shifting. Such capabilities are now being offered by TV distributors,
such as to shift a program being viewed on a TV to a mobile device, so
the viewer can continue viewing when they must leave home. Desirable
features include control of time-shifting, such as for live viewing,
pause and resume (with or without a short repeat), etc.
- Another common case is to shift all
enhancements (but not the video). This can be useful to shift to or from
lean-back to lean-forward mode, or to make enhancement interactions
private on a personal screen or public on a shared TV screen.
- (Advanced cases are to shift selected
classes of enhancements but not others, as described under screen
- Selectable, Alternative
"Enhancement Channels:" Alternative/indie sync content suppliers
should be empowered and exploited to offer rich CoTV experiences.
- These should be accessible in the
form of alternative "enhancement channels" (ECs) user
selectable channels of enhancements to a given program such as
director commentary, critics, editorial/political spin, humor, etc.
(For example users might select among one or more of liberal,
conservative, or humorous spins on news. Film viewers might select
film commentary from the director, various critics, friends, or other
commentators. Sports viewers might select from commentaries spun
neutrally or to either side, or to a fantasy league.)
- Social TV can be thought of as a
case of one or more ECs, each relating to specific social network
services and/or subgroups.
- ECs might be be sourced from (and
branded by) the programmer/distributor of a given program and/or ad,
or from any number of independent third parties (including
user-generated content), and might be
segregated on that basis.
- EC selector/filter settings might be persistent
across programs and sessions, or specific to a given viewing
- A basic sketch of a companion screen
showing multiple enhancement channels is given in a
from the patent disclosure.
- ECs are related to (and might be
supported by) standards now emerging for "open
Link-and-pause (and sync bookmarks): Users should be able to
enable sync to be frozen on either screen -- such as to pause video
while linking to an enhancement (link-and-pause), or to hold enhancement
for later review (bookmarks). This is explained more fully on the
page linked above, and a
example is also provided.
- Full hypermedia browsing:
The full functionality of CoTV emerges in a context of full hypermedia
browsing, where all media types can be fully interlinked, in a manner
that is fully consistent. Hotspots can serve as link anchors whether in
text, image or video, and targets can be of any media type -- rich
combinations of hypermedia browsing and navigation across devices,
including "Web as program guide," Web to video, and video to Web.
The Twitter-Comcast "See It" button is a promising step in this
- TVC (TV Context): To support the
features described here, communication of TV context can be facilitated
with a simple context parameter that identifies the program and the time
position of the viewer within the program. Web services architectures
can be based on this TVC parameter to allow any Web site to serve content
that is synced to a program.
- Any Web site could be programmed to
check for a TVC parameter to decide if it wants to serve relevant
content, or to ignore it and operate conventionally. Thus CoTV
enhancements can be provided by any and all Web sites. Commerce
sites can provide a home page to some users, and a special landing page
to viewers with a TVC that indicates they are watching a specific TV ad.
- Such TVC parameters can be fully
resolved formats that specifically identify a program (or ad) by Digital
Object Identifiers or any other standard ID code, or unresolved forms
that present a channel number, time zone, and service provider
identifier, to allow reverse lookups in a program guide (or ad tracking
service), or other methods to resolve the actual program identity, such
- TVC can also specify hot spots in video
(such as to select Jennifer Anniston's sweater).
- Full Coactive Internet commerce and
advertising: This is the elephant in the room. It is secondary in
the consumer value proposition, but a major driver of the business case. Having
Web services that ubiquitously sync to TV ads is the Holy Grail.
- Anytime a consumer sees an ad, they can
have access to a companion mini-site or other Web service (preferably on a
second device) that knows the ad, the offer, and the viewer.
- Not just personalized advertising (like
Visible World), but advertising with a closed loop of interaction. TV
ads meet direct marketing.
- Not just the shotgun of siloed "trans-media,"
"multi-screen" advertising channels, any one or more of which a viewer
might just happen to encounter or not, but full, automatic synchronization
for the viewer, driven from the video.
- And, when ads are well targeted and
intelligently interactive, they become not a nuisance to the consumer,
but a benefit.
- Amazon is a natural to exploit this, and
is moving in that direction with its
expanding X-Ray and second-screen features.
- Another easy step would be to combine
Twitter Ad Targeting with the See It button, using See It to link to
telescoping video extensions to a short TV ad spot.
- For more on this, see an old, but
still timely, discussion of CoTV
- Third-party linking rights/fees:
Related to the above features are questions of rights to link from
video content by independent parties -- and of fees relating to the value
of such links, when provided as a service for use with relevant
content/services/commerce. Methods can be applied to monitor such links
and to effect charges in any desired direction.
- The industry has been hung up on
questions of who can enhance a program: whether 3rd party enhancement
should be permitted (and is legal) and whether producer enhancements
should be syndicated.
- Fees are one solution, but it seems some
producers (such as
Fox) are recognizing that it is to their advantage to get their
enhancements out though all distributors.
Other opportunities and issues from the perspective of
- "Over The Top" vs. Incumbents: As consumers
mix Internet video sources ("Over The Top," OTT) with use of incumbent, closed TV services, they will
demand consistent support for coactivity. Operation of EBIF triggers
versus ACR should be transparent and consistent -- technology should support the user, not
get in their way. Incumbents should use their power to provide superior
services and compete on that basis -- trying to compete by blocking
alternatives is likely to be self-defeating. Reasonable interoperability
is essential to maximizing revenues in the long run, and to building the
critical mass of usage that will give
incumbents a healthy slice of a large pie. Recent talk of cable
operators supporting Netflix on their set top boxes (or even ceding TV
service to focus on broadband carriage) suggest a growing pragmatic
flexibility, as the handwriting on the wall gets clearer.
- The dumb question of smart TV:
The emergence of connected TVs has added to the opportunities to offer
effective CoTV services, but the question of "smart TV" is
distraction. Connected TVs need not be smart, as long as they are connected
to smarts -- whether in a head-end, an Internet cloud, and/or another device.
Some smarts in the TV is desirable, but what is smart today will be
dumb and very limiting in a few years. Consumers will not want to buy a
TV every few years, just to get current intelligence features that can
be provided in better ways (and with much nicer UI). Too smart is dumb.
The $35 Chromecast dongle is making this point very apparent.
Media Concierge services ("Context is King"): Exploiting the Web
as program guide, rich media concierge services can be built to cross
media boundaries, and provide discovery services, recommenders, and
guides, that apply knowledge of users, their social networks, their
media providers, and other sources in an integrated, holistic service.
Think of TV Guide + EPGs + Netflix Queue + iTunes + Genius/Pandora/Spotify +
Social Web + MetaCritic/RottenTomatoes/IMDB, in an integrated, open,
mashup. As others have observed, it may well be that providing
such context will increasingly be more profitable than providing the content. This is more fully
explained in a
2005 blog post.