Video First

Search is fragmenting into verticals.     In the past year two meaningful verticals have emerged — one is video — the other is real time search.   Let me play out what happened in video since its indicative of what is happening in the now web.     YouTube.com is now the second largest search site online — YouTube generates domestically close to 3BN searches per month — it’s a bigger search destination than Yahoo.     The Google team nailed this one.    Lucky or smart — they got it dead right.    When they bought YouTube the conventional thinking was they are moving into media –  in hindsight — its media but more importantly to Google — YouTube is search.     They figured out that video search was both hard and different and that owning the asset would give them both a media destination (browse, watch, share) and a search destination (find, watch, share).  Video search is different because it alters the line or distinction between search, browse and navigation.       I remember when Jon Miller and I were in the meetings with Brin and Page back in November of 2006 — I tried to convince them that video was primarily a browse experience and that a partnership with AOL should include a video JV around YouTube.     Today this blurring of the line between searching, browsing and navigation is becoming more complex as distribution and access of YouTube grows outside of YouTube.com.    44% of YouTube views happen in the embedded YouTube player (ie off YouTube.com) and late last year they added search into the embedded experience.    YouTube is clearly a very different search experience to Google.com.       A last point here before I move to real time search.    Look at the speed at which YouTube picked up market share.  YouTube searches grew 114% year over year from Nov 2007 to Nov 2008!?!     This is amazing — for years the web search shares numbers have inched up in Google favor — as AOL, Yahoo and others inch down, one percentage point here or there.    But this YouTube share shift blows away the more gradual shifts taking place in the established search market.     Video search now represents 26% of Google’s total search volume.

The rise of the Notificator

I started thinking about search on the now web in earnest last spring.    betaworks had invested in Summize and the first version of the product (a blog sentiment engine) was not taking off with users.   The team had created a tool to mine sentiments in real-time from the Twitter stream of data.    It was very interesting — a little grid that populated real time sentiments.   We worked with Jay, Abdur, Greg and Gerry Campbell to make the decision to shift the product focus to Twitter search.   The Summize Twitter search product was launched in mid April.   I remember the evening of the launch — the trending topic was IMAP — I thought “that cant be right, why would IMAP be trending”, I dug into the Tweets and saw that Gmail IMAP was having issues.    I sat there looking at the screen — thinking here was an issue (Gmail IMAP is broken) that had emerged out of the collective Twitter stream — Something that an algorithmically based search engine, based on the relationships between links, where the provider is applying math to context less pages could never identify in real time.

A few weeks later I was on a call with Dave Winer and the Switchabit team — one member of the team (Jay) all of a sudden said there was an explosion outside.   He jumped off the conference call to figure out what had happened.    Dave asked the rest of us where Jay lived — within seconds he had Tweeted out “Explosion in Falls Church, VA?”  Over the nxt hour and a half the Tweets flowed in and around the issue (for details see & click on the picture above).    What emerged was a minor earthquake had taken place in Falls Church, Virginia.    All of this came out of a blend of Dave’s tweet and a real time search platform.  The conversations took a while to zero in on the facts — it was messy and rough on the edges but it all happened hours before main stream news, the USGS or any “official” body picked it up the story.  Something new was emerging — was it search, news — or a blend of the two.   By the time Twitter acquired Summize in July of ’08 it was clear that Now Web Search was an important new development.

Fast forward to today and take a simple example of how Twitter Search changes everything.    Imagine you are in line waiting for coffee and you hear people chattering about a plane landing on the Hudson.   You go back to your desk and search Google for plane on the Hudson — today — weeks after the event, Google is replete with results — but the DAY of the incident there was nothing on the topic to be found on Google.  Yet at http://search.twitter.com the conversations are right there in front of you.    The same holds for any topical issues — lipstick on pig? — for real time questions, real time branding analysis, tracking a new product launch — on pretty much any subject if you want to know whats happening now, search.twitter.com will come up with a superior result set.

How is real time search different?     History isnt that relevant — relevancy is driven mostly by time.    One of the Twitter search engineers said to me a few months ago that his CS professor wouldn’t technically regard Twitter Search as search.   The primary axis for relevancy is time — this is very different to traditional search.   Next, similar to video search — real time search melds search, navigation and browsing.       Way back in early Twitter land there was a feature called Track.  It let you monitor or track — the use of a word on Twitter.    As Twitter scaled up Track didn’t and the feature was shut off.   Then came Summize with the capability to refresh results — to essentially watch the evolution of a search query.      Today I use a product called Tweetdeck (note disclosure below) — it offers a simple UX where you can monitor multiple searches — real time — in unison.    This reformulation of search as navigation is, I think, a step into a very new and different future.   Google.com has suddenly become the source for pages — not conversations, not the real time web.   What comes next?   I think context is the next hurdle.    Social context and page based context.    Gerry Campbell talks about the importance of what happens before the query in a far more articulate way than I can and in general Abdur, Greg, EJ, Gerry, Jeff Jonas and others have thought a lot more about this than I have.    But the question of how much you can squeeze out of a context less pixel and how context can to be wrapped around data seems to be the beginning of the next chapter.    People have been talking about this for years– its not that this is new — its just that the implementation of Twitter and the timing seems to be right — context in Twitter search is social.   74 years later the Notificator is finally reaching scale.

A side bar thought: I do wonder whether Twitter’s success is partially base on Google teaching us how to compose search strings?    Google has trained us how to search against its index by composing  concise, intent driven statements.   Twitter with its 140 character limit picked right up from the Google search string.    The question is different (what are you doing? vs. what are you looking for?)  but  the compression of meaning required by Twitter is I think a behavior that Google helped engender.     Maybe, Google taught us how to Twitter.

On the subject of inheritance.  I also believe Facebook had to come before Twitter.    Facebook is the first US based social network — to achieve scale, that is based on real identity.  Geocities, Tripod, Myspace — you have to dig back into history to bbs’s to find social platforms where people used their real names, but none of these got to scale.    The Twitter experience is grounded in identity – you knowing who it was who posted what.    Facebook laid the ground work for that.

What would Google do?

I love the fact that Twitter is letting its business plan emerge in a crowd sourced manner.   Search is clearly a very big piece of the puzzle — but what about the incumbents?   What would Google do, to quote Jarvis?   Let me play out some possible moves on the chess board.   As I see it Google faces a handful of challenges to launching a now web search offering.    First up — where do they launch it,  Google.com or now.Google.com?    Given that now web navigational experience is different to Google.com the answer would seem to be now.google.com.   Ok — so move number one — they need to launch a new search offering lets call it now.google.com.    Where does the data come from for now.google.com?    The majority of the public real time data stream exists within Twitter so any http://now.google.com/ like product will affirm Twitter’s dominance in this category and the importance of the Twitter data stream.    Back when this started Summize was branded “Conversational Search” not Twitter Search.     Yet we did some analysis early on and concluded that the key stream of real time data was within Twitter.    Ten months later Twitter is still the dominant, open, now web data stream.   See the Google trend data below – Twitter is lapping its competition, even the sub category “Twitter Search” is trending way beyond the other services.   (Note: I am using Google trends here because I think they provide the best proxy for inbound attention to the real time microbloggging networks.   Its a measure of who is looking for these services.    It would be preferable to measure actual traffic measured but Comscore, Hitwise, Compete, Alexa etc. all fail to account for API traffic — let alone the cross posting of data (a significant portion of traffic to one service is actually cross postings from Twitter).   The data is messy here, and prone to misinterpretation, so much so that the images may seem blurry).   Also note the caveat re; open.   Since most of the other scaled now web streams of data are closed / and or not searchable (Facebook, email etc.).

Google is left with a set of conflicting choices.     And there is a huge business model question.     Does Ad Sense work well in the conversational sphere?   My experience turning Fotolog into a business suggests that it would work but not as well as it does on Google.com.    The intent is different when someone posts on Twitter vs. searching on Google.   Yet, Twitter as a venture backed company has the resources to figure out exactly how to tune AdSense or any other advertising or payments platform to its stream of data.    Lastly, I would say that there is a human obstacle here.     As always the creative destruction is coming from the bottom up — its scrappy and and prone to been written off as NIH.     Twitter search today is crude — but so was Google.com once upon a not so long time ago.     Its hard to keep this perspective, especially given the pace that these platforms reach scale.     It would be fun to play out the chess moves in detail but I will leave that to another post.   I’m running out of steam here.

AOL has taken a long time to die.    I thought the membership (paid subscribers) and audience would fall off faster than it has.    These shifts happen really fast but business models and organizations are slow to adapt.  Maybe its time for the Notificator to go public and let people vote with their dollars.   Google has built an incredible franchise — and a business model with phenomenal scale and operating leverage.   Yet once again the internet is proving that cycles turn — the platform is ripe for innovation and just when you think you know what is going on you get blindsided by the Notificator.

Note:    Gerry Campbell wrote a piece yesterday about the evolution of search and ways to thread social inference into  search.    Very much worth a read — the chart below, from Gerry’s piece, is useful as a construct to outline the opportunity.

Disclosure.   I am CEO of betaworks.    betaworks is a Twitter shareholder.  We are also a Tweetdeck shareholder.  betaworks companies are listed on our web site.

As announced this am, Twitter is acquiring 100% of Summize. Deals between two private companies are easy to consider and hard to close. In this case we had both companies on a tear and the teams on both sides who were interested in a partnership — the hope here is that what makes sense today only makes more sense down the road. Search on twitter will evolve into more than search — this is starting to happen today (more below), but bringing these teams together will only accelerate the pace of that evolution. The deal started with a conversation with Fred Wilson about how conversational search can evolve into navigation, about how important navigation becomes for UGC as you go mainstream — it concluded with the deal that was announced this morning. Betaworks is now a twitter shareholder, and excited to be one.

Finding a pain point
The history of most startup’s is made up of iterations, learning and restarts — Summize was no exception. The Summize team worked hard for a little over a year developing sentiment based algorithms aimed at crawling the review and blogosphere. Late last year they formally launched a web product that let you search reviews for books, movies and music. It worked well — offering summaries of all the reviews for a particular book, structured programmatically so they could be organized and swiftly digested by users or publishers. Yet it was complicated — not in theory or in its presentation — but in practice it was a complicated problem that most end users didnt know they needed. As an old friend would put it Summize v1. didn’t address a discernible need or pain point.

I remember early this year we took the Summize team over to meet with an executive at News Corp. After the WSJ/Dow Jones acquisition, News Corp. was thinking about data centric media and how conversational media — the blogosphere — can be mapped and structured in a scalable manner. Jeremy was fascinated by the technology but pushed us hard as to whether we knew whether people were really looking for programmatic structured access to sentiment. By March it was clear we couldn’t get the sentiment focussed company funded by VC’s — many people were interested but no one was ready to take the risk. I think this is part of the chasm between east and west coast companies — out west, interesting technology can and is often funded purely on the merits of the technology — out east, not so. At betaworks we decided to work with the Summize team repoint the technology — and launch twitter search. Why Twitter? Three reasons: there was a gap in the market for a scaled search / navigation experience of twitter, summize technology was very capable of providing and scaling a great search experience across the twitter’s live river of conversations and finally Twitter, the base data set, was growing like a weed.

Growth
It’s astounding how fast the Summize service took off. The growth is charted in this post. The premise was that there is a real time data distributed across services online that is hard to digest and that search is a well know metaphor to aggregate up these conversations into something meaningful for people. Twitter was the logical starting point – traffic was exploding and Twitter was quickly becoming a real time, one to many communications platform. Search is so often viewed as a destination experience — get this result and move on. Summize search is different — because its conversational and real time you keep searches running and open in tabs, you repeat them time and time again, to watch the conversation evolve and change — watch that refresh bar on any of the topics linked to above. The approach worked. Traffic exploded, not only on the UI but also on the API. Distributed, live search — very, very different to how search has been done to date on the web.

Now web
There is something new going on here. Somewhere in the past few months the way that I experience the Internet and specifically live information changed — there is a “now web” emerging out of an ecosystem of loosely coupled products. There has always been an immediate, instant component to the web and web communications — it goes back to mailing lists, IM, email & blog commenting. But its taking on a whole new form — the density of the conversations and the speed at which they emerge and evolve is different. I first sensed the shift with the trending topic list on front page of Summize. This is a feature that the team created right out of the sentiment based technology of Summize v1. The first night we launched v2. I recall seeing the word IMAP was trending — my first thought this has to be a mistake, but when I ran the search it turned out that Gmail was having IMAP issues. Then a few weeks later during a telephone call one participant on the call heard an explosion outside his home. He jumped off the call to see what was happening, Jay came back 5 mins later, shook up but with no idea what the noise was. This post shows the Summize stream of responses to a simple question — there had been a minor earthquake in VA. A few weeks later the earthquake in China was also emerged out of the twitter stream before it hit MSM.

We experienced this again last week — in full force — when we launched the bit.ly product. A deceptively simple URL shortener that we developed with Dave Winer. Six days after its launch bit.ly is on a tear. The launch last week started with a fantastic write up by Marshall Kirkpatrick — it moved from there into twitter and summize and within mins we were getting live feedback on the product, how to tune and test it, complaints about the lack of privacy policy and ton of great ideas. I am learning as I go — but its a whole new world out there and thanks to Summize we can converse with in a far more direct and organized manner. This should be evident again today — run a search for this or this and watch it evolve.

In summary
Summize is a great example of what we aspire to do at betaworks. Working with a great team of technologists who created a wonderful product, one that on the surface is deceptively simple — where the smarts are all under the hood. One that we helped launch and scale. Many thanks to the Summize team. Jay, Abdur, Greg, Eric and team worked very very hard to make this happen — they peered into startup abyss and decided they werent going there — you guys are smart and brave. Thank you to the advisors who worked w/ Summize the make this happen — Gerry Campbell and Josh Auerbach. And thanks to the Twitter team. I have great hopes for the joint team.

Also see Summize post by Jay

A testament to what a great product and UI can achieve in no time at all.   This past week with the launch of bit.ly I spent much of my time on Twitter, Summize, Friend Feed and a handful of other services.  Google is playing nxt to no part in the now-web that is emerging out of this ecosystem.   Rafer also pointed me to this chart on compete.    More on search and navigation to come, for now some pictures — Summize traffic and a wonderful fireworks display from this evening in Shelter Island.

On the switchAbit/bitly/twitabit blog we did the official launch post.  Save you the jump here is the summary of what we offer and why its different

1. History — we remember the last 15 shortened URLs you’ve created. They’re displayed on the home page next time you go back. Cookie-based, sign in will come but the first rule of the game was keep it simple.

2. Click/Referrer tracking — Every time someone clicks on a short URL we add 1 to the count of clicks for that page and for the referring page.

3. There’s a simple API for creating short URLs from your web apps.

4. We automatically create three thumbnail images for each page you link through bit.ly, small, medium and large size. You can use these in presenting choices to your users.

5. We automatically mirror each page, never know when you might need a backup. 

6. Most important for professional applications, you can access all the data about each page through a simple XML or JSON interface. Example.

7. All the standard features you expect from serious url-shorteners.    

And it’s just the beginning, we’re tracking lots more data so that as more URLs are shortened by bit.ly we’ll be able to turn on more features.   Marshall talks about some of what we are going to do on the data side in the RWW article below. 

More to come on how this fits with switchabit, twitabit, findings — the cluster of services we are building.    For now some commentary:

ReadWriteWeb

Bit.ly Is a Big Deal URL Shortener

Scripting News

Alley Insider

Summize

NilsR

I’ll save you the jump. The chart below shows the effect on pageviews of Summize integration into Hahlo. Search changes the way people interact with an application — see the engagement jump.

People came, they searched … but they also seem to have left the browsers on, watching the WWDC conversations flow by. This is interesting and unusual — search as a browse/monitor experience is different to the way search has been thought of to date. We have also seen this with the trending topics on Summize. Conversational search is a big idea – the Summize team are starting to figure it out.

You can see the tweet stream below via a search on Summize.  We  talk about this stuff ad-infinitum but its amazing to see it unfold before one’s eyes.   The first tweet is from 1.35pm right after the quake.   The last one on the screen shot was approx. 3.10pm — it links to the confirmation from the USG:

(USGS has confirmed a magnitude 1.8 “micro” earthquake occurred near Annandale, VA at 1:30pm.  There have been no reports of damage or injuries.)

note the screen shot below is a compilation of tweets, re-run the search on falls church at Summize.com